A comparative analysis of multiple-choice questions and short answer questions in formative assessment: Psychometric properties and educational impact in first-year medical students

Abstract

Assessment methods are essential for evaluating medical students’ knowledge and cognitive skills. MCQs are widely used for their objectivity, while SAQs are considered better for assessing higher-order cognition, yet comparative psychometric evidence in formative settings is limited. To compare the difficulty index (DIF), discrimination index (DI), and overall test performance of MCQs and SAQs used in a formative assessment of first-year medical students, and to examine differences in item performance across achievement levels. Descriptive analytical cross-sectional study. The study included 178 first-year medical students enrolled in the Medical Parasitology course within the Locomotor and Skin (MLS-1) module at Ain Shams University during the 2024–2025 academic year. Students completed both MCQ and SAQ examinations, each comprising 20 items mapped to identical intended learning outcomes. Psychometric analysis included item difficulty, discrimination indices, internal consistency reliability, and comparative statistical testing between formats. Students scored significantly higher on MCQs than SAQs (mean 13.5 vs. 11.1 out of 20, p < .001, Cohen’s d = 0.59). MCQs were significantly easier than SAQs (mean DIF 0.67 vs. 0.55, p = .005), while discrimination indices did not differ significantly (mean DI 0.44 vs. 0.41, p = .276). Both formats demonstrated excellent reliability (Cronbach’s α > 0.80) and strong discrimination between highand low achievers. Pass/fail concordance was 66.3%, with more students passing MCQs but failing SAQs. MCQs and SAQs show strong psychometric quality. MCQs are easier, while SAQs are more challenging yet equally discriminative, supporting their complementary use in formative medical education