Two-point regression standard setting, improving discrimination at both ends of the grading scale in OSCEs.

Background

Standard setting in Objective Structured Clinical Examinations (OSCEs) typically identifies a single cut score to distinguish competent from non-competent candidates. However, this approach does not help defensible differentiation between 'competent' and 'excellent' students.

Objective

This study introduces a Two-Point Regression model, an extension of the borderline regression method, to establish two cut scores, one for competence and one for excellence, enabling more nuanced classification of student performance.

Methods

A retrospective analysis was performed on 10 OSCEs delivered to UK medical students. For each station, linear regressions of global ratings (0-3) against checklist scores were used to generate the traditional pass cut score (cX = 1). Additional thresholds for excellence were generated at global thresholds eX = 2, 2.25, 2.5 and 3. Outcomes included: proportion graded as excellent (≥70% after scaling), mean scaled grades, and mean global scores of 'excellent' students. Statistical analyses included chi-square, ANOVA, and post-hoc testing as appropriate.

Results

Pass/fail classifications were unchanged across all methods. The Two-Point regression model substantially reduced the proportion of students achieving excellence from 47% to 17.2% when the excellence cut score was set to a global threshold of eX = 2.25, aligning with national benchmarks. Mean global scores of 'excellent' students increased in line with threshold choice, demonstrating strong internal consistency (R2=0.9939).

Conclusion

The Two-Point Regression method offers an easily implementable and defensible approach for identifying excellence in numerically graded OSCEs, improving alignment between examiner global judgements and awarded grades. Threshold choice should be context-dependent, but the Two-Point Regression method provides a robust and adaptable framework.