Development and evaluation of a simple structured rubric for grading short answer questions.

Background

Constructed-response questions (CRQs) effectively assess pharmacy students' therapeutic reasoning but are underutilized due to inconsistent and resource-intensive grading. This study describes the development and evaluation of a simple, structured rubric template designed to address these challenges.

Methods

A 6-level rubric used in a Doctor of Pharmacy curriculum was modified into a 3-to-4-level rubric based on a single-point rubric design. It focused on defining "borderline pass" criteria and incorporated three specific therapeutic reasoning domains (making a decision, analytical skills, and content knowledge). It was implemented across three integrated organ-system courses in the 2023-2024 academic year. Perceptions of graders, item writers, and course directors were evaluated via a survey and focus groups. Interrater reliability between expert (faculty) and non-expert (residents) graders was evaluated using the intraclass correlation coefficient (ICC) across 50 randomly selected student answers for 8 CRQs.

Results

The survey response rate was 96%. Respondents reported high clarity regarding borderline criteria (95.7%), and over 90% of inexperienced graders found the rubric easy to use. Pre-grading calibration sessions using live student answers were highly valued (>90%). Focus group themes highlighted improved grading efficiency and item writing, alongside challenges in defining borderline criteria for basic science items. The grading consistency between expert and non-expert graders was good (ICC = 0.85; range: 0.82-0.87).

Conclusion

The simple, structured rubric template was successfully implemented, demonstrating high clarity and good grading consistency between expert and non-expert graders. The rubric may help optimize resource utilization and support the expanded use of CRQs in pharmacy education.