Evaluating teacher-supervised small medical language models for institution-constrained clinical QA.
TL;DR
For institution-constrained clinical QA, in-domain teacher-supervised fine-tuning was the most reliable evaluated adaptation path, and public-benchmark transfer was not a dependable proxy for local utility in the primary internal comparison.
Abstract
Objective
To evaluate whether small language models adapted on public medical benchmarks transfer reliably to institution-constrained open-answer clinical QA, and to identify which adaptation choices-supervision format, optimization objective, and backbone-determine local answer quality and evidence coverage.
Methods
We used DistilGPT2 (82M) as the primary student and Llama3 70B as the teacher. We compared four strategies: public-benchmark answer-only adaptation, public-benchmark chain-of-thought adaptation, in-domain teacher-supervised question-answer fine-tuning (QAFT), and direct preference optimization (DPO). We evaluated each strategy on cleaned public benchmarks and on an internal EHR-grounded open-answer task using token-F1, exact match, hallucinated evidence rate, and evidence coverage. We tested robustness via multi-seed reruns, a controlled DPO pair-construction ablation, a hard-negative DPO variant, and cross-backbone replication on TinyLlama 1.1B and a modern Qwen2.5-3B model.
Results
Public-benchmark adaptation did not produce stable gains in repeated-seed external evaluation. In the primary internal comparison, in-domain QAFT outperformed public-benchmark transfer (F1 0.1310 vs 0.1090). DPO did not improve answer quality uniformly; instead, it shifted models toward shorter, stricter responses with lower evidence coverage. In a controlled fixed-split three-run ablation, this shift varied with rejected-response construction. TinyLlama replication showed backbone-specific DPO effects.
Conclusion
For institution-constrained clinical QA, in-domain teacher-supervised fine-tuning was the most reliable evaluated adaptation path. Public-benchmark transfer was not a dependable proxy for local utility in the primary internal comparison, and DPO should be interpreted as operating-point control with local validation before deployment. We propose a local evaluation protocol: adapt on the target task, validate DPO as an operating-point control, and interpret groundedness jointly with evidence coverage.