Skip to content

Comparison of techniques for fine-tuning open-weight models for entity extraction from radiology reports

Sep 2026 · arXiv.org · Vol abs/2609.40236 · 0 citations · 31 references
Computer Science

TL;DR

For narrow, high-value clinical label-extraction tasks, distilling real reports, rather than generating synthetic ones, is what closes the gap to a hosted model, enabling a private, low-cost, version-stable on-premises alternative.

Abstract

Converting free-text radiology reports into structured labels supports cohort building, quality assurance, and monitoring of clinical imaging models, but the strongest label extractors are hosted proprietary models whose use raises privacy, cost, and reproducibility concerns. We asked whether a fine-tuned open-weight model (Gemma-3-12B) can match GPT-4o at multi-label intracranial hemorrhage (ICH) acuity extraction from non-contrast head-CT reports, and which ingredients matter. Using a 2x2 design, we crossed two adaptation strategies (a discriminative classification head, CH; generative instruction fine-tuning, IFT) with two training-data sources (distillation of real GPT-4o-labeled reports; synthetic reports generated by GPT-4o from real exemplars), across five training sizes, benchmarked on 100 expert-adjudicated reports against GPT-4o and the un-tuned open-weight base. The distilled instruction-tuned model (DIFT) matched GPT-4o (macro-F1 0.845 vs 0.850; p = 1.000) and exceeded the base model by 0.178. The decisive factor was the training-data source, not the fine-tuning method: both synthetic-data models failed to exceed the un-tuned open-weight base at any training size and underperformed the distilled models across all acuity classes. Fine-tuning and inference fit within the memory envelope of a single 24 GB consumer GPU. For narrow, high-value clinical label-extraction tasks, distilling real reports, rather than generating synthetic ones, is what closes the gap to a hosted model, enabling a private, low-cost, version-stable on-premises alternative.

View source

Similar papers

Open access Sep 2026

A framework for extraction of clinical information from radiological mammography reports using large language models and retrieval augmented generation

A structured Retrieval-Augmented Generation (RAG) framework that leverages annotated mammography reports as in-context demonstrations, enabling joint named entity and relation extraction without task-specific fine-tuning is proposed and achieves performance comparable to fine-tuned BETO models in low-resource settings,...

Eduardo Godoy, J. De Ferrari, Sofia Lazo et al. · 1 citation · ⚡1

Representation-guided in-context learning for medical image interpretation with multimodal large language models

RG-ICL is introduced, a training-free inference framework that retrieves query-aligned demonstrations using frozen encoders, without task-specific parameter updates, and indicates that for medical image interpretation, curating which reference cases an MLLM sees is a practical alternative to retraining it.

Min-Da Zhao, Fang-Yu Hu, Yan Luo et al. · 0 citations
Open access Sep 2026

An Automated, Contamination-Controlled VQA Benchmark for Evaluating Vision-Language Models on 3D Oncology Imaging

Vision-language models (VLMs) are increasingly applied to medical imaging, yet public benchmarks may reward memorization over perception: their images and questions can enter pretraining corpora, and many items remain answerable from question text alone. We present an automated, agent-driven pipeline that builds multip...

Bo Liu, Han Gu, Xiang-Rui Li et al. · 0 citations
#artificial intelligence Preprint Sep 2026

ChestPheNoT: Deployable, Auditable Label-Status-Evidence Extraction from Radiology Reports

CHESTPHENOT is a compact 0.5-3B language model that jointly extracts finding labels, three-class status (present/absent/uncertain), and verbatim supporting evidence spans and demonstrates that locally deployable models can provide competitive and directly auditable radiology-report extraction without relying on externa...

Kai Yu, Chen-Yu Zhu, Zai-Fu Zhan et al. · 0 citations

Related blog posts

MIT News · Artificial Intelligence Nov 17, 2026

Learning the language of machine learning

By studying the computational foundations of language learning, Associate Professor Jacob Andreas builds intelligent systems that can communicate effectively with humans.

MIT News · Artificial Intelligence Aug 27, 2026

Looking beyond natural sequences

A new machine-learning framework aims to improve the success rate of computational protein design while moving away from results that reproduce sequences found in nature.

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.