Skip to content

Domain-Specific Jargon in Large Language Models: A Comparative Analysis between General-Purpose and Specialist Models

Sep 2026 · 0 citations · 41 references
Computer Science

TL;DR

The results provide a case study in which a medically fine-tuned checkpoint does not improve jargon comprehension over its general-purpose counterpart, highlighting that domain adaptation should not be assumed to yield better performance on specialized terminology.

Abstract

Large Language Models (LLMs) have shown remarkable proficiency on general-purpose tasks, yet their performance often degrades in highly-specialized technical domains. Moreover, little is known about how parametric knowledge of domain-specific terms is encoded within these models. We address this gap by contributing two novel medical jargon evaluation benchmarks and evaluate a general-purpose Llama-3.1 model against a variant fine-tuned on medical-domain data. Surprisingly, the general-purpose model outperforms the medically fine-tuned model on both tasks. Using mechanistic interpretability tools, we find systematic patterns of miscalibration for the medically fine-tuned model. Instead of reorganizing parametric knowledge, the fine-tuned model places greater emphasis on a small subset of model components associated with jargon-favoring predictions. We find that applying component reweighting strategies against the benchmark tasks successfully suppresses these components and closes the gap with the general-purpose baseline. We also observe that some jargon-sensitive components transfer knowledge to the same tasks involving materials science jargon, suggesting they encode a partially domain-agnostic notion of specialized terminology. Our results provide a case study in which a medically fine-tuned checkpoint does not improve jargon comprehension over its general-purpose counterpart, highlighting that domain adaptation should not be assumed to yield better performance on specialized terminology.

View source

Similar papers

Open access Aug 2026

Autogenerating a Domain-Specific Question-Answering Data Set to Enable High-Performing Language Models for Magnetic Materials

Results demonstrate that domain-specific BERT models only need to be finetuned from vanilla BERT models (i.e. no domain-adaptive pretraining is needed), pending the availability of sufficiently large, high-quality, domain-specific QA data sets that this work shows how to autogenerate.

Charles R. Kelly, Jacqueline M. Cole · 0 citations
#artificial intelligence Preprint Sep 2026

Rethinking Domain Specialization for Open-Ended Scientific Reasoning in Astronomy Language Models

This work compares open-weight and API-served general-purpose, multimodal, and astronomy-specialized models using judge-based correctness and complementary reference metrics and highlights the role of domain-specific evaluation in determining which models, capabilities, and evaluation criteria are appropriate for scien...

Vanessa Lama, Sanjay Das, Emily J. Herron et al. · 0 citations
Conference Open access Sep 2025

DiverValue-Bench: A Benchmark and Fine-Tuning Framework for Aligning Large Language Models with Diverse Human Values

DiverValue-Bench is introduced, a population-aware benchmark for evaluating multi-dimensional value alignment across 74 countries/regions and it is shown that lightweight preference-based fine-tuning with Low-Rank Adaptation and Direct Preference Optimization substantially improves in-domain value alignment while yield...

Yao Liang, Dong-Cheng Zhao, Fei-Fei Zhao et al. · 0 citations
#small language model Open access Oct 2026

RETRIEVAL BEATS FINE-TUNING: ADAPTING A KAZAKH SMALL LANGUAGE MODEL FOR HISTORICAL QUESTION ANSWERING

Large language models (LLMs) have shown a strong ability to solve a wide range of language tasks. However, they became less practical for subject-specific applications, especially when dealing with low-resource languages. This paper shows an example of adaptation of small language models (SLMs) to solve subject-specifi...

G. Askar, A. Mutaikhan, M. Aitkozhayeva et al. · 0 citations
Open access

Evaluation and Distillation of Source Code Generation Tasks by Large Language Models

Two novel contributions are introduced: CodeEval and CodeQual, an open-source execution framework that provides researchers with a ready-to-use evaluation pipeline for evaluating and improving LLMs in software engineering contexts, encompassing both functional correctness assessment and subjective code quality evaluati...

Danny Brahman · 0 citations
Review Open access Oct 2026

Training datasets tailored for domain-specific large language models: challenges, methods and trends

Adapting LLMs to specialized domains critically depends on high-quality training data, making dataset construction a decisive factor in domain performance. This survey reviews methods for constructing domain-adaptation training datasets through a structured, data-centric analysis. Based on a comparative analysis of rep...

Xin-Yuan Qiu, Ke-Lan Ren, Xiao-Long Cui et al. · 0 citations

Related blog posts

MIT News · Artificial Intelligence Nov 17, 2026

Learning the language of machine learning

By studying the computational foundations of language learning, Associate Professor Jacob Andreas builds intelligent systems that can communicate effectively with humans.

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.