Can Large Language Models Identify Technical Terms in Documents Like Human Annotators?

Abstract

Processing technical terms in documents is essential for digital libraries and knowledge organisation, supporting applications such as scholarly search, information retrieval, and domain terminology construction. While document-level term identification, a sub-task of automatic term extraction (ATE), has been tackled as a sequence labelling task, the emergence of large language models (LLMs) opens new avenues for prompt-based extraction. However, evaluation practices have persistently relied on simple success or failure against a single gold standard, without analysing how model behaviours align with human annotations, especially on borderline cases where annotators disagree. To address this issue, this study investigates to what extent LLMs can perform term identification using prompt instructions mirroring human annotator conditions. By evaluating multiple commercial and open-weight LLMs against double human annotations in ACL RD-TEC 2.0, an ATE benchmark, we show that LLMs achieve performance close to human annotators (up to 76.6% F1, within 5 percentage points of the inter-human agreement of 81.6%), with open-weight 30B-scale models outperforming larger commercial counterparts. Crucially, our analysis reveals a distinctly conservative nature: LLMs reliably extract terms on which humans consistently agree, but omit terms on which humans disagree. Our findings reveal the limitations of relying on a single gold standard, highlighting the need for annotator-aligned prompting and controllable precision–recall tuning.