Skip to content
Log in

Labeling Training Data for Entity Matching Using Large Language Models

Aaron Steiner Christian Bizer
Jun 2026 · arXiv.org · Vol abs/2606.28823 · 0 citations · 27 references
Computer Science

TL;DR

Results indicate that current LLMs, when combined with a suitable pair-selection method, can substantially reduce or even eliminate the manual effort required to label use case-specific training data for entity matching.

Abstract

Recent large language models (LLMs) achieve strong performance on entity matching without requiring task-specific training data. However, applying these models to large sets of candidate pairs remains slow and costly. In contrast, entity matchers using traditional machine learning methods or small language models (SLMs), such as RoBERTa, offer much faster inference but require task-specific training data. This paper investigates whether the need to provide task-specific training data can be avoided by using knowledge-distillation workflows, in which an LLM serves as a teacher model to label training pairs that are subsequently used to train a smaller student model. We investigate knowledge distillation for entity matching along the following dimensions: pair-selection strategy, teacher model, label post-processing method, and student model. We evaluate the workflows using the Abt-Buy, Walmart-Amazon, WDC Products, DBLP-ACM, and DBLP-Scholar benchmarks, and compare the performance of student models trained with machine-labeled data to the performance of the same models trained using the benchmark training sets. Our experiments show that student models trained using the machine-labeled sets perform approximately on par with models trained on the benchmark training sets, with the remaining differences in both directions staying below two F1 points. Using GPT-5.2 to label the training sets for all five benchmarks costs US\$28.31 to US\$40.88, whereas manually labeling the same training sets is estimated to require 470 hours of work. At inference time, Ditto is 41.5 to 534 times faster than directly using an LLM to perform the matching tasks. These results indicate that current LLMs, when combined with a suitable pair-selection method, can substantially reduce or even eliminate the manual effort required to label use case-specific training data for entity matching.

DOI arXiv Semantic Scholar
View source

Similar papers

Open access Aug 2026

Optimizing sample selection for large language model-based entity matching using AssistEM

AssistEM, a framework for efficient LLM adaptation to EM via principled data selection, demonstrates that selective fine-tuning not only accelerates adaptation but also improves training efficiency (requiring fewer GPU hours), enabling open-source LLMs to rival–and in some cases outperform–closed-source models.

John Bosco Mugeni, S. Lynden, Toshiyuki Amagasa et al. · 0 citations
Preprint Jul 2026

Beyond Scale and Generation: Understanding Language Model-based Entity Matching

The factors underlying performance differences across matcher architectures are clarified and motivate future research and benchmark designs that better disentangle architectural choices from model-level factors while explicitly evaluating distribution shift and cross-dataset transferability.

Zeyu Zhang, Xue Li, Iacer Calixto et al. · 0 citations
Open access 2026

LMEnt: A Suite for Analyzing Knowledge in Language Models from Pretraining Data to Representations

LMEnt is released to support studies of knowledge in LMs, including knowledge representations, plasticity, editing, attribution, hallucinations, and learning dynamics, finding that entity co-occurrence and mention forms—which are difficult to study with existing tools—affect learning trends.

Daniela Gottesman, Alon Gilaie-Dotan, Ido Cohen et al. · 0 citations
Open access 2026

Cost-Effective Entity Matching With Large Language Models and Diversity-Aware Example Selection

Entity matching is the task of identifying database entries that refer to the same entity across multiple data sources, enabling data integration across domains. Recent studies have explored large language models (LLMs) for entity matching using in-context learning (ICL), where a small number of labeled examples are provided in the prompt. However, existing approaches implicitly assume the availability of large labeled datasets and do not consider the cost of annotation. In this paper, we propose a cost-effective LLM-based entity matching framework that reduces the number of annotations required for entity matching. The framework generates candidate pairs from input data sources using blocking and selects a small set of representative pairs for annotation. We introduce D-diversity, a diversity-aware example selection method that selects examples based on differences between entity pairs to improve ICL effectiveness. Experimental results show that diversity-aware selection generally improves average performance over random selection across multiple datasets. The proposed framework achieves competitive performance with only 10 annotated examples, reducing the number of annotated examples required by more than 97% compared with a deep learning-based method. Furthermore, we analyze the effects of varying the number of annotated examples, example ordering, and prompt design, and show that their impact varies depending on the model and dataset. These findings provide practical insights for applying LLM-based entity matching under limited annotation budgets.

Kazuaki Akashi, Manami Ogawa, Kenichi Tayama · 0 citations

Augmenting Datasets for Fine-Tuning Large Language Models Using Semantic Variations

This study explores a semantic variation methodology to augment training data by generating question-answer pairs with explicit control over semantic similarity, and shows that semantically controlled augmentation improves domain-specific knowledge acquisition while preserving consistency.

Alexander Chen, Caroline Tang, Jennifer Sleeman · 0 citations