2026· Proceedings of the First Workshop on Structured Understanding, Retrieval, and Generation in the LLM Era (SURGeLLM 2026)· pp. 209-218· 0 citations· 22 references
TL;DR
It is shown that format choice remains important even after fine-tuning; models learn more efficiently with specific formats rather than adapting to any format; this finding allows format selection to be done via inference alone, avoiding costly trial-and-error fine-tuning runs.
Abstract
Language models are often used for tasks involving structured data like tables and graphs, but there is no principled approach for choosing the best format to represent such data for fine-tuning. We address this in three steps. First, we show that format choice remains important even after fine-tuning; models learn more efficiently with specific formats rather than adapting to any format. Second, we show that a pre-trained model can suggest its own candidate formats by auto-completing partial prompts, reducing reliance on developer intuition. Third, and most importantly, we demonstrate that base model performance across formats reliably predicts post-fine-tuning performance: the format that performs best before fine-tuning remains among the top candidates after fine-tuning in 16 out of 18 settings across three data structure types, three models, and six tasks. This finding allows format selection to be done via inference alone, avoiding costly trial-and-error fine-tuning runs.
This study explores a semantic variation methodology to augment training data by generating question-answer pairs with explicit control over semantic similarity, and shows that semantically controlled augmentation improves domain-specific knowledge acquisition while preserving consistency.
Alexander Chen, Caroline Tang, Jennifer Sleeman· 0 citations
TopoTuner is competitive with full fine-tuning while training only 1-2% of the model parameters, and outperforms LoRA in 7 out of 9 model-dataset settings, which can change up to 39.57% of the projection parameters.
Abdulkadir Erol, Yash Mahajan, Vepaul Hariprashad et al.· 0 citations
The factors underlying performance differences across matcher architectures are clarified and motivate future research and benchmark designs that better disentangle architectural choices from model-level factors while explicitly evaluating distribution shift and cross-dataset transferability.
Zeyu Zhang, Xue Li, Iacer Calixto et al.· 0 citations
As the context window of Large Language Models (LLMs) continues to expand, the data required to effectively train and evaluate these capabilities remains underexplored. With existing research primarily focuses on architectural optimization, there is a need for a systematic, data-centric review. This survey bridges this gap by investigating the data foundations of Long-Context Language Models (LCMs). We begin by examining current data strategies alongside their strengths and limitations, mapping the required data to desired model capabilities. Building on this, we explore how targeted training data designs drive core, often interconnected skills such as retrieval, reasoning, and aggregation. Furthermore, we analyze the evaluation landscape, illustrating how selecting appropriate benchmarks is crucial for probing capability boundaries and guiding effective model selection. Finally, we synthesize actionable guidelines for data construction and outline critical future directions to propel the advancement of long-context language models, including quantifying data quality, establishing scaling laws for length distributions, and developing dynamic evaluation frameworks.
Zechen Sun, Yu-Yang Sun, Zhao-yu Su et al.· Transactions of the Associat...· 0 citations
LMEnt is released to support studies of knowledge in LMs, including knowledge representations, plasticity, editing, attribution, hallucinations, and learning dynamics, finding that entity co-occurrence and mention forms—which are difficult to study with existing tools—affect learning trends.
Daniela Gottesman, Alon Gilaie-Dotan, Ido Cohen et al.· Transactions of the Associat...· 0 citations
While Large Language Models (LLMs) show great promise for personalization, they often lack specialized domain knowledge. Conventional solutions like fine-tuning struggle with high computational costs and rapid data updates, while Retrieval-Augmented Generation fails to provide fine-grained, token-level steering. Furthermore, chat-based interfaces remain dominant, whereas productive co-writing paradigms have not yet been well exploited beyond the coding domain. To this end, we introduce SteerWrite, a training-free framework designed for personalized co-writing. Our method effectively adapts the base model to specialized domains without gradient updates, with specific designs tailored to small datasets. Experiments demonstrate that SteerWrite achieves state-of-the-art performance across diverse datasets, metrics, and models, significantly reducing human editing effort.
Wenhao Mao, Chengbin Hou, Weixiao Wang et al.· 0 citations