Jun 2026· arXiv.org· Vol abs/2606.29130· 1 citation· 32 references
Computer Science
TL;DR
Results demonstrate that knowledge distillation provides a practical and scalable solution for historical document processing, achieving competitive performance without excessive computational cost.
Abstract
We present DistilledGemma, an efficient and accurate system for the HIPE-2026 shared task on person-place relation extraction from multilingual historical newspaper articles in English, German, and French. Our approach adopts a three-stage knowledge distillation pipeline designed to balance classification accuracy with computational efficiency. In the first stage, we systematically explored prompt engineering strategies across eight large language models to identify the most effective reasoning architecture for this challenging task. In the second stage, we applied supervised fine-tuning (SFT) via QLoRA to a Gemma 4 26B A4B teacher model, leveraging its strong multilingual capabilities to generate silver-standard chain-of-thought traces across the training corpus. In the final stage, we performed response-level distillation to transfer these learned reasoning patterns into a compact Gemma 4 E2B student model. In the official evaluation, our team WHEREAMI ranked 3rd on the standard test set with an accuracy profile mean score of 0.688, and 2nd on the binary test set with a mean score of 0.8156. Notably, by distilling knowledge from the 26B teacher to the 2.3B student, we preserved strong reasoning capabilities while reducing the deployed model size to approximately 2.3B effective parameters; the LoRA adapters used during training were merged into the student for inference. This configuration ranked 2nd in the balanced efficiency-accuracy profile across both the standard and binary test sets. These results demonstrate that knowledge distillation provides a practical and scalable solution for historical document processing, achieving competitive performance without excessive computational cost.
This team investigates how far a lightweight, interpretable system can go without any pretrained language model at the relation classification stage on the HIPE-2026 shared task, finding that minimum character distance alone captures most of the classification signal.
This paper presents the PingAn-NLP system developed for SemEval-2026 Task 9, focusing on multilingual online polarization identification across 18 languages. We pro-pose a multi-stage optimization framework combining Supervised Fine-Tuning (SFT) and Group Relative Policy Optimization (GRPO). To overcome label imbalance and linguistic nuances, we utilized synthetic reasoning chain augmentation via a high-capacity teacher model (Qwen3-235B) and developed a Smart-Tradeoff reward mechanism to balance precision and recall during reinforcement learning. A language-aware tiered ensemble voting strategy was further implemented to optimize inference performance across diverse linguistic tracks. Our 8B-GRPO-Vote configuration achieved the highest Macro-F1 scores among our experimental variants in 7 out of 18 languages. Officially, our system secured second place in the Bengali, English, Odia, and Turkish tracks.
We present S EMANTIC V ECTORS , our sys-tem for POLAR@SemEval-2026 Task 9 on multilingual online polarization detection across 22 typologically diverse languages. Polarization is frequently conveyed through implicit rhetorical framing, making cross-lingual detection highly challenging. We address this with a Siamese dual-encoder jointly fine-tuning mDeBERTa-v3-base and XLM-RoBERTa-large via 4-bit QLoRA, fused with language-specific expert models (GBERT, Italian BERT, Swahili BERT) through an XGBoost meta-stacker with per-language Platt calibration. Rather than addressing class imbalance, focal loss functions as a hard-example miner , concentrating gradients on subtly framed instances rather than lexically obvious ones. Combined with per-language threshold optimization, our system achieves macro-F1 = 0.797 and accuracy = 0.827 across all 22 languages.
A. Dash, Priyanshu Mittal, Piyush Prashant et al.· SemEval@ACL· 1 citation
Frontier large language models achieve broad linguistic competence but degrade on specialist domains underrepresented in pre-training corpora. Domain-adaptive post-training (DAPT) on curated professional text partially closes this gap, yet the dominant approach flattens structured discourse into isolated document units, discarding the collaborative reasoning embedded in multi-party exchanges. This paper investigates whether preserving the full recursive structure of user forum threads during post-training is a more effective first step toward knowledge extraction than flattened question-answer pairs. Four open-source decoder-only models (TinyLlama 1.1B, Phi-2 2.7B, LLaMA-2-7B 6.8B, LLaMA-2-13B 13B) are post-trained using parameter-efficient LoRA adaptation on 4970 threads from AgTalk, an agricultural producer forum, under three conditions: flattened Q → A pairs, full recursive threads preserving reply order, and shuffled recursive threads with randomly permuted intermediate replies. Five hypotheses are tested through paired Wilcoxon signed-rank comparisons across 29 thread positions. DAPT significantly reduces perplexity relative to pretrained baselines across all architectures (H0 supported). Recursive training reduces perplexity relative to flattened training, an advantage clearest for the two LLaMA-2 models under matched-context evaluation (Wilcoxon win rates near 72%) and present but obscured by outlier variance at the 1.1B and 2.7B scales (H1 supported). However, ordered recursive training provides only a marginal advantage over shuffled (H2 inconclusive), attention analysis reveals identical U-shaped endpoint-weighted profiles regardless of training condition (H3: architectural not learned), and perplexity shows no systematic decrease with accumulating thread depth (H4 not supported). These results are attributed to Rotary Position Embedding constraints in decoder-only architectures that systematically underweight middle-thread content. Encoder–decoder architectures with bidirectional cross-attention are identified as a promising next step for exploiting the full collaborative structure of forum discourse.
Jeffrey D. Vitale· Machine Learning and Knowled...· 0 citations
AssistEM, a framework for efficient LLM adaptation to EM via principled data selection, demonstrates that selective fine-tuning not only accelerates adaptation but also improves training efficiency (requiring fewer GPU hours), enabling open-source LLMs to rival–and in some cases outperform–closed-source models.
John Bosco Mugeni, S. Lynden, Toshiyuki Amagasa et al.· International Journal of Dat...· 0 citations