Aug 2026· Proceedings of the 32nd ACM SIGKDD Conference on Knowledge Discovery and Data Mining V.2· 1 citation· 10 references
TL;DR
This work proposes a novel framework, DOMINO, that learns a minimal sufficient domain representation from reference samples and leverages it to guide the generation of domain-aligned synthetic data, enabling practical and scalable domain adaptation without manual prompt design or natural language domain specifications.
Abstract
Large Language Models have demonstrated remarkable progress in general-purpose capabilities and can achieve strong performance in specific domains through fine-tuning on domain-specific data. However, acquiring high-quality data for target domains remains a significant challenge. Existing data synthesis approaches follow a deductive paradigm, heavily relying on explicit domain descriptions expressed in natural language and careful prompt engineering, limiting their applicability in real-world scenarios where domains are difficult to describe or formally articulate. In this work, we tackle the underexplored problem of domain-specific data synthesis through an inductive paradigm, where the target domain is defined only through a set of reference examples, particularly when domain characteristics are difficult to articulate in natural language. We propose a novel framework, DOMINO, that learns a minimal sufficient domain representation from reference samples and leverages it to guide the generation of domain-aligned synthetic data. DOMINO integrates prompt tuning with a contrastive disentanglement objective to separate domain-level patterns from sample-specific noise, mitigating overfitting while preserving core domain characteristics. Theoretically, we prove that DOMINO expands the support of the synthetic data distribution, ensuring greater diversity. Empirically, on challenging coding benchmarks where domain definitions are implicit, fine-tuning on data synthesized by DOMINO improves Pass@1 accuracy by up to 4.63% over strong, instruction-tuned backbones, demonstrating its effectiveness and robustness. This work establishes a new paradigm for domain-specific data synthesis, enabling practical and scalable domain adaptation without manual prompt design or natural language domain specifications. The full version is available at https://arxiv.org/abs/2605.30039.
Natural Language Understanding (NLU) addresses cross-domain challenges in understanding language due to inconsistent data distributions, uncertain concept semantics, and inconsistent language use specific to individual domains across different geographical locations. The majority of classical deep learning systems perform poorly because it cannot generalize, leading to cross-domain underperformance in practical settings. Therefore, this research develops a new class of meta-learning that provides a systematic means to advance cross-domain adaptability for NLU in the context of distributed knowledge systems. More specifically, this approach combines federated learning methods, model-agnostic meta-learning (MAML), and domain-adaptive transformers for the first time to enable rapid extraction of domain-invariant representations while concurrently maintaining local contextual semantics across reasonably distributed datasets. The author conducted benchmark assessments on the most recent multi-domain evaluations and cross-lingual and domain-shifted data to simulate distributed knowledge systems. The custom model outperformed all baselines, transformers, and fine-tuning models by an average of 12.8% cross-domain accuracy, 15.3% lower generalization error, and 18.6% faster convergence rate. In the scenario, the new model decreased to less than 90% and maintained over 90% of its performance after a 40% reduction in training samples. Meta-learning enables superior transfer efficiency and model adaptation to new, previously unseen domains with minimal training. The new meta-learning frameworks successfully addressed the required performance and adaptability for advanced intelligent information systems for cross-domain NLU in distributed knowledge bases.
Deepti Patnaik, Rakshak Bharti· 2026 6th International Confe...· 0 citations
Mixture of Experts (MoE) Large Language Models (LLMs) have demonstrated exceptional performance in recent years. However, their significantly increased parameter count poses substantial challenges for achieving a fine-tuned model without modifying the MoE architecture or quantity, particularly under memory-constrained conditions. Previous studies have shown that MoE tends to have a subset of representative experts in a specific domain. This inherent characteristic creates the possibility of fine-tuning such models exclusively during the training phase by loading and training only a targeted subset of experts. To address the challenge, we propose an algorithm framework named DR-EFT (Domain-Representative Experts for Fine-Tuning), which explores and loads the domain-representative experts for subsequent retraining and reincorporation. DR-EFT operates based on a structured two-stage learning mechanism. Firstly, it achieves the representative experts via quantized model fine-tuning to remove the obstacle brought by the observed phenomenon of easily overlooked expert activation drift during fine-tuning. Then it enables continuous fine-tuning through denoting a novel MoE training dynamic. It finds that secondary relevant experts play a crucial role and should be included in the representative subset, which differs from existing pruning strategies that focus on the most relevant experts, thus enriching the theoretical framework of domain-specific expert. We propose two strategies of static fine-tuning or expert switching to achieve continuous adaptation of the retrained experts. Extensive experiments on multiple downstream tasks show that the proposed DR-EFT framework reduces the memory consumption of MoEs by close to 50% with only a marginal performance loss. Furthermore, our method demonstrates robustness through validations on popular MoE LLMs, including Qwen, DeepSeek, and Ernie.
Zhaomeng Cheng, Zhong Ji, Yan Zhang et al.· Neural Networks· 0 citations
This paper introduces Caduceus, a family of MoE-enhanced foundation models built with a hierarchical pre-training paradigm to jointly integrate biological and natural language, and incorporates a multi-task instruction tuning phase, enabling robust protein parsing and natural language question answering.
Mingze Yin, Yiheng Zhu, Jialu Wu et al.· Proceedings of the 32nd ACM...· 0 citations
This study explores a semantic variation methodology to augment training data by generating question-answer pairs with explicit control over semantic similarity, and shows that semantically controlled augmentation improves domain-specific knowledge acquisition while preserving consistency.
Alexander Chen, Caroline Tang, Jennifer Sleeman· 0 citations
While Large Language Models (LLMs) show great promise for personalization, they often lack specialized domain knowledge. Conventional solutions like fine-tuning struggle with high computational costs and rapid data updates, while Retrieval-Augmented Generation fails to provide fine-grained, token-level steering. Furthermore, chat-based interfaces remain dominant, whereas productive co-writing paradigms have not yet been well exploited beyond the coding domain. To this end, we introduce SteerWrite, a training-free framework designed for personalized co-writing. Our method effectively adapts the base model to specialized domains without gradient updates, with specific designs tailored to small datasets. Experiments demonstrate that SteerWrite achieves state-of-the-art performance across diverse datasets, metrics, and models, significantly reducing human editing effort.
Wenhao Mao, Chengbin Hou, Weixiao Wang et al.· 0 citations
An automated framework that constructs domain-specific benchmarks directly from unstructured corpora and systematically discovers tasks, enriches contextual grounding via iterative Socratic prompting, and generates diverse, progressively challenging evaluation instances that preserve established model-level evaluation trends are proposed.
Qingqing Lyu, Linjuan Wu, Yongliang Shen et al.· Annual Meeting of the Associ...· 0 citations