Domain-Specific Data Synthesis for LLMs via Minimal Sufficient Representation Learning

• URL: http://arxiv.org/abs/2605.30039v2
• Date: Fri, 29 May 2026 13:58:00 GMT
• Title: Domain-Specific Data Synthesis for LLMs via Minimal Sufficient Representation Learning
• Abstract summary: Large Language Models can achieve strong performance in specific domains through fine-tuning on domain-specific data.<n>Existing data synthesis approaches rely on explicit domain descriptions expressed in natural language and careful prompt engineering.<n>We propose a novel framework, DOMINO, that learns a minimal sufficient domain representation from reference samples and leverages it to guide the generation of domain-aligned synthetic data.
• Score: 72.60775633696593
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Large Language Models have demonstrated remarkable progress in general-purpose capabilities and can achieve strong performance in specific domains through fine-tuning on domain-specific data. However, acquiring high-quality data for target domains remains a significant challenge. Existing data synthesis approaches follow a deductive paradigm, heavily relying on explicit domain descriptions expressed in natural language and careful prompt engineering, limiting their applicability in real-world scenarios where domains are difficult to describe or formally articulate. In this work, we tackle the underexplored problem of domain-specific data synthesis through an inductive paradigm, where the target domain is defined only through a set of reference examples, particularly when domain characteristics are difficult to articulate in natural language. We propose a novel framework, DOMINO, that learns a minimal sufficient domain representation from reference samples and leverages it to guide the generation of domain-aligned synthetic data. DOMINO integrates prompt tuning with a contrastive disentanglement objective to separate domain-level patterns from sample-specific noise, mitigating overfitting while preserving core domain characteristics. Theoretically, we prove that DOMINO expands the support of the synthetic data distribution, ensuring greater diversity. Empirically, on challenging coding benchmarks where domain definitions are implicit, fine-tuning on data synthesized by DOMINO improves Pass@1 accuracy by up to 4.63\% over strong, instruction-tuned backbones, demonstrating its effectiveness and robustness. This work establishes a new paradigm for domain-specific data synthesis, enabling practical and scalable domain adaptation without manual prompt design or natural language domain specifications.

Related papers

• Understanding Cross-Domain Adaptation in Low-Resource Topic Modeling [25.915607750636333]
  We introduce domain adaptation for low-resource topic modeling, where a high-resource source domain informs a low-resource target domain without overwhelming it with irrelevant content.<n>We propose DALTA, a new framework that employs a shared encoder for domain-invariant features, specialized decoders for domain-specific nuances, and adversarial alignment to selectively transfer relevant information.<n>Experiments on diverse low-resource datasets demonstrate that DALTA consistently outperforms state-of-the-art methods in terms of topic coherence, stability, and transferability.
  arXiv  Detail & Related papers  (2025-06-09T05:59:18Z)
• Let Synthetic Data Shine: Domain Reassembly and Soft-Fusion for Single Domain Generalization [68.41367635546183]
  Single Domain Generalization aims to train models with consistent performance across diverse scenarios using data from a single source.<n>We propose Discriminative Domain Reassembly and Soft-Fusion (DRSF), a training framework leveraging synthetic data to improve model generalization.
  arXiv  Detail & Related papers  (2025-03-17T18:08:03Z)
• StyDeSty: Min-Max Stylization and Destylization for Single Domain Generalization [85.18995948334592]
  Single domain generalization (single DG) aims at learning a robust model generalizable to unseen domains from only one training domain. State-of-the-art approaches have mostly relied on data augmentations, such as adversarial perturbation and style enhancement, to synthesize new data. We propose emphStyDeSty, which explicitly accounts for the alignment of the source and pseudo domains in the process of data augmentation.
  arXiv  Detail & Related papers  (2024-06-01T02:41:34Z)
• UniGen: Universal Domain Generalization for Sentiment Classification via Zero-shot Dataset Generation [6.3823202275924125]
  We propose a novel approach to universal domain generalization that generates a dataset regardless of the target domain. Our experiments indicate that the proposed method accomplishes generalizability across various domains while using a parameter set that is orders of magnitude smaller than PLMs.
  arXiv  Detail & Related papers  (2024-05-02T05:46:13Z)
• Domain-Agnostic Prior for Transfer Semantic Segmentation [197.9378107222422]
  Unsupervised domain adaptation (UDA) is an important topic in the computer vision community. We present a mechanism that regularizes cross-domain representation learning with a domain-agnostic prior (DAP) Our research reveals that UDA benefits much from better proxies, possibly from other data modalities.
  arXiv  Detail & Related papers  (2022-04-06T09:13:25Z)
• Hierarchical Neural Data Synthesis for Semantic Parsing [16.284764879030448]
  We propose a purely neural approach of data augmentation for semantic parsing. We achieve the state-of-the-art performance on the development set (77.2% accuracy) using our zero-shot augmentation.
  arXiv  Detail & Related papers  (2021-12-04T01:33:08Z)
• Unsupervised Domain Adaptation for Semantic Segmentation via Low-level Edge Information Transfer [27.64947077788111]
  Unsupervised domain adaptation for semantic segmentation aims to make models trained on synthetic data adapt to real images. Previous feature-level adversarial learning methods only consider adapting models on the high-level semantic features. We present the first attempt at explicitly using low-level edge information, which has a small inter-domain gap, to guide the transfer of semantic information.
  arXiv  Detail & Related papers  (2021-09-18T11:51:31Z)
• Domain Adaptation for Semantic Parsing [68.81787666086554]
  We propose a novel semantic for domain adaptation, where we have much fewer annotated data in the target domain compared to the source domain. Our semantic benefits from a two-stage coarse-to-fine framework, thus can provide different and accurate treatments for the two stages. Experiments on a benchmark dataset show that our method consistently outperforms several popular domain adaptation strategies.
  arXiv  Detail & Related papers  (2020-06-23T14:47:41Z)
• Unsupervised Domain Clusters in Pretrained Language Models [61.832234606157286]
  We show that massive pre-trained language models implicitly learn sentence representations that cluster by domains without supervision. We propose domain data selection methods based on such models. We evaluate our data selection methods for neural machine translation across five diverse domains.
  arXiv  Detail & Related papers  (2020-04-05T06:22:16Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.