EmbGen: Teaching with Reassembled Corpora

• URL: http://arxiv.org/abs/2605.19394v1
• Date: Tue, 19 May 2026 05:40:12 GMT
• Title: EmbGen: Teaching with Reassembled Corpora
• Abstract summary: We introduce EmbGen, a synthetic data generation pipeline that decomposes a corpus into entity-description pairs.<n>We evaluate EmbGen against EntiGraph, InstructLab and Knowledge-Instruct on three datasets of varied semantic heterogeneity.
• Score: 0.300988853836121
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Adapting small instruction-tuned models to specialized domains often relies on supervised fine-tuning (SFT) on curated instruction-response examples, which is expensive to collect at scale. Synthetic training examples generated by a teacher LLM from a domain corpus can reduce this cost, but existing pipelines can produce homogenized outputs and do not consistently capture cross-passage or cross-document dependencies. We introduce EmbGen, a synthetic data generation pipeline that decomposes a corpus into entity-description pairs, reassembles them using semantic structure inferred from embedding similarity, and then generates question-answer (QA) pairs via proximity, intra-cluster, and inter-cluster sampling with cluster-specialized system prompts. We evaluate EmbGen against EntiGraph, InstructLab and Knowledge-Instruct on three datasets of varied semantic heterogeneity, under fixed token budgets (5 and 20 million tokens). We use lexical overlap metrics, an LLM-as-a-judge rubric, and Binary Accuracy, a composed metric combining Factual Accuracy and Completeness for evaluation. EmbGen improves Binary Accuracy on the most heterogeneous dataset by 12.5% at 5M and 88.9% at 20M tokens budget, relative to the strongest baseline, while remaining competitive across other datasets with lower heterogeneity.

Related papers

• Querit-Reranker: Training Compact Multilingual Rerankers via Efficient Label-Free Distribution Adaptation [43.66118282567413]
  We present Querit-Reranker, a family of multilingual cross-encoder rerankers trained with a data-centric pipeline.<n>Our pipeline first learns general relevance modeling from large-scale ranking-oriented data, then adapts to target distributions.<n>Querit-Reranker-A0.4B improves average nDCG@10 from 54.11 to 59.28 on BEIR and from 59.87 to 67.70 on MIRACL.
  arXiv  Detail & Related papers  (2026-06-17T13:06:47Z)
• AGOP-IxG: A Gradient Covariance Filter for Local Feature Attribution on Tabular Data, with a Controlled Benchmark [0.0]
  AGOP-IxG is a fast per-sample attribution method that pre-multiplies the per-sample gradient by a top-$K$ rank-truncated Average Gradient Outer Product matrix.<n>In Part 1, we construct three synthetic multi-class tabular tasks (linear, sparse nonlinear, interaction-based) where ground-truth attribution per sample is analytically or numerically derivable.<n>In Part 2, we evaluate global faithfulness on Adult Income and Credit Card Default using the ROAR protocol.
  arXiv  Detail & Related papers  (2026-05-15T07:45:35Z)
• MinerU2.5-Pro: Pushing the Limits of Data-Centric Document Parsing at Scale [92.09717763663873]
  We present MinerU2.5-Pro, which advances the state of the art purely through data engineering and training strategy design.<n>At its core is a Data Engine co-designed around coverage, informativeness, and annotation accuracy.<n>We show that MinerU2.5-Pro achieves 95.69 on OmniDocBench v1.6, improving over the same-architecture baseline by 2.71 points and surpassing all existing methods.
  arXiv  Detail & Related papers  (2026-04-06T15:44:18Z)
• Distributional Open-Ended Evaluation of LLM Cultural Value Alignment Based on Value Codebook [50.10594064510559]
  DOVE is a distributional evaluation framework that directly compares human-written text with LLM-generated outputs.<n>We show that DOVE achieves superior predictive validity, attaining a 31.56% correlation with downstream tasks, while maintaining high reliability with as few as 500 samples per culture.
  arXiv  Detail & Related papers  (2026-03-16T08:33:10Z)
• Bridging the Semantic Gap for Categorical Data Clustering via Large Language Models [64.58262227709842]
  ARISE (Attention-weighted Representation with Integrated Semantic Embeddings) is presented.<n>It builds semantic-aware representations that complement the metric space of categorical data for accurate clustering.<n>Experiments on eight benchmark datasets demonstrate consistent improvements over seven representative counterparts.
  arXiv  Detail & Related papers  (2026-01-03T11:37:46Z)
• IVGAE: Handling Incomplete Heterogeneous Data with a Variational Graph Autoencoder [4.935498694293104]
  We present textbfIVGAE, a Variational Graph Autoencoder framework for robust imputation of incomplete heterogeneous data.<n>IVGAE constructs a bipartite graph to represent sample-feature relationships and applies graph representation learning to model structural dependencies.<n>Experiments on 16 real-world datasets show that IVGAE achieves consistent improvements in RMSE and downstream F1 across MCAR, MAR, and MNAR missing scenarios under 30% missing rates.
  arXiv  Detail & Related papers  (2025-11-27T05:14:50Z)
• PairUni: Pairwise Training for Unified Multimodal Language Models [50.01064884121653]
  Unified vision-language models (UVLMs) must perform both understanding and generation within a single architecture.<n>We propose PairUni, a unified framework that reorganizes data into understanding-generation (UG) pairs.<n>Our approach achieves balanced improvements on various UVLMs, outperforming strong UVLM RL baselines.
  arXiv  Detail & Related papers  (2025-10-29T16:47:02Z)
• BiXSE: Improving Dense Retrieval via Probabilistic Graded Relevance Distillation [6.272555849379284]
  BiXSE is a pointwise training method that optimize binary cross-entropy over graded relevance scores.<n>It achieves strong performance with reduced annotation and compute costs.<n>BiXSE offers a robust, scalable alternative for training dense retrieval models.
  arXiv  Detail & Related papers  (2025-08-09T02:15:17Z)
• Domain2Vec: Vectorizing Datasets to Find the Optimal Data Mixture without Training [53.07879717463279]
  textscDomain2Vec decomposes any dataset into a linear combination of several emphmeta-domains<n>textscDomain2Vec helps find the data mixture that enhances downstream task performance with minimal computational overhead.
  arXiv  Detail & Related papers  (2025-06-12T17:53:51Z)
• SEG:Seeds-Enhanced Iterative Refinement Graph Neural Network for Entity Alignment [13.487673375206276]
  This paper presents a soft label propagation framework that integrates multi-source data and iterative seed enhancement. A bidirectional weighted joint loss function is implemented, which reduces the distance between positive samples and differentially processes negative samples. Our method outperforms existing semi-supervised approaches, as evidenced by superior results on multiple datasets.
  arXiv  Detail & Related papers  (2024-10-28T04:50:46Z)
• Noisy Correspondence Learning with Self-Reinforcing Errors Mitigation [63.180725016463974]
  Cross-modal retrieval relies on well-matched large-scale datasets that are laborious in practice. We introduce a novel noisy correspondence learning framework, namely textbfSelf-textbfReinforcing textbfErrors textbfMitigation (SREM)
  arXiv  Detail & Related papers  (2023-12-27T09:03:43Z)
• EM for Mixture of Linear Regression with Clustered Data [6.948976192408852]
  We discuss how the underlying clustered structures in distributed data can be exploited to improve learning schemes. We employ the well-known Expectation-Maximization (EM) method to estimate the maximum likelihood parameters from $m$ batches of dependent samples. We show that if properly, EM on the structured data requires only $O(1)$ to reach the same statistical accuracy, as long as $m$ grows up as $eo(n)$.
  arXiv  Detail & Related papers  (2023-08-22T15:47:58Z)
• Mitigating Generation Shifts for Generalized Zero-Shot Learning [52.98182124310114]
  Generalized Zero-Shot Learning (GZSL) is the task of leveraging semantic information (e.g., attributes) to recognize the seen and unseen samples, where unseen classes are not observable during training. We propose a novel Generation Shifts Mitigating Flow framework for learning unseen data synthesis efficiently and effectively. Experimental results demonstrate that GSMFlow achieves state-of-the-art recognition performance in both conventional and generalized zero-shot settings.
  arXiv  Detail & Related papers  (2021-07-07T11:43:59Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.