Scaling LLM Knowledge Boundaries via Distribution-Optimized Synthesis

• URL: http://arxiv.org/abs/2606.23271v1
• Date: Mon, 22 Jun 2026 12:50:00 GMT
• Title: Scaling LLM Knowledge Boundaries via Distribution-Optimized Synthesis
• Abstract summary: We propose a framework that introduces knowledge density to drive synthesis through a three-stage feedback mechanism.<n>We conduct experiments on models from 0.6B to 16B (Qwen, Ling, LLaMA) and data scales from 1B to 5B tokens.<n>Key findings are: (1) an optimal knowledge distribution consistently maximizes boundary expansion; (2) this distribution is stable across backbones and scales; (3) KDoS outperforms baselines across six knowledge benchmarks.
• Score: 50.70475580369854
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Knowledge injection via synthetic data is crucial for enhancing Large Language Models (LLMs). However, current synthesis methods simply stop at preset token counts or fixed data ratios, lacking awareness of knowledge distribution. This results in some domains being sparse while others are redundant, limiting LLM knowledge boundaries. We revisit knowledge injection from a distribution perspective and hypothesize that an optimal knowledge distribution exists to maximize knowledge boundary expansion. We propose KDoS (Knowledge Distribution-optimized Synthesis), a framework that introduces knowledge density to drive synthesis through a three-stage feedback mechanism, shifting from blind generation to distribution-optimized synthesis. We construct Wikipedia-based synthetic data with varying knowledge distributions and conduct experiments on models from 0.6B to 16B (Qwen, Ling, LLaMA) and data scales from 1B to 5B tokens. Our key findings are: (1) an optimal knowledge distribution consistently maximizes boundary expansion; (2) this distribution is stable across backbones and scales; (3) KDoS outperforms baselines across six knowledge benchmarks. Our work offers a new perspective and practical framework for synthetic data-driven knowledge injection.

Related papers

• Generating Pretraining Tokens from Organic Data for Data-Bound Scaling [28.30636190022749]
  SynPro is a synthetic data generation framework that helps LLMs more thoroughly learn from limited organic data.<n>We pretrain 400M and 1.1B models with 10% of their Chinchilla-optimal tokens (0.8B and 2.2B) from DCLM-Baseline.<n>Our results reveal that organic data is significantly underutilized by standard repetition.
  arXiv  Detail & Related papers  (2026-05-18T04:44:40Z)
• EruDiff: Refactoring Knowledge in Diffusion Models for Advanced Text-to-Image Synthesis [49.883192716595026]
  EruDiff aims to rectify the knowledge distribution of intractable implicit prompts with that of well-defined explicit anchors.<n>We develop the Diffusion Knowledge Distribution Matching (DK-DM) to register the knowledge distribution of implicit prompts with that of well-defined explicit anchors.<n>We employ the Negative-Only Reinforcement Learning (NO-RL) strategy for fine-grained correction.
  arXiv  Detail & Related papers  (2026-03-21T14:04:08Z)
• Key-Augmented Neural Triggers for Knowledge Sharing [3.8718804267599416]
  Key-Augmented Neural Triggers embeds knowledge anchors into both training and inference.<n>It reduces fragmentation and grounding inference in localized context.<n>It is well-suited for scalable, low-latency, on-premise deployments.
  arXiv  Detail & Related papers  (2025-08-05T11:40:56Z)
• Structural Entropy Guided Agent for Detecting and Repairing Knowledge Deficiencies in LLMs [11.724887822269528]
  Large language models (LLMs) have achieved unprecedented performance by leveraging vast pretraining corpora.<n>Their performance remains suboptimal in knowledge-intensive domains such as medicine and scientific research.<n>We propose a novel Structural Entropy-guided Knowledge Navigator (SENATOR) framework that addresses the intrinsic knowledge deficiencies of LLMs.
  arXiv  Detail & Related papers  (2025-05-12T02:21:36Z)
• Scaling Laws of Synthetic Data for Language Models [125.41600201811417]
  We introduce SynthLLM, a scalable framework that transforms pre-training corpora into diverse, high-quality synthetic datasets.<n>Our approach achieves this by automatically extracting and recombining high-level concepts across multiple documents using a graph algorithm.
  arXiv  Detail & Related papers  (2025-03-25T11:07:12Z)
• ExaRanker-Open: Synthetic Explanation for IR using Open-Source LLMs [60.81649785463651]
  We introduce ExaRanker-Open, where we adapt and explore the use of open-source language models to generate explanations. Our findings reveal that incorporating explanations consistently enhances neural rankers, with benefits escalating as the LLM size increases.
  arXiv  Detail & Related papers  (2024-02-09T11:23:14Z)
• Propagating Knowledge Updates to LMs Through Distillation [97.3628651636153]
  We show that a context-based approach can both impart knowledge about entities and propagate that knowledge to enable broader inferences. Our experiments demonstrate that this approach is more effective at propagating knowledge updates than fine-tuning and other gradient-based knowledge-editing methods.
  arXiv  Detail & Related papers  (2023-06-15T17:39:50Z)
• Principled Knowledge Extrapolation with GANs [92.62635018136476]
  We study counterfactual synthesis from a new perspective of knowledge extrapolation. We show that an adversarial game with a closed-form discriminator can be used to address the knowledge extrapolation problem. Our method enjoys both elegant theoretical guarantees and superior performance in many scenarios.
  arXiv  Detail & Related papers  (2022-05-21T08:39:42Z)
• Surrogate- and invariance-boosted contrastive learning for data-scarce applications in science [2.959890389883449]
  We introduce surrogate- and invariance-boosted contrastive learning (SIB-CL), a deep learning framework which incorporates three inexpensive'' and easily obtainable auxiliary information sources to overcome data scarcity. We demonstrate SIB-CL's effectiveness and generality on various scientific problems, e.g., predicting the density-of-states of 2D photonic crystals and solving the 3D time-independent Schrodinger equation.
  arXiv  Detail & Related papers  (2021-10-15T23:08:24Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.