Dynamic Context Evolution for Scalable Synthetic Data Generation

• URL: http://arxiv.org/abs/2604.07147v1
• Date: Wed, 08 Apr 2026 14:38:11 GMT
• Title: Dynamic Context Evolution for Scalable Synthetic Data Generation
• Abstract summary: Large language models produce repetitive output when prompted independently across many batches.<n>We introduce Dynamic Context Evolution (DCE), comprising three mechanisms.<n>DCE reconstructs the generation prompt each batch using memory state and rotating diversity strategies.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large language models produce repetitive output when prompted independently across many batches, a phenomenon we term cross-batch mode collapse: the progressive loss of output diversity when a language model is prompted repeatedly without access to its prior generations. Practitioners have long mitigated this with ad hoc deduplication and seed rotation, but no principled framework exists. We introduce Dynamic Context Evolution (DCE), comprising three mechanisms: (1) verbalized tail sampling (the model labels each idea with a guess about how obvious it is, and obvious ideas are discarded), which filters high-probability candidates via model self-assessment; (2) semantic memory, which maintains a persistent embedding index to reject near-duplicates across batches; and (3) adaptive prompt evolution, which reconstructs the generation prompt each batch using memory state and rotating diversity strategies. In experiments across three domains (sustainable packaging concepts, educational exam questions, and creative writing prompts) and two model families (gpt-5-mini and claude-haiku-4-5), a component ablation across 2-3 random seeds per method shows that DCE achieves 0.0 +/- 0.0% collapse versus 5.6 +/- 2.0% for naive prompting, while producing 17-18 HDBSCAN clusters per seed versus naive's volatile 2-17, indicating reliably richer conceptual structure. These results are validated with an independent embedding model (all-MiniLM-L6-v2) and hold across sensitivity sweeps of the VTS threshold tau and dedup threshold delta. Deduplication and prompt evolution are individually insufficient but jointly effective, at approximately $0.50 per 1,000 candidates using only standard API calls, with no fine-tuning or custom architectures required.

Related papers

• Architecture-Dependent Causal Transfer of Activation States Across Large Language Models [51.56484100374058]
  Internal language activation states can be transferred causally between different large language models.<n>Injecting projected activations into a target model during generation produces a statistically significant, preregistered causal effect.<n>We conclude that end-to-end activation-state transfer between LLMs, as currently implemented, is architecture-dependent rather than universal.
  arXiv  Detail & Related papers  (2026-08-17T09:53:27Z)
• Opaque Epistemic Mediation: How LLM Deployment Configurations Shape the Validation of Pseudo-Science [0.0]
  Commercial large language models are increasingly used as knowledge references, yet their stance on contested scientific claims is neither stable nor transparent.<n>We tested how four major LLM families evaluate ethnonationalist pseudo-science derived from Frank Salter's biosocial framework.<n>Grok's Fast versions consistently assigned credibility scores of 70-75, two to five times higher than all other models.
  arXiv  Detail & Related papers  (2026-07-24T17:32:43Z)
• Speculative Refinement: A Hybrid Autoregressive Diffusion Decoding Strategy and Its Behavior Across Benchmarks [1.9800951131982487]
  We use Speculative Refinement (SpecRef) to warm-start a masked diffusion language model from an AR draft.<n>We evaluate SpecRef across six benchmarks (HumanEval, MBPP, GSM8K, BBH, ARC-Challenge, HellaSwag) with three distinct evaluation protocols.<n>These observations apply to any multi-stage or non-autoregressive generation pipeline and point toward more diagnostic evaluation practices.
  arXiv  Detail & Related papers  (2026-06-25T18:52:24Z)
• VeriEvol: Scaling Multimodal Mathematical Reasoning via Verifiable Evol-Instruct [71.22358682581215]
  Scaling reinforcement learning for visual mathematical reasoning requires more than generating harder questions.<n>We instantiate this as VeriEvol, an iterative framework with two components.<n>On a five-benchmark visual-math suite, scaling evolved SFT data from 10K to 250K samples raises the mean accuracy from 35.42 to 54.73.
  arXiv  Detail & Related papers  (2026-06-22T16:17:30Z)
• DeltaPrompts: Escaping the Zero-Delta Trap in Multimodal Distillation [49.98710755440242]
  Distillation enables compact Vision-Language Models (VLMs) to obtain strong reasoning capabilities.<n>We reveal a critical inefficiency in this approach: up to 69% of the prompts in standard chart / document reasoning datasets are effectively zero-delta.<n>We propose a staged synthesis pipeline that repurposes existing datasets as seeds, actively targeting student failure modes to produce better prompts.
  arXiv  Detail & Related papers  (2026-05-15T02:04:12Z)
• PLUME: Latent Reasoning Based Universal Multimodal Embedding [52.35354073629127]
  Universal multimodal embedding (UME) maps heterogeneous inputs into a shared retrieval space with a single model.<n>Recent approaches improve UME by generating explicit chain-of-thought (CoT) rationales before extracting embeddings.<n>We propose PLUME, a latent reasoning framework that advances UME by replacing verbalized CoT with a short autoregressive rollout of continuous latent states.
  arXiv  Detail & Related papers  (2026-04-02T14:04:53Z)
• Hydra: Unifying Document Retrieval and Generation in a Single Vision-Language Model [0.0]
  Hydra is a dual-head approach that provides both ColBERT-style late-interaction retrieval and autoregressive generation from a single vision-language model (VLM)<n>A single LoRA adapter, trained only for retrieval, is toggled at inference; disabling it recovers the base model's generation quality.<n>On ViDoRe V1, Hydra (4B) is within 1 percentage point of a controlled single-head baseline in a single training run.
  arXiv  Detail & Related papers  (2026-03-30T15:17:41Z)
• Do Understanding and Generation Fight? A Diagnostic Study of DPO for Unified Multimodal Models [0.0]
  Unified multimodal models share a language model backbone for both understanding and generating images.<n>Can DPO align both capabilities simultaneously?<n>We present the first systematic study of this question, applying DPO to Janus-Pro at 1B and 7B parameters.
  arXiv  Detail & Related papers  (2026-03-17T18:26:29Z)
• DiffGRM: Diffusion-based Generative Recommendation Model [63.35379395455103]
  Generative recommendation (GR) is an emerging paradigm that represents each item via a tokenizer as an n-digit semantic ID (SID)<n>We propose DiffGRM, a diffusion-based GR model that replaces the autoregressive decoder with a masked discrete diffusion model (MDM)<n> Experiments show consistent gains over strong generative and discriminative recommendation baselines on multiple datasets.
  arXiv  Detail & Related papers  (2025-10-21T03:23:32Z)
• You only need 4 extra tokens: Synergistic Test-time Adaptation for LLMs [50.54173262572369]
  Large language models (LLMs) are increasingly deployed in specialized domains such as finance, medicine, and agriculture.<n>We study label-free test-time adaptation for language models and present SyTTA, an inference-time framework that adapts models on-the-fly without additional supervision.
  arXiv  Detail & Related papers  (2025-10-11T14:00:39Z)
• Datarus-R1: An Adaptive Multi-Step Reasoning LLM for Automated Data Analysis [0.0]
  We present Datarus-R1-14B, a language model fine-tuned from Qwen 2.5-14B-Instruct to act as a virtual data analyst and graduate-level problem solver.<n>Datarus is trained not on isolated question-answer pairs but on full analytical trajectories including reasoning steps, code execution, error traces, self-corrections, and final conclusions.
  arXiv  Detail & Related papers  (2025-08-18T21:58:18Z)
• Discrete Diffusion Models for Language Generation [0.0]
  This thesis investigates the feasibility and performance of discrete diffusion models for natural language generation.<n>We use Bits Per Token (BPT), Negative Log-Likelihood (NLL), Perplexity (PPL), and Batch Processing Speed to assess generative performance.<n>The AR model outperforms in compression with a lower mean BPT of 4.59, but D3PM achieves higher processing speed, reaching up to 3.97 batches per sec.
  arXiv  Detail & Related papers  (2025-07-02T23:43:02Z)
• Ensuring Reproducibility in Generative AI Systems for General Use Cases: A Framework for Regression Testing and Open Datasets [0.0]
  We introduce GPR-bench, a benchmark that operationalizes regression testing for general purpose use cases.<n>We show that newer models generally improve correctness, but the differences are modest and not statistically significant.<n>In contrast, the concise-writing instruction significantly enhances conciseness, demonstrating the effectiveness of prompt engineering.
  arXiv  Detail & Related papers  (2025-05-02T12:31:43Z)
• CFReID: Continual Few-shot Person Re-Identification [127.60234742605832]
  Lifelong ReID has been proposed to learn and accumulate knowledge across multiple domains incrementally.<n>LReID models need to be trained on large-scale labeled data for each unseen domain, which are typically inaccessible due to privacy and cost concerns.<n>We propose Continual Few-shot ReID, which requires models to be incrementally trained using few-shot data and tested on all seen domains.
  arXiv  Detail & Related papers  (2025-03-24T09:17:05Z)
• AvgOut: A Simple Output-Probability Measure to Eliminate Dull Responses [97.50616524350123]
  We build dialogue models that are dynamically aware of what utterances or tokens are dull without any feature-engineering. The first model, MinAvgOut, directly maximizes the diversity score through the output distributions of each batch. The second model, Label Fine-Tuning (LFT), prepends to the source sequence a label continuously scaled by the diversity score to control the diversity level. The third model, RL, adopts Reinforcement Learning and treats the diversity score as a reward signal.
  arXiv  Detail & Related papers  (2020-01-15T18:32:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.