Spokes: Optimizing for Diverse Pretraining Data Selection

• URL: http://arxiv.org/abs/2606.15216v1
• Date: Sat, 13 Jun 2026 09:28:46 GMT
• Title: Spokes: Optimizing for Diverse Pretraining Data Selection
• Abstract summary: We introduce a probabilistic diversification framework based on the G-Vendi score, optimized via exponentiated gradient descent.<n>Our method produces subsets substantially more diverse than those obtained via random sampling.<n>We evaluate our approach on FineWeb and DCLM, where it consistently outperforms existing methods.
• Score: 82.66872118512403
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Diversity plays a critical role in data selection, improving performance under fixed data budgets by reducing redundancy and repetition. However, optimizing for diversity is inherently challenging, as it is a set-level property that depends on interactions between data points rather than individual examples. As a result, existing approaches typically rely on proxies or approximations, which often fail to ensure sufficiently diverse subsets. In this work, we directly optimize diversity by introducing a probabilistic diversification framework based on the G-Vendi score, optimized via exponentiated gradient descent. Our method produces subsets that are substantially more diverse than those obtained via random sampling, achieving a +489 increase in G-Vendi score on a 500k-sample subset. We evaluate our approach on FineWeb and DCLM, where it consistently outperforms existing methods. Notably, SPOKES (diversity-only) improves average downstream performance by +0.4 and +0.5 points over random sampling on DCLM and FineWeb, respectively. More importantly, jointly optimizing for both quality and diversity yields the strongest results: SPOKES achieves gains of +1.5 and +1.4 points on DCLM and FineWeb, outperforming all baselines, including semantic deduplication and quality filtering.

Related papers

• GRaSp: Automatic Example Optimization for In-Context Learning in Low-Data Tasks [0.3374875022248866]
  In-context learning enables large language models to adapt to new tasks, but their performance is sensitive to the selected examples.<n>We propose GRaSp, a three-stage framework for automatic in-context example optimization.<n>By first generating a large synthetic candidate pool, then structuring it with clustering and dimensionality reduction, and finally using genetic algorithms to find the optimal in-context examples, the framework shows consistent improvements on the NER task.
  arXiv  Detail & Related papers  (2026-05-08T09:00:28Z)
• GDPO: Group reward-Decoupled Normalization Policy Optimization for Multi-reward RL Optimization [133.27496265096445]
  We show how to apply Group Relative Policy Optimization under multi-reward setting without examining its suitability.<n>We then introduce Group reward-Decoupled Normalization Policy Optimization (GDPO), a new policy optimization method to resolve these issues.<n>GDPO consistently outperforms GRPO, demonstrating its effectiveness and generalizability for multi-reward reinforcement learning optimization.
  arXiv  Detail & Related papers  (2026-01-08T18:59:24Z)
• Exploring the Heterogeneity of Tabular Data: A Diversity-aware Data Generator via LLMs [7.355858495660162]
  We introduce Diversity-Aware Tabular data gEnerator (DATE), a framework that prepares high-quality and distributionally distinct examples for in-context learning.<n>DATE harnesses Large Language Models (LLMs) to explore the diversity of the partitioned distribution with decision tree reasoning as feedback, generating high-quality labeled data for each subset.<n>On average, DATE achieves a 23.75% reduction in error rate with just 100 generated data.
  arXiv  Detail & Related papers  (2025-12-26T08:02:51Z)
• Combatting Dimensional Collapse in LLM Pre-Training Data via Diversified File Selection [65.96556073745197]
  DiverSified File selection algorithm (DiSF) is proposed to select the most decorrelated text files in the feature space.<n>DiSF saves 98.5% of 590M training files in SlimPajama, outperforming the full-data pre-training within a 50B training budget.
  arXiv  Detail & Related papers  (2025-04-29T11:13:18Z)
• Add-One-In: Incremental Sample Selection for Large Language Models via a Choice-Based Greedy Paradigm [50.492124556982674]
  This paper introduces a novel choice-based sample selection framework.<n>It shifts the focus from evaluating individual sample quality to comparing the contribution value of different samples.<n>We validate our approach on a larger medical dataset, highlighting its practical applicability in real-world applications.
  arXiv  Detail & Related papers  (2025-03-04T07:32:41Z)
• Diversity-oriented Data Augmentation with Large Language Models [9.548912625579947]
  We propose a textbfunderlineDi-textbfunderlineoriented data textbfunderlineAugmentation framework (textbfDoAug)<n>Specifically, we utilize a diversity-oriented fine-tuning approach to train an LLM as a diverse paraphraser, which is capable of augmenting textual datasets by generating diversified paraphrases.<n>The results show that our fine-tuned LLM augmenter improves diversity while preserving label consistency, thereby enhancing the robustness and performance of downstream tasks.
  arXiv  Detail & Related papers  (2025-02-17T11:00:40Z)
• Diversify and Conquer: Diversity-Centric Data Selection with Iterative Refinement [8.509688686402438]
  Finetuning large language models on instruction data is crucial for enhancing pre-trained knowledge and improving instruction-following capabilities. This work addresses the question: How can we determine the optimal subset of data for effective training? Our method employs k-means clustering to ensure the selected subset effectively represents the full dataset.
  arXiv  Detail & Related papers  (2024-09-17T17:25:31Z)
• Controllable Prompt Tuning For Balancing Group Distributional Robustness [53.336515056479705]
  We introduce an optimization scheme to achieve good performance across groups and find a good solution for all without severely sacrificing performance on any of them. We propose Controllable Prompt Tuning (CPT), which couples our approach with prompt-tuning techniques. On spurious correlation benchmarks, our procedures achieve state-of-the-art results across both transformer and non-transformer architectures, as well as unimodal and multimodal data.
  arXiv  Detail & Related papers  (2024-03-05T06:23:55Z)
• Local policy search with Bayesian optimization [73.0364959221845]
  Reinforcement learning aims to find an optimal policy by interaction with an environment. Policy gradients for local search are often obtained from random perturbations. We develop an algorithm utilizing a probabilistic model of the objective function and its gradient.
  arXiv  Detail & Related papers  (2021-06-22T16:07:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.