Greedy Information Projection for LLM Data Selection

• URL: http://arxiv.org/abs/2603.13790v1
• Date: Sat, 14 Mar 2026 06:42:20 GMT
• Title: Greedy Information Projection for LLM Data Selection
• Abstract summary: textscGIP casts selection as maximizing mutual information between a subset of examples and task-specific query signals.<n>textscGIP selects small subsets that match full-data fine-tuning while using only a fraction of examples.
• Score: 6.670884998638381
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We present \emph{Greedy Information Projection} (\textsc{GIP}), a principled framework for choosing training examples for large language model fine-tuning. \textsc{GIP} casts selection as maximizing mutual information between a subset of examples and task-specific query signals, which may originate from LLM quality judgments, metadata, or other sources. The framework involves optimizing a closed-form mutual information objective defined using both data and query embeddings, naturally balancing {\it quality} and {\it diversity}. Optimizing this score is equivalent to maximizing the projection of the query embedding matrix onto the span of the selected data, which provides a geometric explanation for the co-emergence of quality and diversity. Building on this view, we employ a fast greedy matching-pursuit procedure with efficient projection-based updates. On instruction-following and mathematical reasoning datasets, \textsc{GIP} selects small subsets that match full-data fine-tuning while using only a fraction of examples and compute, unifying quality-aware and diversity-aware selection for efficient fine-tuning.

Related papers

• A Critical Look at Targeted Instruction Selection: Disentangling What Matters (and What Doesn't) [14.070675074621043]
  Instruction fine-tuning involves selecting a subset of instruction training data from a large candidate pool, using a small query set from the target task.<n>Despite growing interest, the literature on targeted instruction selection remains fragmented and opaque.<n>In this work, we aim to bring clarity to this landscape by disentangling and systematically analyzing the two core ingredients: data representation and selection algorithms.
  arXiv  Detail & Related papers  (2026-02-16T12:33:05Z)
• Exploring Instruction Data Quality for Explainable Image Quality Assessment [58.345719195248314]
  We investigate the role of data quality of instruction tuning dataset for explainable IQA.<n>We find that selecting a subset of the data set randomly can even lead to better results than training with the entire instruction tuning dataset.<n>We propose a clustering-based data selection framework with three stages: clustering feature extraction, cluster quota allocation, and cluster sampling strategy.
  arXiv  Detail & Related papers  (2025-10-04T17:12:54Z)
• KITE: Kernelized and Information Theoretic Exemplars for In-Context Learning [30.471243464952625]
  In-context learning (ICL) has emerged as a powerful paradigm for adapting large language models to new and data-scarce tasks.<n>We study the problem of example selection in ICL from a principled, information theory-driven perspective.<n>We derive a principled surrogate objective that is approximately submodular, enabling the use of a greedy algorithm with an approximation guarantee.
  arXiv  Detail & Related papers  (2025-09-19T06:50:03Z)
• Subset Selection for Fine-Tuning: A Utility-Diversity Balanced Approach for Mathematical Domain Adaptation [0.0]
  We propose a refined approach to efficiently fine-tune large language models (LLMs) on specific domains like the mathematical domain.<n>Our approach combines utility and diversity metrics to select the most informative and representative training examples.
  arXiv  Detail & Related papers  (2025-05-02T18:20:44Z)
• Add-One-In: Incremental Sample Selection for Large Language Models via a Choice-Based Greedy Paradigm [50.492124556982674]
  This paper introduces a novel choice-based sample selection framework.<n>It shifts the focus from evaluating individual sample quality to comparing the contribution value of different samples.<n>We validate our approach on a larger medical dataset, highlighting its practical applicability in real-world applications.
  arXiv  Detail & Related papers  (2025-03-04T07:32:41Z)
• TSDS: Data Selection for Task-Specific Model Finetuning [39.19448080265558]
  The efficacy of task-specific finetuning largely depends on the selection of appropriate training data.<n>We present TSDS (Task-Specific Data Selection), a framework to select data for task-specific model finetuning.<n>We show that instruction tuning using data selected by our method with a 1% selection ratio often outperforms using the full dataset.
  arXiv  Detail & Related papers  (2024-10-15T05:54:17Z)
• Adapt-$\infty$: Scalable Continual Multimodal Instruction Tuning via Dynamic Data Selection [89.42023974249122]
  Adapt-$infty$ is a new multi-way and adaptive data selection approach for lifelong instruction tuning.<n>We construct pseudo-skill clusters by grouping gradient-based sample vectors.<n>We select the best-performing data selector for each skill cluster from a pool of selector experts.<n>This data selector samples a subset of the most important samples from each skill cluster for training.
  arXiv  Detail & Related papers  (2024-10-14T15:48:09Z)
• An incremental preference elicitation-based approach to learning potentially non-monotonic preferences in multi-criteria sorting [53.36437745983783]
  We first construct a max-margin optimization-based model to model potentially non-monotonic preferences. We devise information amount measurement methods and question selection strategies to pinpoint the most informative alternative in each iteration. Two incremental preference elicitation-based algorithms are developed to learn potentially non-monotonic preferences.
  arXiv  Detail & Related papers  (2024-09-04T14:36:20Z)
• LESS: Selecting Influential Data for Targeted Instruction Tuning [64.78894228923619]
  We propose LESS, an efficient algorithm to estimate data influences and perform Low-rank gradiEnt Similarity Search for instruction data selection. We show that training on a LESS-selected 5% of the data can often outperform training on the full dataset across diverse downstream tasks. Our method goes beyond surface form cues to identify data that the necessary reasoning skills for the intended downstream application.
  arXiv  Detail & Related papers  (2024-02-06T19:18:04Z)
• DsDm: Model-Aware Dataset Selection with Datamodels [81.01744199870043]
  Standard practice is to filter for examples that match human notions of data quality. We find that selecting according to similarity with "high quality" data sources may not increase (and can even hurt) performance compared to randomly selecting data. Our framework avoids handpicked notions of data quality, and instead models explicitly how the learning process uses train datapoints to predict on the target tasks.
  arXiv  Detail & Related papers  (2024-01-23T17:22:00Z)
• Auto-weighted Multi-view Feature Selection with Graph Optimization [90.26124046530319]
  We propose a novel unsupervised multi-view feature selection model based on graph learning. The contributions are threefold: (1) during the feature selection procedure, the consensus similarity graph shared by different views is learned. Experiments on various datasets demonstrate the superiority of the proposed method compared with the state-of-the-art methods.
  arXiv  Detail & Related papers  (2021-04-11T03:25:25Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.