R.I.P.: Better Models by Survival of the Fittest Prompts

• URL: http://arxiv.org/abs/2501.18578v2
• Date: Wed, 26 Feb 2025 18:58:53 GMT
• Title: R.I.P.: Better Models by Survival of the Fittest Prompts
• Abstract summary: We introduce a method for evaluating data integrity based on the assumption that low-quality input prompts result in high variance and low quality responses.<n>This is achieved by measuring the rejected response quality and the reward gap between the chosen and rejected preference pair.
• Score: 51.2293437372642
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Training data quality is one of the most important drivers of final model quality. In this work, we introduce a method for evaluating data integrity based on the assumption that low-quality input prompts result in high variance and low quality responses. This is achieved by measuring the rejected response quality and the reward gap between the chosen and rejected preference pair. Our method, Rejecting Instruction Preferences (RIP) can be used to filter prompts from existing training sets, or to make high quality synthetic datasets, yielding large performance gains across various benchmarks compared to unfiltered data. Using Llama 3.1-8B-Instruct, RIP improves AlpacaEval2 LC Win Rate by 9.4%, Arena-Hard by 8.7%, and WildBench by 9.9%. Using Llama 3.3-70B-Instruct, RIP improves Arena-Hard from 67.5 to 82.9, which is from 18th place to 6th overall in the leaderboard.

Related papers

• CircuitLens: Reasoning Circuits as Data Selection Signals for Reinforcement Learning with Verifiable Rewards [9.41727721095383]
  Circuit Reasoning Score (CRS) is a selection signal derived from 46 reasoning-sensitive attention heads.<n>CRS runs against the intuitive hypothesis that stronger reasoning-circuit engagement produces better training data.
  arXiv  Detail & Related papers  (2026-09-07T08:14:26Z)
• Less Data, Better Alignment: Data-Centric Multi-Evaluator Agreement for Preference Optimization [20.147316178118732]
  DMAPO generates candidate responses from the target policy, evaluates helpfulness, factuality, and conciseness with rubric-specialized evaluators.<n>This procedure accepts 1,871 of 54,236 Mistral-7B candidates (3.45%)<n>KTO trained on this set reaches 7.50 on MT-Bench, 95.5% length-controlled win rate against a text-davinci-003 reference, and 57.3% IFEval prompt accuracy.
  arXiv  Detail & Related papers  (2026-07-27T23:05:12Z)
• Multi-Turn Reinforcement Learning for Tool-Calling Agents with Iterative Reward Calibration [0.0]
  Training tool-calling agents with reinforcement learning on multi-turn tasks remains challenging.<n>We present the first application of MT-GRPO and GTPO for training a tool-calling agent on realistic customer service tasks.
  arXiv  Detail & Related papers  (2026-04-03T08:36:03Z)
• TTRV: Test-Time Reinforcement Learning for Vision Language Models [38.36757421304928]
  Existing methods for extracting reward signals in Reinforcement Learning typically rely on labeled data and dedicated training splits.<n>We propose TTRV to enhance vision language understanding by adapting the model on the fly at inference time, without the need for any labeled data.<n>Our approach delivers consistent gains across both object recognition and visual question answering (VQA), with improvements of up to 52.4% and 29.8%, respectively, and average boosts of 24.6% and 10.0% across 16 datasets.
  arXiv  Detail & Related papers  (2025-10-08T09:10:31Z)
• ReSURE: Regularizing Supervision Unreliability for Multi-turn Dialogue Fine-tuning [72.05731026796335]
  Multi-turn dialogue systems often suffer from degraded performance when exposed to low-quality data.<n>We propose ReSURE, an adaptive learning method that dynamically down-weights unreliable supervision without explicit filtering.<n>Experiments on both single-source and mixed-quality datasets show improved stability and response quality.
  arXiv  Detail & Related papers  (2025-08-27T15:54:01Z)
• RICo: Refined In-Context Contribution for Automatic Instruction-Tuning Data Selection [29.459431336830267]
  We propose a gradient-free method that quantifies the fine-grained contribution of individual samples to both task-level and global-level model performance.<n>We introduce a lightweight selection paradigm trained on RICo scores, enabling scalable data selection with a strictly linear inference complexity.
  arXiv  Detail & Related papers  (2025-05-08T15:17:37Z)
• Think, Prune, Train, Improve: Scaling Reasoning without Scaling Models [1.96238419451815]
  Large language models (LLMs) have demonstrated strong capabilities in programming and mathematical reasoning tasks, but are constrained by limited high-quality training data. We introduce a scalable framework that iteratively fine-tunes models on their own reasoning traces, using ground-truth pruning to ensure high-quality training data. This approach yields improved performance: on GSM8K, Gemma2-2B achieves a Pass@1 of 57.6% (from 41.9%), Gemma2-9B reaches 82%, matching LLaMA-3.1-70B, and LLaMA-3.1-70B attains 91%, even surpassing GPT-4o
  arXiv  Detail & Related papers  (2025-04-25T06:48:55Z)
• Clear Preferences Leave Traces: Reference Model-Guided Sampling for Preference Learning [59.11519451499754]
  Direct Preference Optimization (DPO) has emerged as a de-facto approach for aligning language models with human preferences.<n>Recent work has shown DPO's effectiveness relies on training data quality.<n>We discover that reference model probability space naturally detects high-quality training samples.
  arXiv  Detail & Related papers  (2025-01-25T07:21:50Z)
• CDR: Customizable Density Ratios of Strong-over-weak LLMs for Preference Annotation [15.776175440446414]
  Preference tuning of large language models (LLMs) relies on high-quality human preference data. We introduce customized density ratio (CDR), a training-free and highly effective method that leverages off-the-shelf LLMs for preference data annotation. We show that tailoring the density ratio reward function with specific criteria and preference exemplars enhances performance across domains and within target areas.
  arXiv  Detail & Related papers  (2024-11-04T18:54:39Z)
• Reward-Augmented Data Enhances Direct Preference Alignment of LLMs [63.32585910975191]
  We introduce reward-conditioned Large Language Models (LLMs) that learn from the entire spectrum of response quality within the dataset.<n>We propose an effective yet simple data relabeling method that conditions the preference pairs on quality scores to construct a reward-augmented dataset.
  arXiv  Detail & Related papers  (2024-10-10T16:01:51Z)
• Iterative Nash Policy Optimization: Aligning LLMs with General Preferences via No-Regret Learning [55.65738319966385]
  We propose a novel online algorithm, iterative Nash policy optimization (INPO) Unlike previous methods, INPO bypasses the need for estimating the expected win rate for individual responses. With an LLaMA-3-8B-based SFT model, INPO achieves a 42.6% length-controlled win rate on AlpacaEval 2.0 and a 37.8% win rate on Arena-Hard.
  arXiv  Detail & Related papers  (2024-06-30T08:00:34Z)
• WildBench: Benchmarking LLMs with Challenging Tasks from Real Users in the Wild [57.272096543738336]
  We introduce WildBench, an automated evaluation framework designed to benchmark large language models (LLMs) WildBench consists of 1,024 tasks carefully selected from over one million human-chatbot conversation logs. We have developed two metrics, WB-Reward and WB-Score, which are computable using advanced LLMs.
  arXiv  Detail & Related papers  (2024-06-07T09:15:44Z)
• SimPO: Simple Preference Optimization with a Reference-Free Reward [43.136307294076545]
  Direct Preference Optimization (DPO) is a widely used offline preference optimization algorithm. We propose SimPO, a simpler yet more effective approach to DPO. SimPO consistently and significantly outperforms DPO without substantially increasing response length.
  arXiv  Detail & Related papers  (2024-05-23T16:01:46Z)
• Towards a Competitive End-to-End Speech Recognition for CHiME-6 Dinner Party Transcription [73.66530509749305]
  In this paper, we argue that, even in difficult cases, some end-to-end approaches show performance close to the hybrid baseline. We experimentally compare and analyze CTC-Attention versus RNN-Transducer approaches along with RNN versus Transformer architectures. Our best end-to-end model based on RNN-Transducer, together with improved beam search, reaches quality by only 3.8% WER abs. worse than the LF-MMI TDNN-F CHiME-6 Challenge baseline.
  arXiv  Detail & Related papers  (2020-04-22T19:08:33Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.