Metadata-Free Meta-Reweighted Direct Preference Optimization under Noisy Preference Labels

• URL: http://arxiv.org/abs/2607.09796v1
• Date: Thu, 09 Jul 2026 09:20:25 GMT
• Title: Metadata-Free Meta-Reweighted Direct Preference Optimization under Noisy Preference Labels
• Abstract summary: We propose a bilevel optimization framework and prove, under certain assumptions, that it can recover the DPO optimum under clean data.<n>Considering that high-quality metadata may be difficult to obtain, we propose a task-agnostic meta-knowledge-driven method.<n> Experiments on TL;DR summarization and Anthropic HH single-turn dialogue show that the proposed method improves training performance over multiple DPO baselines under different noise rates.
• Score: 4.826561433595647
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Direct Preference Optimization (DPO) has become an important method for aligning large language models (LLMs) with human preferences because it removes the need for explicit reward modeling and reinforcement learning optimization. However, its performance depends heavily on the quality of preference data, and noisy preference data in real-world settings can weaken alignment performance. To address this issue, we propose a bilevel optimization framework and prove, under certain assumptions, that this framework can recover the DPO optimum under clean data. We further derive a prior form for the learnable weighting function under asymmetric label-flipping noise. Considering that high-quality metadata may be difficult to obtain, we propose a task-agnostic meta-knowledge-driven method that enables meta-learning even when metadata is completely unavailable. To reduce the high cost of higher-order gradients in LLM meta-learning, we combine central-difference approximation with LoRA fine-tuning and develop a scalable training scheme. Experiments on TL;DR summarization and Anthropic HH single-turn dialogue show that the proposed method improves training performance over multiple DPO baselines under different noise rates.

Related papers

• Dynamic Important Example Mining for Reinforcement Finetuning [53.98969373888718]
  Reinforcement fine-tuning (RFT) is increasingly used to strengthen the reasoning abilities of large models.<n>Most data-centric RFT methods rely on static or sample selection, implicitly assuming a sample's value is fixed over training.<n>We propose Dynamic Important Example Mining (DIEM), a principled and fully automated framework that makes data utilization adaptive throughout RFT.
  arXiv  Detail & Related papers  (2026-08-29T13:12:25Z)
• Unbiased Alignment for Large Language Models with Noisy Preferences [74.69408994904965]
  We present a theoretical framework for unbiased alignment, introducing the Unbiased Reward Model (URM) loss and the Unbiased Direct Preference Optimization (UDPO) loss.<n>By mathematically correcting the distortion induced by preference noise, our novel objectives enable unbiased model training directly from noisy datasets.
  arXiv  Detail & Related papers  (2026-07-03T12:04:43Z)
• FedPDPO: Federated Personalized Direct Preference Optimization for Large Language Model Alignment [55.97027207627]
  Fed PDPO (Federated Personalized Direct Preference Optimization) is a personalized framework for preference alignment of large language models (LLMs)<n>It adopts a parameter-efficient fine-tuning architecture where each client maintains a frozen pretrained LLM backbone augmented with a Low-Rank Adaptation (LoRA) adapter, enabling communication-efficient aggregation.<n>Experiments on multiple preference datasets demonstrate state-of-the-art performance, achieving up to 4.80% average accuracy improvements in federated intra-domain and cross-domain settings.
  arXiv  Detail & Related papers  (2026-03-20T08:24:49Z)
• When Data is the Algorithm: A Systematic Study and Curation of Preference Optimization Datasets [29.94723846950853]
  We present the first comprehensive, data-centric analysis of popular open-source DPO corpora.<n>We leverage the Magpie framework to annotate each sample for task category, input quality, and preference reward.<n>This enables a scalable, fine-grained inspection of preference quality across datasets, revealing structural and qualitative discrepancies in reward margins.
  arXiv  Detail & Related papers  (2025-11-14T06:12:16Z)
• Margin Adaptive DPO: Leveraging Reward Model for Granular Control in Preference Optimization [0.0]
  Margin-Adaptive Direct Preference Optimization provides a stable, data-preserving, and instance-level solution.<n>We provide a comprehensive theoretical analysis, proving that MADPO has a well-behaved optimization landscape.<n>It achieves performance gains of up to +33.3% on High Quality data and +10.5% on Low Quality data over the next-best method.
  arXiv  Detail & Related papers  (2025-10-06T20:09:37Z)
• Alignment through Meta-Weighted Online Sampling: Bridging the Gap between Data Generation and Preference Optimization [32.84452172765487]
  Preference optimization is crucial for aligning large language models with human values and intentions.<n>Existing methods attempt to reduce this gap using statics or decoupled online sampling strategies.<n>We propose Meta-Weighted Adaptive Preference Optimization (MetaAPO), a novel framework that dynamically couples data generation with model training.
  arXiv  Detail & Related papers  (2025-09-27T15:38:24Z)
• Beyond Single: A Data Selection Principle for LLM Alignment via Fine-Grained Preference Signals [46.58760908162995]
  We propose a novel, theoretically-grounded data selection principle for large language models.<n>We prove the optimality of this strategy by analyzing the loss bounds of the Direct Preference Optimization objective.<n>Our strategy achieves over 10% relative improvement against both the standard holistic preference and a stronger oracle.
  arXiv  Detail & Related papers  (2025-08-11T05:43:02Z)
• On Symmetric Losses for Robust Policy Optimization with Noisy Preferences [55.8615920580824]
  This work focuses on reward modeling, a core component in reinforcement learning from human feedback.<n>We propose a principled framework for robust policy optimization under noisy preferences.<n>We prove that symmetric losses enable successful policy optimization even under noisy labels.
  arXiv  Detail & Related papers  (2025-05-30T15:30:43Z)
• Less is More: Improving LLM Alignment via Preference Data Selection [46.9163802899686]
  Direct Preference Optimization (DPO) has emerged as a promising approach for aligning large language models with human preferences.<n>We propose a novel margin-maximization principle for dataset curation in DPO training.<n>We show that our approach seamlessly extends to iterative DPO, yielding a roughly 3% improvement with 25% online data.
  arXiv  Detail & Related papers  (2025-02-20T13:45:17Z)
• Reward-Augmented Data Enhances Direct Preference Alignment of LLMs [63.32585910975191]
  We introduce reward-conditioned Large Language Models (LLMs) that learn from the entire spectrum of response quality within the dataset.<n>We show that our approach consistently boosts DPO by a considerable margin.<n>Our method not only maximizes the utility of preference data but also mitigates the issue of unlearning, demonstrating its broad effectiveness beyond mere data expansion.
  arXiv  Detail & Related papers  (2024-10-10T16:01:51Z)
• Adaptive Preference Scaling for Reinforcement Learning with Human Feedback [103.36048042664768]
  Reinforcement learning from human feedback (RLHF) is a prevalent approach to align AI systems with human values. We propose a novel adaptive preference loss, underpinned by distributionally robust optimization (DRO) Our method is versatile and can be readily adapted to various preference optimization frameworks.
  arXiv  Detail & Related papers  (2024-06-04T20:33:22Z)
• Self-Augmented Preference Optimization: Off-Policy Paradigms for Language Model Alignment [104.18002641195442]
  We introduce Self-Augmented Preference Optimization (SAPO), an effective and scalable training paradigm that does not require existing paired data. Building on the self-play concept, which autonomously generates negative responses, we further incorporate an off-policy learning pipeline to enhance data exploration and exploitation.
  arXiv  Detail & Related papers  (2024-05-31T14:21:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.