When Weak LLMs Speak with Confidence, Preference Alignment Gets Stronger

• URL: http://arxiv.org/abs/2603.04968v1
• Date: Thu, 05 Mar 2026 09:06:25 GMT
• Title: When Weak LLMs Speak with Confidence, Preference Alignment Gets Stronger
• Authors: Amirabbas Afzali, Myeongho Jeon, Maria Brbic,
• Abstract summary: Preference alignment is an essential step in adapting large language models to human values.<n>We propose Confidence-Weighted Preference Optimization (CW-PO), a framework that re-weights training samples by a weak LLM's confidence.<n>CW-PO with just 20% of human annotations outperforms the model trained with 100% of annotations under standard DPO.
• Score: 12.541521203916867
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Preference alignment is an essential step in adapting large language models (LLMs) to human values, but existing approaches typically depend on costly human annotations or large-scale API-based models. We explore whether a weak LLM can instead act as an effective annotator. We surprisingly find that selecting only a subset of a weak LLM's highly confident samples leads to substantially better performance than using full human annotations. Building on this insight, we propose Confidence-Weighted Preference Optimization (CW-PO), a general framework that re-weights training samples by a weak LLM's confidence and can be applied across different preference optimization objectives. Notably, the model aligned by CW-PO with just 20% of human annotations outperforms the model trained with 100% of annotations under standard DPO. These results suggest that weak LLMs, when paired with confidence weighting, can dramatically reduce the cost of preference alignment while even outperforming methods trained on fully human-labeled data.

Related papers

• Selective LLM-Guided Regularization for Enhancing Recommendation Models [7.406718588794206]
  We introduce a model-agnostic and efficient framework that activates LLM based pairwise ranking supervision only when a trainable gating mechanism informs by user history length, item popularity, and model uncertainty predicts the LLM to be reliable.<n> Experiments across multiple datasets show that this selective strategy consistently improves overall accuracy and yields substantial gains in cold start and long tail regimes, outperforming global distillation baselines.
  arXiv  Detail & Related papers  (2025-12-25T06:30:00Z)
• IPO: Your Language Model is Secretly a Preference Classifier [1.8921784053120494]
  Reinforcement learning from human feedback (RLHF) has emerged as the primary method for aligning large language models with human preferences.<n>We propose Implicit Preference Optimization (IPO), an alternative approach that leverages generative language models as preference classifiers.<n>Our findings demonstrate that models trained through IPO achieve performance comparable to those utilizing state-of-the-art reward models for obtaining preferences.
  arXiv  Detail & Related papers  (2025-02-22T10:59:11Z)
• Improve LLM-as-a-Judge Ability as a General Ability [40.288846894608966]
  Large language models (LLMs) can evaluate responses across diverse scenarios, providing accurate preference signals.<n>Recent studies have raised many methods to train LLM as generative judges, but most of them are data consuming or lack accuracy.<n>In this work, we implement a two-stage training approach, comprising supervised fine-tuning (SFT) warm-up and direct preference optimization (DPO) enhancement.
  arXiv  Detail & Related papers  (2025-02-17T11:28:43Z)
• A Little Help Goes a Long Way: Efficient LLM Training by Leveraging Small LMs [74.35290684163718]
  A primary challenge in large language model (LLM) development is their onerous pre-training cost. This paper explores a promising paradigm to improve LLM pre-training efficiency and quality by leveraging a small language model (SLM)
  arXiv  Detail & Related papers  (2024-10-24T14:31:52Z)
• Fairer Preferences Elicit Improved Human-Aligned Large Language Model Judgments [41.25558612970942]
  We show that large language models (LLMs) exhibit preference biases and worrying sensitivity to prompt designs. Motivated by this phenomenon, we propose an automatic Zero-shot Evaluation-oriented Prompt Optimization framework, ZEPO.
  arXiv  Detail & Related papers  (2024-06-17T09:48:53Z)
• Spread Preference Annotation: Direct Preference Judgment for Efficient LLM Alignment [72.99676237703099]
  We propose a new framework that boosts the alignment of large language models with human preferences.<n>Our key idea is leveraging the human prior knowledge within the small (seed) data.<n>We introduce a noise-aware preference learning algorithm to mitigate the risk of low quality within generated preference data.
  arXiv  Detail & Related papers  (2024-06-06T18:01:02Z)
• Multi-Reference Preference Optimization for Large Language Models [56.84730239046117]
  We introduce a novel closed-form formulation for direct preference optimization using multiple reference models. The resulting algorithm, Multi-Reference Preference Optimization (MRPO), leverages broader prior knowledge from diverse reference models. Our experiments demonstrate that LLMs finetuned with MRPO generalize better in various preference data, regardless of data scarcity or abundance.
  arXiv  Detail & Related papers  (2024-05-26T00:29:04Z)
• Dissecting Human and LLM Preferences [80.55271307662365]
  We find that humans are less sensitive to errors, favor responses that support their stances, and show clear dislike when models admit their limits. advanced LLMs like GPT-4-Turbo emphasize correctness, clarity, and harmlessness more. We show that preference-based evaluation can be intentionally manipulated.
  arXiv  Detail & Related papers  (2024-02-17T14:34:31Z)
• Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Models [52.98743860365194]
  We propose a new fine-tuning method called Self-Play fIne-tuNing (SPIN) At the heart of SPIN lies a self-play mechanism, where the LLM refines its capability by playing against instances of itself. This sheds light on the promise of self-play, enabling the achievement of human-level performance in LLMs without the need for expert opponents.
  arXiv  Detail & Related papers  (2024-01-02T18:53:13Z)
• On Learning to Summarize with Large Language Models as References [101.79795027550959]
  Large language models (LLMs) are favored by human annotators over the original reference summaries in commonly used summarization datasets. We study an LLM-as-reference learning setting for smaller text summarization models to investigate whether their performance can be substantially improved.
  arXiv  Detail & Related papers  (2023-05-23T16:56:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.