SPA: Achieving Consensus in LLM Alignment via Self-Priority Optimization

• URL: http://arxiv.org/abs/2511.06222v1
• Date: Sun, 09 Nov 2025 04:43:32 GMT
• Title: SPA: Achieving Consensus in LLM Alignment via Self-Priority Optimization
• Abstract summary: In high-stakes scenarios-such as self-harm, legal, or medical queries-LLMs must be both trustworthy and helpful.<n>We propose priority alignment, a new alignment paradigm that enforces a strict "trustworthy-before-helpful" ordering.
• Score: 34.18333262699658
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: In high-stakes scenarios-such as self-harm, legal, or medical queries-LLMs must be both trustworthy and helpful. However, these goals often conflict. We propose priority alignment, a new alignment paradigm that enforces a strict "trustworthy-before-helpful" ordering: optimization of helpfulness is conditioned on first meeting trustworthy thresholds (e.g., harmlessness or honesty). To realize this, we introduce Self-Priority Alignment (SPA)-a fully unsupervised framework that generates diverse responses, self-evaluates them and refines them by the model itself, and applies dual-criterion denoising to remove inconsistency and control variance. From this, SPA constructs lexicographically ordered preference pairs and fine-tunes the model using an uncertainty-weighted alignment loss that emphasizes high-confidence, high-gap decisions. Experiments across multiple benchmarks show that SPA improves helpfulness without compromising safety, outperforming strong baselines while preserving general capabilities. Our results demonstrate that SPA provides a scalable and interpretable alignment strategy for critical LLM applications.

Related papers

• Confidence-Aware Alignment Makes Reasoning LLMs More Reliable [65.44962502963378]
  We introduce CASPO, a framework that aligns token-level confidence with step-wise logical correctness without training a separate reward model.<n>During inference, we propose Confidence-aware Thought (CaT), which dynamically prunes uncertain reasoning branches with negligible O(V) latency.<n>Experiments across ten benchmarks and multiple model families show that CASPO consistently improves reasoning reliability and inference efficiency.
  arXiv  Detail & Related papers  (2026-05-08T07:08:25Z)
• BAS: A Decision-Theoretic Approach to Evaluating Large Language Model Confidence [22.382291859991472]
  We introduce the Behavioral Alignment Score (BAS), a decision-theoretic metric for evaluating how well large language models support abstention-aware decision making.<n>BAS is derived from an explicit answer-or-abstain utility model and aggregates realized utility across a continuum of risk thresholds.<n>We show theoretically that truthful confidence estimates uniquely maximize expected BAS utility, linking calibration to decision-optimal behavior.
  arXiv  Detail & Related papers  (2026-04-03T17:44:32Z)
• Confidence Before Answering: A Paradigm Shift for Efficient LLM Uncertainty Estimation [47.91529693614168]
  Existing methods are predominantly answer-first, producing confidence only after generating an answer.<n>We study a confidence-first paradigm, where the model outputs its confidence before answering, interpreting this score as the model's probability of answering the question correctly.<n>We propose CoCA, a reinforcement learning framework that jointly optimize confidence calibration and answer accuracy via segmented credit assignment.
  arXiv  Detail & Related papers  (2026-03-06T04:03:13Z)
• Alignment-Weighted DPO: A principled reasoning approach to improve safety alignment [13.463606100715504]
  Large language models are vulnerable to attacks that disguise harmful intent.<n>This vulnerability stems from shallow alignment mechanisms that lack deep reasoning.<n>We propose enhancing alignment through reasoning-aware post-training.
  arXiv  Detail & Related papers  (2026-02-24T20:30:51Z)
• Unifying Stable Optimization and Reference Regularization in RLHF [64.16830602324345]
  This paper introduces a unified regularization approach that balances objectives of preventing reward hacking and maintaining stable policy updates.<n>Our simple yet principled alignment objective yields a weighted supervised fine-tuning loss with a superior trade-off, which demonstrably improves both alignment results and implementation complexity.
  arXiv  Detail & Related papers  (2026-02-12T03:31:19Z)
• SLIME: Stabilized Likelihood Implicit Margin Enforcement for Preference Optimization [0.0]
  We introduce SLIME, a reference-free alignment objective designed to decouple preference learning from generation quality.<n>Our results demonstrate that SLIME achieves superior performance compared to state-of-the-art baselines.
  arXiv  Detail & Related papers  (2026-02-02T17:46:06Z)
• Conformal Feedback Alignment: Quantifying Answer-Level Reliability for Robust LLM Alignment [5.556653374703646]
  Conformal Feedback Alignment (CFA) is a framework that grounds preference weighting in the statistical guarantees of Conformal Prediction (CP)<n>CFA quantifies answer-level reliability by constructing conformal prediction sets with controllable coverage and aggregates these reliabilities into principled weights for both DPO- and PPO-style training.
  arXiv  Detail & Related papers  (2026-01-24T06:23:18Z)
• Reasoning over Precedents Alongside Statutes: Case-Augmented Deliberative Alignment for LLM Safety [59.01189713115365]
  We evaluate the impact of explicitly specifying extensive safety codes versus demonstrating them through illustrative cases.<n>We find that referencing explicit codes inconsistently improves harmlessness and systematically degrades helpfulness.<n>We propose CADA, a case-augmented deliberative alignment method for LLMs utilizing reinforcement learning on self-generated safety reasoning chains.
  arXiv  Detail & Related papers  (2026-01-12T21:08:46Z)
• ConfPO: Exploiting Policy Model Confidence for Critical Token Selection in Preference Optimization [48.50761200321113]
  We introduce ConfPO, a method for preference learning in Large Language Models (LLMs)<n>It identifies and optimize preference-critical tokens based solely on the training policy's confidence, without requiring any auxiliary models or compute.<n> Experimental results on challenging alignment benchmarks, including AlpacaEval 2 and Arena-Hard, demonstrate that ConfPO consistently outperforms uniform DAAs.
  arXiv  Detail & Related papers  (2025-06-10T11:54:22Z)
• Bounded Rationality for LLMs: Satisficing Alignment at Inference-Time [52.230936493691985]
  We propose SITAlign, an inference framework that addresses the multifaceted nature of alignment by maximizing a primary objective while satisfying threshold-based constraints on secondary criteria.<n>We provide theoretical insights by deriving sub-optimality bounds of our satisficing based inference alignment approach.
  arXiv  Detail & Related papers  (2025-05-29T17:56:05Z)
• Data-Driven Calibration of Prediction Sets in Large Vision-Language Models Based on Inductive Conformal Prediction [0.0]
  We propose a model-agnostic uncertainty quantification method that integrates dynamic threshold calibration and cross-modal consistency verification.<n>We show that the framework achieves stable performance across varying calibration-to-test split ratios, underscoring its robustness for real-world deployment in healthcare, autonomous systems, and other safety-sensitive domains.<n>This work bridges the gap between theoretical reliability and practical applicability in multi-modal AI systems, offering a scalable solution for hallucination detection and uncertainty-aware decision-making.
  arXiv  Detail & Related papers  (2025-04-24T15:39:46Z)
• Aligning Large Language Models for Faithful Integrity Against Opposing Argument [71.33552795870544]
  Large Language Models (LLMs) have demonstrated impressive capabilities in complex reasoning tasks.<n>They can be easily misled by unfaithful arguments during conversations, even when their original statements are correct.<n>We propose a novel framework, named Alignment for Faithful Integrity with Confidence Estimation.
  arXiv  Detail & Related papers  (2025-01-02T16:38:21Z)
• Deliberative Alignment: Reasoning Enables Safer Language Models [64.60765108418062]
  We introduce Deliberative Alignment, a new paradigm that teaches the model safety specifications and trains it to explicitly recall and accurately reason over the specifications before answering.<n>We used this approach to align OpenAI's o-series models, and achieved highly precise adherence to OpenAI's safety policies, without requiring human-written chain-of-thoughts or answers.
  arXiv  Detail & Related papers  (2024-12-20T21:00:11Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.