Rethinking the Evaluation of Alignment Methods: Insights into Diversity, Generalisation, and Safety

• URL: http://arxiv.org/abs/2509.12936v1
• Date: Tue, 16 Sep 2025 10:32:59 GMT
• Title: Rethinking the Evaluation of Alignment Methods: Insights into Diversity, Generalisation, and Safety
• Abstract summary: We show that DPO and KTO excel in factual accuracy, PPO and DPO lead in safety, and PPO best balances conciseness with proactivity.<n>Our findings provide insights into trade-offs of common alignment methods, guiding the development of more balanced and reliable large language models.
• Score: 1.0194024892377178
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Large language models (LLMs) require careful alignment to balance competing objectives - factuality, safety, conciseness, proactivity, and diversity. Existing studies focus on individual techniques or specific dimensions, lacking a holistic assessment of the inherent trade-offs. We propose a unified evaluation framework that compares LLM alignment methods (PPO, DPO, ORPO, KTO) across these five axes, using both in-distribution and out-of-distribution datasets. Leveraging a specialized LLM-as-Judge prompt, validated through human studies, we reveal that DPO and KTO excel in factual accuracy, PPO and DPO lead in safety, and PPO best balances conciseness with proactivity. Our findings provide insights into trade-offs of common alignment methods, guiding the development of more balanced and reliable LLMs.

Related papers

• Aligning Language Models with Selective Prediction [17.07077522375248]
  Large language models (LLMs) are increasingly deployed as critical decision-making components in high-stakes real-world AI systems.<n>In this paper, we focus on enhancing LLM reliability through selective prediction (SP)<n>We propose a novel alignment framework, Reinforcement Learning for Selection Reward (RLSR), which targets the area under the risk-coverage curve (AURC) as its alignment objective.
  arXiv  Detail & Related papers  (2026-07-03T17:59:02Z)
• Bounded Ratio Reinforcement Learning [62.09379476369526]
  This paper introduces the Bounded Ratio Reinforcement Learning (BRRL) framework to solve on-policy reinforcement learning problems.<n>We develop a policy optimization algorithm called Bounded Policy Optimization (BPO) that minimizes an advantage-weighted divergence between the policy and the analytic optimal solution from BRRL.<n> Notably, our framework also provides a new theoretical lens to interpret the success of the PPO loss, and connects trust region policy optimization and the Cross-Entropy Method (CEM).
  arXiv  Detail & Related papers  (2026-04-20T17:59:01Z)
• What Is Preference Optimization Doing, How and Why? [73.78865043839939]
  A common belief is that DPO is supervised learning while PPO is reinforcement learning, yet deeper analyses for the reasons underlying these differences remain lacking.<n>First, we examine the target directions of gradient-based updates and find that DPO follows stable targets, whereas PPO follows dynamic targets that balance exploration and exploitation.<n>Second, we examine the roles of positive learning, negative learning, and loss reweighting, which are three key components in PO methods.
  arXiv  Detail & Related papers  (2025-11-30T08:27:59Z)
• Systematic Evaluation of Uncertainty Estimation Methods in Large Language Models [1.8374839804848957]
  We evaluate four approaches for confidence estimation in large language models (LLMs)<n>We conduct experiments on four question-answering tasks using a state-of-the-art open-source LLM.<n>Our results show that each uncertainty metric captures a different facet of model confidence and that the hybrid CoCoA approach yields the best reliability overall.
  arXiv  Detail & Related papers  (2025-10-23T11:50:47Z)
• Unveiling Trust in Multimodal Large Language Models: Evaluation, Analysis, and Mitigation [51.19622266249408]
  MultiTrust-X is a benchmark for evaluating, analyzing, and mitigating the trustworthiness issues of MLLMs.<n>Based on the taxonomy, MultiTrust-X includes 32 tasks and 28 curated datasets.<n>Our experiments reveal significant vulnerabilities in current models.
  arXiv  Detail & Related papers  (2025-08-21T09:00:01Z)
• A Comprehensive Evaluation framework of Alignment Techniques for LLMs [5.9090038202345]
  This paper introduces a multi-dimensional evaluation of alignment techniques for Large Language Models (LLMs)<n>Our framework assesses methods along four key dimensions: alignment detection, alignment quality, computational efficiency, and robustness.
  arXiv  Detail & Related papers  (2025-08-13T16:42:01Z)
• Alignment and Safety in Large Language Models: Safety Mechanisms, Training Paradigms, and Emerging Challenges [47.14342587731284]
  This survey provides a comprehensive overview of alignment techniques, training protocols, and empirical findings in large language models (LLMs) alignment.<n>We analyze the development of alignment methods across diverse paradigms, characterizing the fundamental trade-offs between core alignment objectives.<n>We discuss state-of-the-art techniques, including Direct Preference Optimization (DPO), Constitutional AI, brain-inspired methods, and alignment uncertainty quantification (AUQ)
  arXiv  Detail & Related papers  (2025-07-25T20:52:58Z)
• The Hidden Link Between RLHF and Contrastive Learning [56.45346439723488]
  We show that Reinforcement Learning from Human Feedback (RLHF) and the simple Direct Preference Optimization (DPO) can be interpreted from the perspective of mutual information (MI)<n>Within this framework, both RLHF and DPO can be interpreted as methods that performing contrastive learning based on the positive and negative samples derived from base model.<n>We propose the Mutual Information Optimization (MIO) to mitigate the late-stage decline in chosen-likelihood observed in DPO.
  arXiv  Detail & Related papers  (2025-06-27T18:51:25Z)
• The Scales of Justitia: A Comprehensive Survey on Safety Evaluation of LLMs [57.1838332916627]
  Large Language Models (LLMs) have shown remarkable capabilities in Natural Language Processing (NLP)<n>Their widespread deployment has also raised significant safety concerns.<n>LLMs-generated content can exhibit unsafe behaviors such as toxicity, bias, or misinformation, especially in adversarial contexts.
  arXiv  Detail & Related papers  (2025-06-06T05:50:50Z)
• Beyond Single-Point Judgment: Distribution Alignment for LLM-as-a-Judge [24.862965044243168]
  Previous methods rely on single-point evaluations, overlooking the inherent diversity and uncertainty in human evaluations.<n>We propose a novel training framework that explicitly aligns the LLM-generated judgment distribution with empirical human distributions.<n>Our framework significantly outperforms existing closed-source LLMs and conventional single-point alignment methods.
  arXiv  Detail & Related papers  (2025-05-18T08:33:09Z)
• Value Compass Benchmarks: A Platform for Fundamental and Validated Evaluation of LLMs Values [76.70893269183684]
  Large Language Models (LLMs) achieve remarkable breakthroughs.<n> aligning their values with humans has become imperative for their responsible development.<n>There still lack evaluations of LLMs values that fulfill three desirable goals.
  arXiv  Detail & Related papers  (2025-01-13T05:53:56Z)
• Evaluating and Advancing Multimodal Large Language Models in Perception Ability Lens [30.083110119139793]
  We introduce textbfAbilityLens, a unified benchmark designed to evaluate MLLMs in six key perception abilities.<n>We identify the strengths and weaknesses of current main-stream MLLMs, highlighting stability patterns and revealing a notable performance gap between state-of-the-art open-source and closed-source models.
  arXiv  Detail & Related papers  (2024-11-22T04:41:20Z)
• Bi-Factorial Preference Optimization: Balancing Safety-Helpfulness in Language Models [94.39278422567955]
  Fine-tuning large language models (LLMs) on human preferences has proven successful in enhancing their capabilities.<n>However, ensuring the safety of LLMs during fine-tuning remains a critical concern.<n>We propose a supervised learning framework called Bi-Factorial Preference Optimization (BFPO) to address this issue.
  arXiv  Detail & Related papers  (2024-08-27T17:31:21Z)
• Trustworthy LLMs: a Survey and Guideline for Evaluating Large Language Models' Alignment [35.42539816648068]
  This paper presents a comprehensive survey of key dimensions that are crucial to consider when assessing large language models (LLMs) The survey covers seven major categories of LLM trustworthiness: reliability, safety, fairness, resistance to misuse, explainability and reasoning, adherence to social norms, and robustness. Results indicate that, in general, more aligned models tend to perform better in terms of overall trustworthiness.
  arXiv  Detail & Related papers  (2023-08-10T06:43:44Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.