Enhancing Multilingual Counterfactual Generation through Alignment-as-Preference Optimization

• URL: http://arxiv.org/abs/2605.11632v1
• Date: Tue, 12 May 2026 06:56:18 GMT
• Title: Enhancing Multilingual Counterfactual Generation through Alignment-as-Preference Optimization
• Abstract summary: We introduce Macro, a preference alignment framework that applies Direct Preference Optimization to multilingual SCE generation.<n>We show that Macro improves validity by 12.55% on average over the chain-of-thought baseline without degrading minimality.<n>Our results highlight preference optimization as a promising direction for enhancing multilingual model explanations.
• Score: 14.36297254909447
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Self-generated counterfactual explanations (SCEs) are minimally modified inputs (minimality) generated by large language models (LLMs) that flip their own predictions (validity), offering a causally grounded approach to unraveling black-box LLM behavior. Yet extending them beyond English remains challenging: existing methods struggle to produce valid SCEs in non-dominant languages, and a persistent trade-off between validity and minimality undermines explanation quality. We introduce Macro, a preference alignment framework that applies Direct Preference Optimization (DPO) to multilingual SCE generation, using a composite scoring function to construct preference pairs that effectively translate the trade-off into measurable preference signals. Experiments across four LLMs and seven typologically diverse languages show that Macro improves validity by 12.55\% on average over the chain-of-thought baseline without degrading minimality, while avoiding the severe minimality violations of the translation-based baseline. Compared to supervised fine-tuning, Macro achieves superior performance on both metrics, confirming that explicit preference optimization is essential for balancing this trade-off. Further analyses reveal that Macro increases cross-lingual perturbation alignment and mitigates common generation errors. Our results highlight preference optimization as a promising direction for enhancing multilingual model explanations.

Related papers

• Multimodal Large Language Models with Adaptive Preference Optimization for Sequential Recommendation [60.33386541343322]
  We propose a Multimodal Large Language Models framework that integrates Hardness-aware and Noise-regularized preference optimization for Recommendation (HaNoRec)<n>Specifically, HaNoRec dynamically adjusts optimization weights based on both the estimated hardness of each training sample and the policy model's real-time responsiveness.
  arXiv  Detail & Related papers  (2025-11-24T04:10:46Z)
• AMaPO: Adaptive Margin-attached Preference Optimization for Language Model Alignment [25.526336903358757]
  offline preference optimization offers a simpler and more stable alternative to RLHF for aligning language models.<n>We propose Adaptive Margin-attached Preference Optimization (AMaPO), a simple yet principled algorithm.<n>AMaPO employs an instance-wise adaptive margin, refined by Z-normalization and exponential scaling, which dynamically reallocates learning effort by amplifying gradients for misranked samples and suppressing them for correct ones.
  arXiv  Detail & Related papers  (2025-11-12T14:51:59Z)
• CAPO: Confidence Aware Preference Optimization Learning for Multilingual Preferences [4.460583138505673]
  Preference optimization is used to align large language models with human preferences, typically by fine-tuning on ranked response pairs.<n>We propose Confidence-Aware Preference Optimization (CAPO), which replaces DPO's fixed treatment of preference pairs with a dynamic loss scaling mechanism.<n>CAPO enhances robustness to noisy or low-margin comparisons, typically encountered in multilingual text.
  arXiv  Detail & Related papers  (2025-11-10T23:28:12Z)
• Beyond Single-Reward: Multi-Pair, Multi-Perspective Preference Optimization for Machine Translation [44.04325848740683]
  We introduce M2PO: Multi-Pair, Multi-Perspective Preference Optimization.<n>Our framework integrates a multi-perspective reward engine that creates a more robust signal.<n>On challenging WMT21-22 benchmarks, M2PO substantially outperforms existing preference optimization methods.
  arXiv  Detail & Related papers  (2025-10-15T11:30:49Z)
• Mitigating Hallucination Through Theory-Consistent Symmetric Multimodal Preference Optimization [69.05600758833471]
  Direct Preference Optimization (DPO) has emerged as an effective approach for mitigating hallucination in Multimodal Large Language Models (MLLMs)<n>We propose a Symmetric Multimodal Preference Optimization (SymMPO) which conducts symmetric preference learning with direct preference supervision (i.e., response pairs)<n>In addition to conventional ordinal preference learning, SymMPO introduces a preference margin consistency loss to quantitatively regulate the preference gap between symmetric preference pairs.
  arXiv  Detail & Related papers  (2025-06-13T12:29:15Z)
• Leveraging Robust Optimization for LLM Alignment under Distribution Shifts [51.74394601039711]
  Preference alignment methods are increasingly critical for steering large language models to generate outputs consistent with human values.<n>We propose a novel distribution-aware optimization framework that improves preference alignment despite such shifts.
  arXiv  Detail & Related papers  (2025-04-08T09:14:38Z)
• Correcting the Mythos of KL-Regularization: Direct Alignment without Overoptimization via Chi-Squared Preference Optimization [78.82586283794886]
  $chi2$-Preference Optimization ($chi$PO) is an efficient offline alignment algorithm provably robust to overoptimization.<n>$chi$PO implements the principle of pessimism in the face of uncertainty via regularization.<n>$chi$PO's simplicity and strong guarantees make it the first practical and general-purpose offline alignment algorithm provably robust to overoptimization.
  arXiv  Detail & Related papers  (2024-07-18T11:08:40Z)
• Multi-Reference Preference Optimization for Large Language Models [56.84730239046117]
  We introduce a novel closed-form formulation for direct preference optimization using multiple reference models. The resulting algorithm, Multi-Reference Preference Optimization (MRPO), leverages broader prior knowledge from diverse reference models. Our experiments demonstrate that LLMs finetuned with MRPO generalize better in various preference data, regardless of data scarcity or abundance.
  arXiv  Detail & Related papers  (2024-05-26T00:29:04Z)
• Distributionally Robust Multilingual Machine Translation [94.51866646879337]
  We propose a new learning objective for Multilingual neural machine translation (MNMT) based on distributionally robust optimization. We show how to practically optimize this objective for large translation corpora using an iterated best response scheme. Our method consistently outperforms strong baseline methods in terms of average and per-language performance under both many-to-one and one-to-many translation settings.
  arXiv  Detail & Related papers  (2021-09-09T03:48:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.