Towards Order Fairness: Mitigating LLMs Order Sensitivity through Dual Group Advantage Optimization

• URL: http://arxiv.org/abs/2605.11974v1
• Date: Tue, 12 May 2026 11:31:18 GMT
• Title: Towards Order Fairness: Mitigating LLMs Order Sensitivity through Dual Group Advantage Optimization
• Abstract summary: Large Language Models (LLMs) suffer from order bias, where their performance is affected by the arrangement order of input elements.<n>textbfDGAO aims to improve model accuracy and order stability simultaneously.
• Score: 20.259122922188126
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Large Language Models (LLMs) suffer from order bias, where their performance is affected by the arrangement order of input elements. This unfairness limits the model's applications in scenarios such as in-context learning and Retrieval-Augmented Generation (RAG). Recent studies attempt to obtain optimal or suboptimal arrangements based on statistical results or using dataset-based search, but these methods increase inference overhead while leaving the model's inherent order bias unresolved. Other studies mitigate order sensitivity through supervised fine-tuning using augmented training sets with multiple order variants, but often at the cost of accuracy, trapping the model in consistent yet incorrect hallucinations. In this paper, we propose \textbf{D}ual \textbf{G}roup \textbf{A}dvantage \textbf{O}ptimization (\textbf{DGAO}), which aims to improve model accuracy and order stability simultaneously. DGAO calculates and balances intra-group relative accuracy advantage and inter-group relative stability advantage, rewarding the policy model for generating order-stable and correct outputs while penalizing order-sensitive or incorrect responses. This marks the first time reinforcement learning has been used to mitigate LLMs' order sensitivity. We also propose two new metrics, Consistency Rate and Overconfidence Rate, to reveal the pseudo-stability of previous methods and guide more comprehensive evaluation. Extensive experiments demonstrate that DGAO achieves superior order fairness while improving performance on RAG, mathematical reasoning, and classification tasks. Our code is available at: https://github.com/Hyalinesky/DGAO.

Related papers

• MixSD: Mixed Contextual Self-Distillation for Knowledge Injection [29.7616760417696]
  Supervised fine-tuning (SFT) is widely used to inject new knowledge into language models.<n>We argue this forgetting arises because fine-tuning targets from humans or external systems diverge from the model's autoregressive distribution.<n>We propose MixSD, a simple external-teacher-free method for distribution-aligned knowledge injection.
  arXiv  Detail & Related papers  (2026-05-16T07:57:09Z)
• Beyond Static Best-of-N: Bayesian List-wise Alignment for LLM-based Recommendation [23.95623437107717]
  Large Language Models have revolutionized recommender systems (LLM4Rec) by leveraging their generative capabilities to model user preferences.<n>Existing LLM4Rec methods rely on token-level objectives, making it difficult to optimize list-level and non-differentiable metrics that define actual recommendation quality.<n>We propose BLADE (Bayesian List-wise Alignment via Dynamic Estimation) to overcome these challenges.
  arXiv  Detail & Related papers  (2026-05-06T07:02:57Z)
• Too Correct to Learn: Reinforcement Learning on Saturated Reasoning Data [55.84428098924793]
  We propose Constrained Uniform Top-K Sampling (CUTS), a parameter-free decoding strategy enforcing structure-preserving exploration.<n>We integrate this into Mixed-CUTS, a training framework synergizing exploitative and exploratory rollouts to amplify intra-group advantage variance.<n> Notably, Mixed-CUTS improves Pass@1 accuracy on the challenging AIME25 benchmark by up to 15.1% over standard GRPO.
  arXiv  Detail & Related papers  (2026-04-20T16:43:28Z)
• APAO: Adaptive Prefix-Aware Optimization for Generative Recommendation [26.371939617653084]
  Generative recommendation is an autoregressive generation process, predicting discrete tokens of the next item conditioned on user interaction histories.<n>Existing generative recommendation models are typically trained with token-level likelihood objectives, such as cross-entropy loss.<n>This leads to a training-inference inconsistency: standard training assumes ground-truth history is always available, ignoring the fact that beam search prunes low-probability branches during inference.
  arXiv  Detail & Related papers  (2026-03-03T08:29:15Z)
• CAIRO: Decoupling Order from Scale in Regression [13.755937210012883]
  We propose a framework that decouples regression into two distinct stages.<n>In the first stage, we learn a scoring function by minimizing a scale-invariant ranking loss.<n>In the second, we recover the target scale via isotonic regression.
  arXiv  Detail & Related papers  (2026-02-16T03:50:05Z)
• Not All Preferences Are Created Equal: Stability-Aware and Gradient-Efficient Alignment for Reasoning Models [52.48582333951919]
  We propose a dynamic framework designed to enhance alignment reliability by maximizing the Signal-to-Noise Ratio of policy updates.<n>SAGE (Stability-Aware Gradient Efficiency) integrates a coarse-grained curriculum mechanism that refreshes candidate pools based on model competence.<n> Experiments on multiple mathematical reasoning benchmarks demonstrate that SAGE significantly accelerates convergence and outperforms static baselines.
  arXiv  Detail & Related papers  (2026-02-01T12:56:10Z)
• Listwise Preference Diffusion Optimization for User Behavior Trajectories Prediction [41.53271688465831]
  We formulate User Behavior Trajectory Prediction (UBTP) as a new task setting that explicitly models long-term user preferences.<n>We introduce Listwise Preference Diffusion Optimization (LPDO), a diffusion-based training framework that directly optimize structured preferences over entire item sequences.<n>To rigorously evaluate multi-step prediction quality, we propose the task-specific metric Sequential Match (SeqMatch), which measures exact trajectory agreement, and adopt Perplexity (PPL), which assesses probabilistic fidelity.
  arXiv  Detail & Related papers  (2025-11-01T12:16:24Z)
• A Systematic Examination of Preference Learning through the Lens of Instruction-Following [83.71180850955679]
  We use a novel synthetic data generation pipeline to generate 48,000 instruction unique-following prompts.<n>With our synthetic prompts, we use two preference dataset curation methods - rejection sampling (RS) and Monte Carlo Tree Search (MCTS)<n>Experiments reveal that shared prefixes in preference pairs, as generated by MCTS, provide marginal but consistent improvements.<n>High-contrast preference pairs generally outperform low-contrast pairs; however, combining both often yields the best performance.
  arXiv  Detail & Related papers  (2024-12-18T15:38:39Z)
• Permutative Preference Alignment from Listwise Ranking of Human Judgments [40.23480751285947]
  We develop an end-to-end alignment algorithm by approximating NDCG with a differentiable surrogate loss.<n>We show that NDCG-based approaches improve ranking accuracy more effectively than B-T-based methods.
  arXiv  Detail & Related papers  (2024-10-06T03:49:28Z)
• Estimating the Hessian Matrix of Ranking Objectives for Stochastic Learning to Rank with Gradient Boosted Trees [63.18324983384337]
  We introduce the first learning to rank method for Gradient Boosted Decision Trees (GBDTs) Our main contribution is a novel estimator for the second-order derivatives, i.e., the Hessian matrix. We incorporate our estimator into the existing PL-Rank framework, which was originally designed for first-order derivatives only.
  arXiv  Detail & Related papers  (2024-04-18T13:53:32Z)
• Enhancing Few-shot NER with Prompt Ordering based Data Augmentation [59.69108119752584]
  We propose a Prompt Ordering based Data Augmentation (PODA) method to improve the training of unified autoregressive generation frameworks. Experimental results on three public NER datasets and further analyses demonstrate the effectiveness of our approach.
  arXiv  Detail & Related papers  (2023-05-19T16:25:43Z)
• Conditional Denoising Diffusion for Sequential Recommendation [62.127862728308045]
  Two prominent generative models, Generative Adversarial Networks (GANs) and Variational AutoEncoders (VAEs) GANs suffer from unstable optimization, while VAEs are prone to posterior collapse and over-smoothed generations. We present a conditional denoising diffusion model, which includes a sequence encoder, a cross-attentive denoising decoder, and a step-wise diffuser.
  arXiv  Detail & Related papers  (2023-04-22T15:32:59Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.