Compositional Bias Control in Large Language Models: Preference Learning Fails, Supervision Succeeds

• URL: http://arxiv.org/abs/2510.22084v1
• Date: Fri, 24 Oct 2025 23:52:37 GMT
• Title: Compositional Bias Control in Large Language Models: Preference Learning Fails, Supervision Succeeds
• Abstract summary: Large Language Models (LLMs) still produce gender-stereotyped language even in occupation-neutral contexts.<n>We compare six control techniques for bias mitigation: prompt-only, generate-and-filter, DFA-based Ctrl-G decoding, Supervised Fine-Tuning (SFT), Direct Preference Optimization (DPO), and Iterative Nullspace Projection (INLP)<n>SFT achieves 99.87 +- 0.15% compliance and high lexical diversity, while DPO, despite similar training stability, fails at 4.53 +- 0.82%.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large Language Models (LLMs) still produce gender-stereotyped language even in occupation-neutral contexts that reflect deep societal biases (Rudinger et al., 2018). To address this, prior work has proposed prompting, constrained decoding (Dathathri et al., 2020; Zhou et al., 2024), post-processing, and fine-tuning-based alignment (Rafailov et al., 2023; Ravfogel et al., 2022). However, the comparative efficacy and learning dynamics remain little understood. We report a comparative analysis of six control techniques for bias mitigation: prompt-only, generate-and-filter, DFA-based Ctrl-G decoding, Supervised Fine-Tuning (SFT), Direct Preference Optimization (DPO), and Iterative Nullspace Projection (INLP). We evaluate each method on a compositional constraint task. This task requires generating sentences that contain at least one agentic and one communal descriptor for each of the twenty Winogender-derived occupations. We quantify trade-offs between control strength and naturalness with evaluations of constraint compliance, lexical diversity, and fluency. Our results reveal key contrasts among the methods: SFT achieves 99.87 +- 0.15% compliance and high lexical diversity, while DPO, despite similar training stability, fails at 4.53 +- 0.82%. Ctrl-G guarantees perfect compliance, but at the cost of severely reduced fluency and diversity. Preference-based learning fundamentally differs: it cannot satisfy compositional constraints, as binary preference signals encode ranking, not logical conjunctions. Only explicit positive supervision enables mitigation of compositional biases; preference-based alignment fails to generalize logical structures, underscoring the limitations of preference learning and the necessity of explicit supervision for fair and fluent controlled generation.

Related papers

• Do LLMs Have Values? A Quantitative Analysis and Alignment Framework for Values in Large Language Models [49.45276299939287]
  Large Language Models (LLMs) increasingly handle complex subjective tasks.<n>LLMs fluctuate unpredictably under minor wording changes.<n> resolving this duality is critical for reliable AI alignment.
  arXiv  Detail & Related papers  (2026-09-15T03:35:04Z)
• Inference-Time Steering for Cross-Lingual Factual Consistency in LLMs [51.56484100374058]
  Large Language Models (LLMs) demonstrate remarkable multilingual fluency, but their internal knowledge representations remain disproportionately biased toward high-resource languages.<n>We investigate whether these biases can be mitigated at inference time, forcing an English-prompted model to answer as if it were queried in target languages.<n>Experiments on Gemma 3 12B Instruct reveal persona prompting to be the strongest overall intervention, balancing efficacy, safety, and out-of-domain generalization.
  arXiv  Detail & Related papers  (2026-07-21T16:15:05Z)
• Ensemble Diversity Optimization for Subjective Supervision [4.032367157209129]
  We introduce Ensemble Diversity Optimization (EDO), a prediction-space framework that jointly optimize ensemble weights, effective cardinality, and calibration.<n>EDO learns ensemble composition and size end-to-end via Gumbel-Softmax relaxation and incorporates a signed diversity regularizer.
  arXiv  Detail & Related papers  (2026-07-09T13:53:02Z)
• Too Correct to Learn: Reinforcement Learning on Saturated Reasoning Data [55.84428098924793]
  We propose Constrained Uniform Top-K Sampling (CUTS), a parameter-free decoding strategy enforcing structure-preserving exploration.<n>We integrate this into Mixed-CUTS, a training framework synergizing exploitative and exploratory rollouts to amplify intra-group advantage variance.<n> Notably, Mixed-CUTS improves Pass@1 accuracy on the challenging AIME25 benchmark by up to 15.1% over standard GRPO.
  arXiv  Detail & Related papers  (2026-04-20T16:43:28Z)
• STaR-DRO: Stateful Tsallis Reweighting for Group-Robust Structured Prediction [0.20165644958637055]
  We present a two-part framework for controllable inference and robust fine-tuning.<n>First, we introduce a task-agnostic prompting strategy to address format drift, label ambiguity, evidence hallucination, and metadata-conditioned confusion.<n>Second, we introduce STaR-DRO, a stateful robust optimization method for group heterogeneity.
  arXiv  Detail & Related papers  (2026-04-09T21:58:42Z)
• Distributional Open-Ended Evaluation of LLM Cultural Value Alignment Based on Value Codebook [50.10594064510559]
  DOVE is a distributional evaluation framework that directly compares human-written text with LLM-generated outputs.<n>We show that DOVE achieves superior predictive validity, attaining a 31.56% correlation with downstream tasks, while maintaining high reliability with as few as 500 samples per culture.
  arXiv  Detail & Related papers  (2026-03-16T08:33:10Z)
• BROTHER: Behavioral Recognition Optimized Through Heterogeneous Ensemble Regularization for Ambivalence and Hesitancy [39.407739937584104]
  Ambivalence and Hesitancy (A/H) in naturalistic video settings remains a significant challenge in affective computing.<n>We propose a highly regularized, multimodal fusion pipeline to predict A/H at the video level.
  arXiv  Detail & Related papers  (2026-03-15T12:53:41Z)
• Optimistic Feasible Search for Closed-Loop Fair Threshold Decision-Making [0.0]
  We study online learning of a one-dimensional threshold policy from bandit feedback.<n>We propose Optimistic Feasible Search (OFS), a simple grid-based method that maintains confidence bounds for reward and constraint residuals.
  arXiv  Detail & Related papers  (2025-12-26T10:44:40Z)
• When in Doubt, Deliberate: Confidence-Based Routing to Expert Debate for Sexism Detection [7.299050989302629]
  We propose a framework to address the combined effects of (i) underrepresentation, (ii) noise, and (iii) conceptual ambiguity in both data and model predictions.<n>Our approach achieves state-of-the-art results across several benchmarks, with a +2.72% improvement in F1 on Tasks EXIST 2025 Task 1.1, and a gains of +4.48% and +1.30% on EDOS A and B, respectively.
  arXiv  Detail & Related papers  (2025-12-21T05:48:57Z)
• From Literal to Liberal: A Meta-Prompting Framework for Eliciting Human-Aligned Exception Handling in Large Language Models [0.3946915822335988]
  Large Language Models (LLMs) are increasingly being deployed as the reasoning engines for agentic AI systems.<n>They exhibit a critical flaw: a rigid adherence to explicit rules that leads to decisions misaligned with human common sense and intent.<n>We introduce the Rule-Intent Distinction (RID) Framework, which elicits human-aligned exception handling in LLMs in a zero-shot manner.
  arXiv  Detail & Related papers  (2025-10-14T16:42:52Z)
• UniAPL: A Unified Adversarial Preference Learning Framework for Instruct-Following [12.924923059340395]
  We argue that post-training alignment is fundamentally a unified Preference Learning problem.<n>UniAPL implements a single-stage unified training objective, jointly learning from mixed batches of SFT and preference data.
  arXiv  Detail & Related papers  (2025-09-29T17:53:09Z)
• Self-Error Adjustment: Theory and Practice of Balancing Individual Performance and Diversity in Ensemble Learning [4.114446284816968]
  Ensemble learning boosts performance by aggregating predictions from multiple base learners.<n>Traditional methods like Bagging and Boosting promote diversity through randomness but lack precise control over the accuracy-diversity trade-off.<n>We propose a novel framework called Self-Error Adjustment (SEA), which decomposes ensemble errors into two distinct components.
  arXiv  Detail & Related papers  (2025-08-07T00:30:26Z)
• NDCG-Consistent Softmax Approximation with Accelerated Convergence [67.10365329542365]
  We propose novel loss formulations that align directly with ranking metrics.<n>We integrate the proposed RG losses with the highly efficient Alternating Least Squares (ALS) optimization method.<n> Empirical evaluations on real-world datasets demonstrate that our approach achieves comparable or superior ranking performance.
  arXiv  Detail & Related papers  (2025-06-11T06:59:17Z)
• Variational Supervised Contrastive Learning [50.79938854370321]
  We propose Variational Supervised Contrastive Learning (VarCon), which reformulates supervised contrastive learning as variational inference over latent class variables.<n>VarCon achieves state-of-the-art performance for contrastive learning frameworks, reaching 79.36% Top-1 accuracy on ImageNet-1K and 78.29% on CIFAR-100 with a ResNet-50 encoder.
  arXiv  Detail & Related papers  (2025-06-09T04:19:12Z)
• T2I-Eval-R1: Reinforcement Learning-Driven Reasoning for Interpretable Text-to-Image Evaluation [60.620408007636016]
  We propose T2I-Eval-R1, a novel reinforcement learning framework that trains open-source MLLMs using only coarse-grained quality scores.<n>Our approach integrates Group Relative Policy Optimization into the instruction-tuning process, enabling models to generate both scalar scores and interpretable reasoning chains.
  arXiv  Detail & Related papers  (2025-05-23T13:44:59Z)
• Discriminative Finetuning of Generative Large Language Models without Reward Models and Human Preference Data [73.04828796123581]
  Supervised fine-tuning (SFT) has become a crucial step for aligning pretrained large language models (LLMs)<n>We introduce Discriminative Fine-Tuning (DFT), an improved variant of SFT, which mitigates the burden of collecting human-labeled preference data.<n>Our contributions include: (i) a discriminative probabilistic framework for fine-tuning LLMs by explicitly modeling the discriminative likelihood of an answer among all possible outputs given an input; (ii) efficient algorithms to optimize this discriminative likelihood; and (iii) extensive experiments demonstrating DFT's effectiveness
  arXiv  Detail & Related papers  (2025-02-25T22:38:55Z)
• The Root Shapes the Fruit: On the Persistence of Gender-Exclusive Harms in Aligned Language Models [91.86718720024825]
  We center transgender, nonbinary, and other gender-diverse identities to investigate how alignment procedures interact with pre-existing gender-diverse bias.<n>Our findings reveal that DPO-aligned models are particularly sensitive to supervised finetuning.<n>We conclude with recommendations tailored to DPO and broader alignment practices.
  arXiv  Detail & Related papers  (2024-11-06T06:50:50Z)
• Optimizing Class-Level Probability Reweighting Coefficients for Equitable Prompting Accuracy [12.287692969438169]
  LLMs often uncover biases from pre-training data's statistical regularities.<n>This leads to persistent, uneven class accuracy in classification and QA.<n>We develop a post-hoc probability reweighting method that directly optimize for non-differentiable performance-driven metrics.
  arXiv  Detail & Related papers  (2024-05-13T10:30:33Z)
• An Information Bottleneck Approach for Controlling Conciseness in Rationale Extraction [84.49035467829819]
  We show that it is possible to better manage this trade-off by optimizing a bound on the Information Bottleneck (IB) objective. Our fully unsupervised approach jointly learns an explainer that predicts sparse binary masks over sentences, and an end-task predictor that considers only the extracted rationale.
  arXiv  Detail & Related papers  (2020-05-01T23:26:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.