The Alignment Tax: Response Homogenization in Aligned LLMs and Its Implications for Uncertainty Estimation

• URL: http://arxiv.org/abs/2603.24124v2
• Date: Fri, 27 Mar 2026 12:31:05 GMT
• Title: The Alignment Tax: Response Homogenization in Aligned LLMs and Its Implications for Uncertainty Estimation
• Abstract summary: RLHF-aligned language models exhibit response homogenization on TruthfulQA.<n>40-79% of questions produce a single semantic cluster across 10 i.i.d. samples.
• Score: 1.8345614451086532
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: RLHF-aligned language models exhibit response homogenization: on TruthfulQA (n=790), 40-79% of questions produce a single semantic cluster across 10 i.i.d. samples. On affected questions, sampling-based uncertainty methods have zero discriminative power (AUROC=0.500), while free token entropy retains signal (0.603). This alignment tax is task-dependent: on GSM8K (n=500), token entropy achieves 0.724 (Cohen's d=0.81). A base-vs-instruct ablation confirms the causal role of alignment: the base model shows 1.0% single-cluster rate vs. 28.5% for the instruct model (p < 10^{-6}). A training stage ablation (Base 0.0% -> SFT 1.5% -> DPO 4.0% SCR) localizes the cause to DPO, not SFT. Cross-family replication on four model families reveals alignment tax severity varies by family and scale. We validate across 22 experiments, 5 benchmarks, 4 model families, and 3 model scales (3B-14B), with Jaccard, embedding, and NLI-based baselines at three DeBERTa scales (all ~0.51 AUROC). Cross-embedder validation with two independent embedding families rules out coupling bias. Cross-dataset validation on WebQuestions (58.0% SCR) confirms generalization beyond TruthfulQA. The central finding -- response homogenization -- is implementation-independent and label-free. Motivated by this diagnosis, we explore a cheapest-first cascade (UCBD) over orthogonal uncertainty signals. Selective prediction raises GSM8K accuracy from 84.4% to 93.2% at 50% coverage; weakly dependent boundaries (|r| <= 0.12) enable 57% cost savings.

Related papers

• Beyond Local Accuracy: A Protocol-Level Identifiability Audit for Controlled LLM Reasoning Evaluation [6.047519836191728]
  We formalize a protocol-level identifiability audit over a finite behavioral policy class.<n>The audit requires zero model calls and resolves our diagnostic case.<n>This case shows how evaluation-design validity can be checked structurally before model inference.
  arXiv  Detail & Related papers  (2026-08-13T14:49:47Z)
• Distribution-First Population Simulation: Collapse, Calibration, and Recall in Non-WEIRD LLM Persona Modeling [0.0]
  Synthetic-population tools run every individual as an independent large language model (LLM) agent.<n>We show that this paradigm has a basic failure mode, and we set a distribution-first corrective against it.
  arXiv  Detail & Related papers  (2026-07-17T15:45:57Z)
• When Can Conformal Risk Control Certify LLM Outputs? Bounds, Impossibility, and Adaptation for Structured Generation [0.11280931253550518]
  Large language models (LLMs) deployed for structured generation (NER, extraction, QA, and classification) lack formal reliability guarantees.<n>We characterize when conformal risk control (CRC) can certify structured LLM outputs and when it provably cannot.
  arXiv  Detail & Related papers  (2026-06-27T19:25:07Z)
• Automated Proving of Shannon-Type Entropy Inequalities via Fine-Tuned Language Models and Guided Tree Search [50.16356451328644]
  Proving Shannon-type entropy inequalities is a fundamental task in information theory.<n>We investigate whether small-scale large language models, fine-tuned on atomic proof steps, can automate this process.<n>We show that GPT-5.5 solves 1.7% samples under zero-shot prompting while Psitip solves 33.3% samples.
  arXiv  Detail & Related papers  (2026-06-04T05:43:12Z)
• Output Type Before Quality: A Standards-Derived XAI Admissibility Rubric for Autonomous-Driving Safety [0.4958589793470846]
  Safety standards for ML-based autonomous driving specify the kind of evidence an assurance case must contain (directed cause-and-effect chains, quantified interventional effects, named root-cause variables)<n>Yet the XAI literature is organised by output type and technique family (saliency maps, feature attribution, counterfactuals, causal graphs, language traces)<n>We name this mismatch the evidence-type gap.<n>XAI method selection for safety assurance should be driven by lifecycle-stage evidence demand, not by method popularity.
  arXiv  Detail & Related papers  (2026-06-03T21:37:52Z)
• Amplifying, Not Learning: Fine-Tuned AI Text Detectors Amplify a Pretrained Direction [51.56484100374058]
  Text detectors amplify a pretrained typicality axis; they do not construct an AI-vs-human boundary.<n>On raw encoders before any task supervision, projecting onto centroid(AI)-centroid(HC3) achieves NYT-vs-HC3 AUROC 0.806/0.944/0.834 across three architectures.<n>On RoBERTa-base, raw projection exceeds fine-tuning; on RoBERTa-base, full fine-tuning reduces discrimination below raw on both fluent-formal populations tested.
  arXiv  Detail & Related papers  (2026-05-20T19:08:38Z)
• Distributional Energy-Based Models for Uncertainty-Aware Structured LLM Reasoning [40.342912574072024]
  Large Language Models produce structured outputs such as travel plans or code solutions.<n>Individual reasoning steps may appear correct while the output as a whole violates budgets, fails test cases, or contradicts earlier deductions.<n>We propose a learned quality scorer with deterministic analytical constraint penalties for verifying structured LLM outputs.
  arXiv  Detail & Related papers  (2026-05-15T17:08:27Z)
• PBT-Bench: Benchmarking AI Agents on Property-Based Testing [29.035258104995204]
  We introduce PBT-Bench, a benchmark of 100 curated property-based testing problems across 40 real Python libraries.<n>Each problem injects one or more semantic bugs (365 in total, mean 3.65 per problem) designed so that default-strategy random inputs almost never trigger them.<n>Bug recall under the PBT-guided prompt ranges from 42.1% to 83.4% across models; under the open-ended baseline, from 31.4% to 76.7%.
  arXiv  Detail & Related papers  (2026-05-13T18:01:05Z)
• The Autocorrelation Blind Spot: Why 42% of Turn-Level Findings in LLM Conversation Analysis May Be Spurious [35.76482964927589]
  We characterize the autocorrelation structure of 66 turn-level metrics across 202 multi-turn conversations.<n>We show that 42% of associations that appear significant under standard pooled testing fail to survive cluster-robust correction.<n>We present a two-stage correction framework combining Chelton (1983) effective degrees of freedom with conversation-level block bootstraps.
  arXiv  Detail & Related papers  (2026-04-15T20:54:39Z)
• The Verification Tax: Fundamental Limits of AI Auditing in the Rare-Error Regime [0.0]
  Most cited calibration result in deep learning -- post-temperature-scaling ECE of 0.012 on CIFAR-100 -- is below the statistical noise floor.<n>We prove this is not a failure of the experiment but a law: the minimax rate for estimating calibration error with model error rate epsilon is Theta((Lepsilon/m)2/3), and no estimator can beat it.
  arXiv  Detail & Related papers  (2026-04-14T16:48:24Z)
• Measuring Faithfulness Depends on How You Measure: Classifier Sensitivity in LLM Chain-of-Thought Evaluation [0.0]
  Recent work on chain-of-thought faithfulness reports single aggregate numbers.<n>This paper provides evidence that faithfulness is not an objective, measurable property of a model.
  arXiv  Detail & Related papers  (2026-03-20T17:48:43Z)
• Cross-Domain Uncertainty Quantification for Selective Prediction: A Comprehensive Bound Ablation with Transfer-Informed Betting [0.0]
  We present a comprehensive ablation of nine finite-sample bound families for selective prediction with risk control.<n>Our main theoretical contribution is Transfer-Informed Betting (TIB), which warm-starts the WSR wealth process using a source domain's risk profile.
  arXiv  Detail & Related papers  (2026-03-09T20:25:18Z)
• Black-Box Reliability Certification for AI Agents via Self-Consistency Sampling and Conformal Calibration [0.0]
  We answer with a reliability level -- a single number per system-task pair.<n>Self-consistency sampling reduces uncertainty exponentially.<n> conformal calibration guarantees correctness within 1/(n+1) of the target level.
  arXiv  Detail & Related papers  (2026-02-24T21:03:50Z)
• Replayable Financial Agents: A Determinism-Faithfulness Assurance Harness for Tool-Using LLM Agents [0.7699235580548228]
  LLM agents struggle with regulatory audit replay: when asked to reproduce a transaction flagged decision with identical inputs, most deployments fail to return consistent results.<n>This paper introduces the DeterminismFaithfulness Assurance Harness (DFAH), a framework for measuring trajectory determinism and evidence-conditioned faithfulness in tool-using agents deployed in financial services.
  arXiv  Detail & Related papers  (2026-01-17T19:47:55Z)
• Encyclo-K: Evaluating LLMs with Dynamically Composed Knowledge Statements [78.87065404966002]
  Existing benchmarks predominantly curate questions at the question level.<n>We propose Encyclo-K, a statement-based benchmark that rethinks benchmark construction from the ground up.
  arXiv  Detail & Related papers  (2025-12-31T13:55:54Z)
• CLUE: Non-parametric Verification from Experience via Hidden-State Clustering [64.50919789875233]
  We show that correctness of a solution is encoded as a geometrically separable signature within the trajectory of hidden activations.<n>ClUE consistently outperforms LLM-as-a-judge baselines and matches or exceeds modern confidence-based methods in reranking candidates.
  arXiv  Detail & Related papers  (2025-10-02T02:14:33Z)
• Causal Understanding by LLMs: The Role of Uncertainty [43.87879175532034]
  Recent papers show LLMs achieve near-random accuracy in causal relation classification.<n>We investigate whether pretraining exposure to causal examples improves causal understanding.
  arXiv  Detail & Related papers  (2025-09-24T13:06:35Z)
• GANDiff FR: Hybrid GAN Diffusion Synthesis for Causal Bias Attribution in Face Recognition [0.0]
  We introduce GANDiff FR, the first synthetic framework that precisely controls demographic and environmental factors to measure, explain, and reduce bias with reproducible rigor.<n>We synthesize 10,000 demographically balanced faces across five cohorts validated for realism via automated detection and human review.<n> Benchmarking ArcFace, CosFace, and AdaFace under matched operating points shows AdaFace reduces inter-group TPR disparity by 60%.<n>Despite around 20% computational overhead relative to pure GANs, GANDiff FR yields three times more attribute-conditioned variants.
  arXiv  Detail & Related papers  (2025-08-15T09:05:57Z)
• Benchmarking Reasoning Robustness in Large Language Models [76.79744000300363]
  We find significant performance degradation on novel or incomplete data.<n>These findings highlight the reliance on recall over rigorous logical inference.<n>This paper introduces a novel benchmark, termed as Math-RoB, that exploits hallucinations triggered by missing information to expose reasoning gaps.
  arXiv  Detail & Related papers  (2025-03-06T15:36:06Z)
• Exploring Response Uncertainty in MLLMs: An Empirical Evaluation under Misleading Scenarios [49.53589774730807]
  Multimodal large language models (MLLMs) have recently achieved state-of-the-art performance on tasks ranging from visual question answering to video understanding.<n>We reveal a response uncertainty phenomenon: twelve state-of-the-art open-source MLLMs overturn a previously correct answer in 65% of cases after receiving a single deceptive cue.
  arXiv  Detail & Related papers  (2024-11-05T01:11:28Z)
• CausalDiff: Causality-Inspired Disentanglement via Diffusion Model for Adversarial Defense [61.78357530675446]
  Humans are difficult to be cheated by subtle manipulations, since we make judgments only based on essential factors.<n>Inspired by this observation, we attempt to model label generation with essential label-causative factors and incorporate label-non-causative factors to assist data generation.<n>For an adversarial example, we aim to discriminate perturbations as non-causative factors and make predictions only based on the label-causative factors.
  arXiv  Detail & Related papers  (2024-10-30T15:06:44Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.