BiAxisAudit: A Novel Framework to Evaluate LLM Bias Across Prompt Sensitivity and Response-Layer Divergence

• URL: http://arxiv.org/abs/2605.09041v1
• Date: Sat, 09 May 2026 16:26:49 GMT
• Title: BiAxisAudit: A Novel Framework to Evaluate LLM Bias Across Prompt Sensitivity and Response-Layer Divergence
• Abstract summary: Bias audits of large language models now operate within governance frameworks such as the EU AI Act.<n>We introduce textscBiAxisAudit, a protocol that reports each bias score together with a reliability estimate on two axes.
• Score: 22.315546054051143
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Bias audits of large language models now operate within governance frameworks such as the EU AI Act, making benchmark reliability a security concern in its own right. Many current benchmarks, however, collapse bias into a single scalar from one prompt format and one surface label. This design misses two failure modes that can be exploited without changing model weights. Across prompts, meaning-preserving format changes shift bias endorsement by more than $0.7$ on a fixed statement pool. Within a response, the discrete Selection and free-text Elaboration can take opposing stances, so an apparently clean aggregate may hide substantial internal inconsistency (a ``cancellation trap''). Selection-only and elaboration-only rankings are therefore nearly uncorrelated across eight LLMs (Spearman $ρ= 0.238$, $p = 0.570$): LLaMA3-70B ranks in the middle under selection-only scoring but highest under elaboration-only scoring on the same responses. We introduce \textsc{BiAxisAudit}, a protocol that reports each bias score together with a reliability estimate on two orthogonal axes. The across-prompt axis evaluates each statement under a factorial grid of task format, perspective, role, and sentiment, treating bias as a distribution rather than a point estimate. The within-response axis uses Split Coding to recover Selection and Elaboration as separate signals, measured by the Inconsistency Rate and Divergence Net Imbalance. Across eight LLMs with $80{,}200$ coded responses each, task format alone explains as much variance as model choice; $63.6\%$ of pooled bias signals (up to $85.2\%$ per model) appear in only one coding layer, and prompt-dimension interactions exceed main effects. The instrument also separates real bias reductions from apparent reductions caused by cross-layer redistribution: some prompt configurations reduce both BER and IR, whereas others suppress only selection-layer bias.

Related papers

• Dual-Branch Cross-Projection Debiasing through Diffusion-based Disentanglement [66.67463557274358]
  Foundation models trained on biased datasets often rely on spurious correlations between target labels and non-causal attributes.<n>We introduce Confidence-guided Concept Mining (CBCM) to identify reliable spurious attributes without attribute annotations.<n>We propose Dual-branch Cross-projection Debiasing (DCD), a prompt-tuning framework that separates target and spurious representations into two branches.
  arXiv  Detail & Related papers  (2026-06-23T05:28:47Z)
• Attention-Discounted Adaptive Sampler for Masked Diffusion Language Models [59.51249894128724]
  Masked diffusion language models can reduce inference steps by revealing multiple tokens per denoising iteration.<n>We propose ADAS, a training-free reranking rule for parallel masked diffusion decoding.
  arXiv  Detail & Related papers  (2026-06-09T13:17:27Z)
• Correcting Selection Bias in Sparse User Feedback for Large Language Model Quality Estimation: A Multi-Agent Hierarchical Bayesian Approach [0.9558392439655014]
  We propose a three-agent hierarchical Bayesian pipeline that does not require ground-truth labels on individual interactions.<n>A mild prior on the feedback channel (typical positive-feedback rate and negative-to-positive ratio) keeps Hierarchical-Informed within 4-13 pp of $Qstar$ as the bias ratio sweeps.<n>Without channel-side priors, every weak-prior variant misses $Qstar$ by 22-33 pp.
  arXiv  Detail & Related papers  (2026-05-12T14:22:06Z)
• CogBias: Measuring and Mitigating Cognitive Bias in Large Language Models [8.81733256907103]
  Large Language Models (LLMs) are increasingly deployed in high-stakes decision-making contexts.<n>We show that cognitive biases emerge systematically across all four families of cognitive biases: Judgment, Information Processing, Social, and Response.<n>We apply activation steering to modulate biased behavior, achieving 26--32% reduction in bias score.
  arXiv  Detail & Related papers  (2026-04-01T20:22:14Z)
• CARE What Fails: Contrastive Anchored-REflection for Verifiable Multimodal [84.71254539482369]
  Group-relative reinforcement learning with verifiable rewards (RLVR) often wastes the most informative data it already has the failures.<n>We present CARE, a failure-centric post-training framework for multimodal reasoning that turns errors into supervision.<n> CARE improves accuracy and training smoothness while explicitly increasing the share of learning signal that comes from failures.
  arXiv  Detail & Related papers  (2025-12-22T16:34:21Z)
• Quantifying and Mitigating Selection Bias in LLMs: A Transferable LoRA Fine-Tuning and Efficient Majority Voting Approach [13.829059542429876]
  Multiple Choice Question (MCQ) answering is a widely used method for evaluating the performance of Large Language Models (LLMs)<n>LLMs often exhibit selection bias in MCQ tasks, where their choices are influenced by factors like answer position or option symbols rather than the content.
  arXiv  Detail & Related papers  (2025-11-17T21:31:37Z)
• Reinforcement Learning with Verifiable yet Noisy Rewards under Imperfect Verifiers [90.50039419576807]
  Reinforcement Learning with Verifiable Rewards (RLVR) trains policies against automated verifiers to avoid costly human labeling.<n>To reduce vulnerability to verifier hacking, many RLVR systems collapse rewards to binary $0,1$ during training.<n>This choice carries a cost: it introduces textitfalse negatives (rejecting correct answers, FNs) and textitfalse positives (accepting incorrect ones, FPs)
  arXiv  Detail & Related papers  (2025-10-01T13:56:44Z)
• SCOPE: Stochastic and Counterbiased Option Placement for Evaluating Large Language Models [0.27309692684728604]
  Large Language Models (LLMs) can achieve inflated scores on multiple-choice tasks by exploiting inherent biases in option positions or labels.<n>This study introduces SCOPE, an evaluation framework designed to measure and mitigate such selection bias in a dataset-independent manner.
  arXiv  Detail & Related papers  (2025-07-24T08:28:17Z)
• What makes Reasoning Models Different? Follow the Reasoning Leader for Efficient Decoding [84.42056293290015]
  We analyze the token-level misalignment between reasoning and non-reasoning models.<n>Motivated by the Local Misalignment Diminish, we propose FoReaL-Decoding.<n>On four popular math-reasoning benchmarks, FoReaL-Decoding reduces theoretical FLOPs by 30 to 50% and trims CoT length by up to 40%.
  arXiv  Detail & Related papers  (2025-06-08T05:08:32Z)
• Regression-Based Estimation of Causal Effects in the Presence of Selection Bias and Confounding [52.1068936424622]
  We consider the problem of estimating the expected causal effect $E[Y|do(X)]$ for a target variable $Y$ when treatment $X$ is set by intervention.<n>In settings without selection bias or confounding, $E[Y|do(X)] = E[Y|X]$, which can be estimated using standard regression methods.<n>We propose a framework that incorporates both selection bias and confounding.
  arXiv  Detail & Related papers  (2025-03-26T13:43:37Z)
• Attention Speaks Volumes: Localizing and Mitigating Bias in Language Models [15.53216696218776]
  We explore the internal mechanisms of how bias emerges in large language models (LLMs) when provided with ambiguous comparative prompts. We propose $textttATLAS$, a technique to localize bias to specific layers of the LLM by analyzing attention scores and then reduce bias by scaling attention in these biased layers.
  arXiv  Detail & Related papers  (2024-10-29T20:15:56Z)
• A Simple yet Effective Self-Debiasing Framework for Transformer Models [49.09053367249642]
  Current Transformer-based natural language understanding (NLU) models heavily rely on dataset biases. We propose a simple yet effective self-debiasing framework for Transformer-based NLU models.
  arXiv  Detail & Related papers  (2023-06-02T20:31:58Z)
• Bias Mimicking: A Simple Sampling Approach for Bias Mitigation [57.17709477668213]
  We introduce a new class-conditioned sampling method: Bias Mimicking. Bias Mimicking improves underrepresented groups' accuracy of sampling methods by 3% over four benchmarks.
  arXiv  Detail & Related papers  (2022-09-30T17:33:00Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.