A Multi-Dimensional Evaluation of Explainability in Media Bias Detection

• URL: http://arxiv.org/abs/2607.19954v1
• Date: Wed, 22 Jul 2026 09:29:49 GMT
• Title: A Multi-Dimensional Evaluation of Explainability in Media Bias Detection
• Abstract summary: We present a multi-dimensional evaluation of explainability in encoder-based media bias detection using the Bias s By Experts dataset.<n>We study BERT and RoBERTa as classifiers along three complementary axes: predictive performance, explanation plausibility, and mechanistic faithfulness.<n>Our findings suggest that predictive performance, attribution plausibility, and mechanistic faithfulness characterize different aspects of model behavior.
• Score: 10.798797986346914
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Detecting media bias automatically is difficult because biased framing is often subtle, yet in domains such as news analysis, accurate predictions alone are insufficient without explanations that reflect the model's underlying reasoning. We present a multi-dimensional evaluation of explainability in encoder-based media bias detection using the Bias Annotations By Experts (BABE) dataset. Specifically, we study BERT and RoBERTa as classifiers (base and large variants) along three complementary axes: predictive performance, explanation plausibility (token-level alignment with expert rationales), and mechanistic faithfulness (whether compact sets of attention heads recover predictive signal under counterfactual rationale masking). To induce variation in plausibility, we additionally investigate attention-supervised finetuning, which incorporates expert rationale annotations as an auxiliary training signal. Attention supervision serves as an intervention on attribution plausibility, while the effectiveness of attribution methods varies substantially across architectures. Circuit analysis further reveals substantial variation in mechanistic recoverability across architectures, suggesting that model scale alone does not determine circuit compressibility. Taken together, our findings suggest that predictive performance, attribution plausibility, and mechanistic faithfulness characterize different aspects of model behavior and should be evaluated separately when studying explainability in media bias detection.

Related papers

• Uncertainty-Aware Deepfake Detection via Multi-View Structural Learning [6.951909224842812]
  Security-critical biometric and forensic applications require accurate predictions and reliable confidence estimates, particularly under distribution shift.<n>We propose an uncertainty-aware deepfake detection framework that identifies manipulations through inconsistencies across complementary evidence sources.<n>We introduce Inter-Branch Disagreement (IBDC), a disagreement-aware uncertainty modeling mechanism that links predictive uncertainty to conflicts among evidence streams.
  arXiv  Detail & Related papers  (2026-07-30T18:42:01Z)
• Beyond the Mean: Modelling Annotation Distributions in Continuous Affect Prediction [3.3187704612685267]
  We propose a distribution-aware framework that models annotation consensus using the Beta distribution.<n>Instead of predicting a single affect value, models estimate the mean and standard deviation of the annotation distribution.<n>As a result, the model captures not only the central tendency of emotional perception but also variability, asymmetry, and uncertainty in annotator responses.
  arXiv  Detail & Related papers  (2026-04-08T15:26:56Z)
• Decoding the Critique Mechanism in Large Reasoning Models [50.821607345799386]
  Large Reasoning Models (LRMs) exhibit backtracking and self-verification mechanisms that enable them to revise intermediate steps and reach correct solutions.<n>We investigate how current LRMs recover from errors by inserting arithmetic mistakes in their intermediate reasoning steps.<n>We find that despite the error propagating through the chain-of-thought, the model still reaches the correct final answer.
  arXiv  Detail & Related papers  (2026-03-17T10:03:30Z)
• On the Effects of Adversarial Perturbations on Distribution Robustness [25.334167445566496]
  Adversarial robustness refers to a model's ability to resist perturbation of inputs.<n> distribution robustness evaluates the performance of the model under data shifts.<n>Prior work has revealed a tradeoff in distribution and adversarial robustness.
  arXiv  Detail & Related papers  (2026-01-23T05:49:46Z)
• Explaining News Bias Detection: A Comparative SHAP Analysis of Transformer Model Decision Mechanisms [0.2538209532048867]
  We present a comparative interpretability study of two bias detection models: a bias detector fine-tuned on the BABE dataset and a domain-adapted pre-trained RoBERTa model fine-tuned on the BABE dataset.<n>We analyze word-level attributions across correct and incorrect predictions to characterize how different model architectures operationalize linguistic bias.
  arXiv  Detail & Related papers  (2025-12-29T19:58:11Z)
• Revisiting Multivariate Time Series Forecasting with Missing Values [65.30332997607141]
  Missing values are common in real-world time series.<n>Current approaches have developed an imputation-then-prediction framework that uses imputation modules to fill in missing values, followed by forecasting on the imputed data.<n>This framework overlooks a critical issue: there is no ground truth for the missing values, making the imputation process susceptible to errors that can degrade prediction accuracy.<n>We introduce Consistency-Regularized Information Bottleneck (CRIB), a novel framework built on the Information Bottleneck principle.
  arXiv  Detail & Related papers  (2025-09-27T20:57:48Z)
• Identifiable Latent Neural Causal Models [82.14087963690561]
  Causal representation learning seeks to uncover latent, high-level causal representations from low-level observed data. We determine the types of distribution shifts that do contribute to the identifiability of causal representations. We translate our findings into a practical algorithm, allowing for the acquisition of reliable latent causal representations.
  arXiv  Detail & Related papers  (2024-03-23T04:13:55Z)
• Quantification of Predictive Uncertainty via Inference-Time Sampling [57.749601811982096]
  We propose a post-hoc sampling strategy for estimating predictive uncertainty accounting for data ambiguity. The method can generate different plausible outputs for a given input and does not assume parametric forms of predictive distributions.
  arXiv  Detail & Related papers  (2023-08-03T12:43:21Z)
• Feature Perturbation Augmentation for Reliable Evaluation of Importance Estimators in Neural Networks [5.439020425819001]
  Post-hoc interpretability methods attempt to make the inner workings of deep neural networks more interpretable. One of the most popular evaluation frameworks is to perturb features deemed important by an interpretability method. We propose feature perturbation augmentation (FPA) which creates and adds perturbed images during the model training.
  arXiv  Detail & Related papers  (2023-03-02T19:05:46Z)
• Self-supervised debiasing using low rank regularization [59.84695042540525]
  Spurious correlations can cause strong biases in deep neural networks, impairing generalization ability. We propose a self-supervised debiasing framework potentially compatible with unlabeled samples. Remarkably, the proposed debiasing framework significantly improves the generalization performance of self-supervised learning baselines.
  arXiv  Detail & Related papers  (2022-10-11T08:26:19Z)
• Deconfounding to Explanation Evaluation in Graph Neural Networks [136.73451468551656]
  We argue that a distribution shift exists between the full graph and the subgraph, causing the out-of-distribution problem. We propose Deconfounded Subgraph Evaluation (DSE) which assesses the causal effect of an explanatory subgraph on the model prediction.
  arXiv  Detail & Related papers  (2022-01-21T18:05:00Z)
• Investigating Bias in Image Classification using Model Explanations [0.0]
  We evaluate whether model explanations could efficiently detect bias in image classification by highlighting discriminating features. We formulated important characteristics for bias detection and observed how explanations change as the degree of bias in models change.
  arXiv  Detail & Related papers  (2020-12-10T05:27:49Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.