Automatic Causal Fairness Analysis with LLM-Generated Reporting

• URL: http://arxiv.org/abs/2604.27011v1
• Date: Wed, 29 Apr 2026 10:31:38 GMT
• Title: Automatic Causal Fairness Analysis with LLM-Generated Reporting
• Abstract summary: We introduce textscFairMind, a software prototype aiming to automatise fairness analysis at the dataset level.<n>We achieve that by resorting to the assumptions of the emphstandard fairness model, recently proposed by Pleko and Bareinboim.
• Score: 39.645183687562806
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: AutoML, intended as the process of automating the application of machine learning to real-world problems, is a key step for AI popularisation. Most AutoML frameworks are not accounting for the potential lack of fairness in the training data and in the corresponding predictions. We introduce \textsc{FairMind}, a software prototype aiming to automatise fairness analysis at the dataset level. We achieve that by resorting to the assumptions of the \emph{standard fairness model}, recently proposed by Plečko and Bareinboim. This allows for a sound fairness evaluation in terms of causal effects, based on \emph{counterfactual} queries involving the target, possibly confounders and mediators, and the different values of an input feature we regard as \emph{protected}. After the necessary data preprocessing, the tool implements a closed-form computation of the effects. LLMs are consequently exploited to generate accurate reports on the fairness levels detected in the training dataset. We achieve that in a zero-shot setup and show by examples the expected advantages with respect to a direct analysis performed by the LLM. To favour applications, extensions to ordinal protected variable and continuous targets and novel decomposition results are also discussed.

Related papers

• LLM-based Agents for Forecasting and Prediction: Methods, Training, Evaluation, and Applications [76.41731220830714]
  Large language models (LLMs) now support forecasting systems that combine language-based reasoning with temporal data, evidence retrieval, external tools, and iterative prediction.<n>We examine applications in finance, weather, health, energy, and operations, and we summarize the benchmarks and datasets used for evaluation.
  arXiv  Detail & Related papers  (2026-08-24T10:02:39Z)
• AutoCluster, AutoTopicModeling, AutoTrendAnalysis: A Complete AutoML Pipeline for Predicting Emerging Trends [0.2333246928962179]
  This paper presents a trend prediction framework based on Automated Machine Learning (AutoML)<n>The proposed system offers a scalable and user-friendly solution suitable for real-time applications and stakeholders with limited machine learning expertise.<n> Experimental results demonstrate that the proposed system's best trial achieves a final RMSE of 7.099, indicating high predictive accuracy.
  arXiv  Detail & Related papers  (2026-06-23T18:35:20Z)
• LEAF: A Living Benchmark for Event-Augmented Forecasting [78.94019460534442]
  LEAF is the first living benchmark for event-augmented forecasting tasks.<n>We evaluate state-of-the-art proprietary and open-weight LLMs.<n>We find that LLMs achieve better performance on equities they confidently identify as more predictable.
  arXiv  Detail & Related papers  (2026-05-09T03:17:59Z)
• Exploring the impact of fairness-aware criteria in AutoML [0.5156484100374058]
  We study the impact of integrating fairness directly into the optimisation component of an AutoML framework.<n>Our work incorporates complementary fairness metrics to capture different dimensions of fairness during the optimisation.<n>Despite a 9.4% decrease in predictive power, the average fairness improved by 14.5%, accompanied by a 35.7% reduction in data usage.
  arXiv  Detail & Related papers  (2026-04-11T14:15:15Z)
• Context-Adaptive Requirements Defect Prediction through Human-LLM Collaboration [1.4499356176178066]
  We propose a Human-LLM Collaboration (HLC) approach that treats defect prediction as an adaptive process rather than a static classification task.<n>We evaluate this approach using the weak word smell on the QuRE benchmark of 1,266 annotated Mercedes-Benz requirements.
  arXiv  Detail & Related papers  (2026-01-05T10:00:14Z)
• Learned Hallucination Detection in Black-Box LLMs using Token-level Entropy Production Rate [0.19676943624884313]
  Hallucinations in Large Language Model (LLM) outputs for Question Answering tasks critically undermine their real-world reliability.<n>This paper introduces an applied methodology for robust, one-shot hallucination detection, specifically designed for scenarios with limited data access.<n>Our approach derives uncertainty indicators directly from these readily available log-probabilities generated during non-greedy decoding.
  arXiv  Detail & Related papers  (2025-09-01T13:34:21Z)
• Context is Key: A Benchmark for Forecasting with Essential Textual Information [87.3175915185287]
  "Context is Key" (CiK) is a forecasting benchmark that pairs numerical data with diverse types of carefully crafted textual context.<n>We evaluate a range of approaches, including statistical models, time series foundation models, and LLM-based forecasters.<n>We propose a simple yet effective LLM prompting method that outperforms all other tested methods on our benchmark.
  arXiv  Detail & Related papers  (2024-10-24T17:56:08Z)
• Debiasing Text Safety Classifiers through a Fairness-Aware Ensemble [2.1450827490014865]
  We present a light-weight, post-processing method for mitigating counterfactual fairness in closed-source text safety classifiers. We introduce two threshold-agnostic metrics to assess the counterfactual fairness of a model, and demonstrate how combining these metrics with Fair Data Reweighting (FDW) helps mitigate biases. Our results show that our approach improves counterfactual fairness with minimal impact on model performance.
  arXiv  Detail & Related papers  (2024-09-05T14:35:35Z)
• ReEval: Automatic Hallucination Evaluation for Retrieval-Augmented Large Language Models via Transferable Adversarial Attacks [91.55895047448249]
  This paper presents ReEval, an LLM-based framework using prompt chaining to perturb the original evidence for generating new test cases. We implement ReEval using ChatGPT and evaluate the resulting variants of two popular open-domain QA datasets. Our generated data is human-readable and useful to trigger hallucination in large language models.
  arXiv  Detail & Related papers  (2023-10-19T06:37:32Z)
• The Devil is in the Errors: Leveraging Large Language Models for Fine-grained Machine Translation Evaluation [93.01964988474755]
  AutoMQM is a prompting technique which asks large language models to identify and categorize errors in translations. We study the impact of labeled data through in-context learning and finetuning. We then evaluate AutoMQM with PaLM-2 models, and we find that it improves performance compared to just prompting for scores.
  arXiv  Detail & Related papers  (2023-08-14T17:17:21Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.