CEL: Comprehensive Counterfactual Explanations Library and Benchmark

• URL: http://arxiv.org/abs/2607.22045v1
• Date: Fri, 24 Jul 2026 07:21:27 GMT
• Title: CEL: Comprehensive Counterfactual Explanations Library and Benchmark
• Abstract summary: We introduce CEL (Counterfactual Explanations Library), a unified library and benchmark for counterfactual explanations.<n>CEL includes 18 datasets of varying size and complexity and provides implementations or reimplementations of 14 widely used counterfactual methods.<n>CEL is the first comprehensive benchmark that systematically evaluates recent counterfactual explanation methods.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Counterfactual explanations are a prominent approach in explainable artificial intelligence (xAI), providing actionable guidance on what input changes would alter a model's prediction to a desired outcome. While early methods primarily focused on minimal feature changes, recent work incorporates additional properties such as sparsity, actionability and plausibility. Despite this progress, fair and systematic evaluation remains challenging. Existing studies often rely on different data splits, predictive models, and evaluation metrics, which limits objective comparison across methods. To fill this gap, we introduce CEL (Counterfactual Explanations Library), a unified library and benchmark for counterfactual explanations designed to support consistent implementation and evaluation. CEL includes 18 datasets of varying size and complexity and provides implementations or reimplementations of 14 widely used counterfactual methods. Using this standardized setup, we conduct a comprehensive quantitative comparison across a variety of methods on datasets that differ in size, number, and types of attributes. The evaluation protocol incorporates multiple complementary metrics capturing validity, coverage, sparsity, proximity, and distributional plausibility, including density- and outlier-based measures to assess the realism of generated counterfactuals. To the best of our knowledge, this is the first comprehensive benchmark that systematically evaluates recent counterfactual explanation methods within a unified and reproducible framework. While prior libraries and benchmarking efforts exist in the literature, many are outdated, limited in scope, or lack consistent evaluation protocols. The proposed benchmark aims to improve reproducibility, enable fair comparison, and establish a workbench for the development of future counterfactual explanation methods.

Related papers

• CollabEval: Statistically Efficient Collaborative Model Evaluation via Matrix Completion [65.53153291902454]
  Collaborative Evaluation (CollabEval) is a simple, effective, and principled method for exploiting dependencies between historical runs of different models on the same tasks to improve statistical efficiency.<n>We build a low-rank approximation of the score matrix, and use the reconstructed values to guarantee unbiased estimates of the true evaluation metric mean.<n>We find that CollabEval substantially reduces the mean confidence interval size, and the mean squared error of the point estimate, compared to baseline methods at the same annotation budget.
  arXiv  Detail & Related papers  (2026-07-06T13:22:04Z)
• scTranslation: A Comprehensive Benchmark for Single-Cell Multi-Omics Modality Translation [61.97613474494102]
  scTranslation is a comprehensive benchmark for single-cell multi-omics modality translation tasks.<n>It includes diverse translation datasets, integrates state-of-the-art models, and provides a comprehensive evaluation metrics.
  arXiv  Detail & Related papers  (2026-06-02T17:00:49Z)
• What-If Explanations Over Time: Counterfactuals for Time Series Classification [5.590488178822208]
  Counterfactual explanations emerge as a powerful approach in explainable AI.<n>We review state-of-the-art methods, spanning instance-based nearest-neighbor techniques, pattern-driven algorithms, gradient-based optimization, and generative models.<n>We highlight unique challenges in generating counterfactuals for temporal data, such as maintaining temporal coherence, plausibility, and actionable interpretability.
  arXiv  Detail & Related papers  (2026-03-29T18:06:23Z)
• UniDial-EvalKit: A Unified Toolkit for Evaluating Multi-Faceted Conversational Abilities [70.79422099851506]
  We present UniDial-EvalKit (UDE), a unified evaluation toolkit for assessing interactive AI systems.<n>UDE standardizes heterogeneous data formats into a universal schema, streamlines complex evaluation pipelines through a modular architecture, and aligns metric calculations under a consistent scoring interface.
  arXiv  Detail & Related papers  (2026-03-24T13:01:31Z)
• IF-RewardBench: Benchmarking Judge Models for Instruction-Following Evaluation [85.56193980646981]
  We propose IF-RewardBench, a comprehensive meta-evaluation benchmark for instruction-following.<n>For each instruction, we construct a preference graph containing all pairwise preferences among multiple responses.<n>Experiments on IF-RewardBench reveal significant deficiencies in current judge models.
  arXiv  Detail & Related papers  (2026-03-05T02:21:17Z)
• Reference-Specific Unlearning Metrics Can Hide the Truth: A Reality Check [60.77691669644931]
  We propose Functional Alignment for Distributional Equivalence (FADE), a novel metric that measures distributional similarity between unlearned and reference models.<n>We show that FADE captures functional alignment across the entire output distribution, providing a principled assessment of genuine unlearning.<n>These findings expose fundamental gaps in current evaluation practices and demonstrate that FADE provides a more robust foundation for developing and assessing truly effective unlearning methods.
  arXiv  Detail & Related papers  (2025-10-14T20:50:30Z)
• Benchmarking and Rethinking Knowledge Editing for Large Language Models [34.80161437154527]
  Knowledge editing aims to update embedded knowledge within Large Language Models (LLMs)<n>Existing approaches, whether through parameter modification or external memory integration, often suffer from inconsistent evaluation objectives and experimental setups.<n>This study offers new insights into the limitations of current knowledge editing methods and highlights the potential of context-based reasoning as a more robust alternative.
  arXiv  Detail & Related papers  (2025-05-24T13:32:03Z)
• Evaluating Generative Language Models in Information Extraction as Subjective Question Correction [49.729908337372436]
  We propose a new evaluation method, SQC-Score. Inspired by the principles in subjective question correction, we propose a new evaluation method, SQC-Score. Results on three information extraction tasks show that SQC-Score is more preferred by human annotators than the baseline metrics.
  arXiv  Detail & Related papers  (2024-04-04T15:36:53Z)
• Accounting for multiplicity in machine learning benchmark performance [0.0]
  State-of-the-art (SOTA) performance refers to the highest performance achieved by some model on a test sample.<n>We argue that SOTA should instead be estimated by the expected performance of the best classifier.
  arXiv  Detail & Related papers  (2023-03-10T10:32:18Z)
• MACE: An Efficient Model-Agnostic Framework for Counterfactual Explanation [132.77005365032468]
  We propose a novel framework of Model-Agnostic Counterfactual Explanation (MACE) In our MACE approach, we propose a novel RL-based method for finding good counterfactual examples and a gradient-less descent method for improving proximity. Experiments on public datasets validate the effectiveness with better validity, sparsity and proximity.
  arXiv  Detail & Related papers  (2022-05-31T04:57:06Z)
• TRUE: Re-evaluating Factual Consistency Evaluation [29.888885917330327]
  We introduce TRUE: a comprehensive study of factual consistency metrics on a standardized collection of existing texts from diverse tasks. Our standardization enables an example-level meta-evaluation protocol that is more actionable and interpretable than previously reported correlations. Across diverse state-of-the-art metrics and 11 datasets we find that large-scale NLI and question generation-and-answering-based approaches achieve strong and complementary results.
  arXiv  Detail & Related papers  (2022-04-11T10:14:35Z)
• TISE: A Toolbox for Text-to-Image Synthesis Evaluation [9.092600296992925]
  We conduct a study on state-of-the-art methods for single- and multi-object text-to-image synthesis. We propose a common framework for evaluating these methods.
  arXiv  Detail & Related papers  (2021-12-02T16:39:35Z)
• CARLA: A Python Library to Benchmark Algorithmic Recourse and Counterfactual Explanation Algorithms [6.133522864509327]
  CARLA (Counterfactual And Recourse LibrAry) is a python library for benchmarking counterfactual explanation methods. We provide an extensive benchmark of 11 popular counterfactual explanation methods. We also provide a benchmarking framework for research on future counterfactual explanation methods.
  arXiv  Detail & Related papers  (2021-08-02T11:00:43Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.