From Top-1 to Top-K: A Reproducibility Study and Benchmarking of Counterfactual Explanations for Recommender Systems

• URL: http://arxiv.org/abs/2604.19663v1
• Date: Tue, 21 Apr 2026 16:48:13 GMT
• Title: From Top-1 to Top-K: A Reproducibility Study and Benchmarking of Counterfactual Explanations for Recommender Systems
• Abstract summary: Counterfactual explanations (CEs) provide an intuitive way to understand recommender systems.<n>Existing CE methods for recommender systems have been evaluated using different datasets, recommenders, metrics, and explanation formats.<n>Our paper systematically reproduces, re-evaluates, and re-implements eleven state-of-the-art CE methods for recommender systems.
• Score: 3.5498952876443917
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Counterfactual explanations (CEs) provide an intuitive way to understand recommender systems by identifying minimal modifications to user-item interactions that alter recommendation outcomes. Existing CE methods for recommender systems, however, have been evaluated under heterogeneous protocols, using different datasets, recommenders, metrics, and even explanation formats, which hampers reproducibility and fair comparison. Our paper systematically reproduces, re-implement, and re-evaluate eleven state-of-the-art CE methods for recommender systems, covering both native explainers (e.g., LIME-RS, SHAP, PRINCE, ACCENT, LXR, GREASE) and specific graph-based explainers originally proposed for GNNs. Here, a unified benchmarking framework is proposed to assess explainers along three dimensions: explanation format (implicit vs. explicit), evaluation level (item-level vs. list-level), and perturbation scope (user interaction vectors vs. user-item interaction graphs). Our evaluation protocol includes effectiveness, sparsity, and computational complexity metrics, and extends existing item-level assessments to top-K list-level explanations. Through extensive experiments on three real-world datasets and six representative recommender models, we analyze how well previously reported strengths of CE methods generalize across diverse setups. We observe that the trade-off between effectiveness and sparsity depends strongly on the specific method and evaluation setting, particularly under the explicit format; in addition, explainer performance remains largely consistent across item level and list level evaluations, and several graph-based explainers exhibit notable scalability limitations on large recommender graphs. Our results refine and challenge earlier conclusions about the robustness and practicality of CE generation methods in recommender systems: https://github.com/L2R-UET/CFExpRec.

Related papers

• UniER: A Unified Benchmark for Item-level and Path-level Exercise Recommendation [53.62147575168325]
  We present a Unified Benchmark for Exercise Recommendation (UniER)<n>UniER is a comprehensive evaluation framework that unifies ILER and PLER.<n>Our results reveal the systematic dominance of PLER and expose the pedagogical failure of ILER's fragmented recommendations.
  arXiv  Detail & Related papers  (2026-05-16T02:07:58Z)
• Curr-RLCER:Curriculum Reinforcement Learning For Coherence Explainable Recommendation [5.971765441013208]
  Curr-RLCER is a reinforcement learning framework for explanation coherent recommendation with dynamic rating alignment.<n>It employs curriculum learning, transitioning from basic predictions to open-ended recommendation explanation generation.<n>The experimental results on three explainable recommendation datasets indicate that the proposed framework is effective.
  arXiv  Detail & Related papers  (2026-04-07T02:25:36Z)
• Sequence-aware Large Language Models for Explainable Recommendation [5.503509958032475]
  Large Language Models (LLMs) have shown strong potential in generating natural language explanations for recommender systems.<n>We propose a framework which integrates explanation generation with utility-aware evaluation.<n>A unified evaluation framework assesses explanations via both textual quality and their effect on recommendation outcomes.
  arXiv  Detail & Related papers  (2026-03-25T10:00:13Z)
• RAGXplain: From Explainable Evaluation to Actionable Guidance of RAG Pipelines [0.7373617024876725]
  RAGXplain is an evaluation framework that quantifies RAG performance and translates these assessments into clear insights.<n> RAGXplain thus bridges quantitative evaluation and practical optimization, empowering users to understand, trust, and enhance their AI systems.
  arXiv  Detail & Related papers  (2025-05-18T17:25:34Z)
• Revisiting Reciprocal Recommender Systems: Metrics, Formulation, and Method [60.364834418531366]
  We propose five new evaluation metrics that comprehensively and accurately assess the performance of RRS. We formulate the RRS from a causal perspective, formulating recommendations as bilateral interventions. We introduce a reranking strategy to maximize matching outcomes, as measured by the proposed metrics.
  arXiv  Detail & Related papers  (2024-08-19T07:21:02Z)
• MVIGER: Multi-View Variational Integration of Complementary Knowledge for Generative Recommender [23.305659180882625]
  This paper focuses on the inconsistency in outputs generated by variations in input prompt templates and item index types.<n>We propose MVIGER, a unified variational framework that models selection among these information sources as a categorical latent variable with a learnable prior.<n>We validate the effectiveness of MVIGER on three real-world datasets.
  arXiv  Detail & Related papers  (2024-08-16T11:59:01Z)
• RAGEval: Scenario Specific RAG Evaluation Dataset Generation Framework [66.93260816493553]
  This paper introduces RAGEval, a framework designed to assess RAG systems across diverse scenarios.<n>With a focus on factual accuracy, we propose three novel metrics: Completeness, Hallucination, and Irrelevance.<n> Experimental results show that RAGEval outperforms zero-shot and one-shot methods in terms of clarity, safety, conformity, and richness of generated samples.
  arXiv  Detail & Related papers  (2024-08-02T13:35:11Z)
• Iterative Utility Judgment Framework via LLMs Inspired by Relevance in Philosophy [66.95501113584541]
  We propose an Iterative utiliTy judgm fraEntMework (ITEM) to promote each step in Retrieval-Augmented Generation (RAG)<n>RAG's three core components -- relevance ranking derived from retrieval models, utility judgments, and answer generation -- align with Schutz's philosophical system of relevances.<n> Experimental results demonstrate significant improvements of ITEM in utility judgments, ranking, and answer generation upon representative baselines.
  arXiv  Detail & Related papers  (2024-06-17T07:52:42Z)
• Introducing User Feedback-based Counterfactual Explanations (UFCE) [49.1574468325115]
  Counterfactual explanations (CEs) have emerged as a viable solution for generating comprehensible explanations in XAI. UFCE allows for the inclusion of user constraints to determine the smallest modifications in the subset of actionable features. UFCE outperforms two well-known CE methods in terms of textitproximity, textitsparsity, and textitfeasibility.
  arXiv  Detail & Related papers  (2024-02-26T20:09:44Z)
• KPEval: Towards Fine-Grained Semantic-Based Keyphrase Evaluation [69.57018875757622]
  We propose KPEval, a comprehensive evaluation framework consisting of four critical aspects: reference agreement, faithfulness, diversity, and utility. Using KPEval, we re-evaluate 23 keyphrase systems and discover that established model comparison results have blind-spots.
  arXiv  Detail & Related papers  (2023-03-27T17:45:38Z)
• Hierarchical Bi-Directional Self-Attention Networks for Paper Review Rating Recommendation [81.55533657694016]
  We propose a Hierarchical bi-directional self-attention Network framework (HabNet) for paper review rating prediction and recommendation. Specifically, we leverage the hierarchical structure of the paper reviews with three levels of encoders: sentence encoder (level one), intra-review encoder (level two) and inter-review encoder (level three) We are able to identify useful predictors to make the final acceptance decision, as well as to help discover the inconsistency between numerical review ratings and text sentiment conveyed by reviewers.
  arXiv  Detail & Related papers  (2020-11-02T08:07:50Z)
• Embedding Ranking-Oriented Recommender System Graphs [10.74267520911262]
  We propose PGRec, a graph-based ranking-oriented recommendation framework. We have evaluated the performance of the proposed method against the state of the art model-based and neighborhood-based recommendation methods, and our experiments show that PGRec outperforms the baseline algorithms up to 3.2% in terms of NDCG@10 in different MovieLens datasets.
  arXiv  Detail & Related papers  (2020-07-31T16:56:54Z)
• Sequential Recommendation with Self-Attentive Multi-Adversarial Network [101.25533520688654]
  We present a Multi-Factor Generative Adversarial Network (MFGAN) for explicitly modeling the effect of context information on sequential recommendation. Our framework is flexible to incorporate multiple kinds of factor information, and is able to trace how each factor contributes to the recommendation decision over time.
  arXiv  Detail & Related papers  (2020-05-21T12:28:59Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.