RankingSHAP -- Listwise Feature Attribution Explanations for Ranking Models

• URL: http://arxiv.org/abs/2403.16085v2
• Date: Tue, 29 Apr 2025 20:09:24 GMT
• Title: RankingSHAP -- Listwise Feature Attribution Explanations for Ranking Models
• Authors: Maria Heuss, Maarten de Rijke, Avishek Anand,
• Abstract summary: We present three key contributions to address this gap.<n>First, we rigorously define listwise feature attribution for ranking models.<n>Second, we introduce RankingSHAP, extending the popular SHAP framework to accommodate listwise ranking attribution.<n>Third, we propose two novel evaluation paradigms for assessing the faithfulness of attributions in learning-to-rank models.
• Score: 48.895510739010355
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: While SHAP (SHapley Additive exPlanations) and other feature attribution methods are commonly employed to explain model predictions, their application within information retrieval (IR), particularly for complex outputs such as ranked lists, remains limited. Existing attribution methods typically provide pointwise explanations, focusing on why a single document received a high-ranking score, rather than considering the relationships between documents in a ranked list. We present three key contributions to address this gap. First, we rigorously define listwise feature attribution for ranking models. Secondly, we introduce RankingSHAP, extending the popular SHAP framework to accommodate listwise ranking attribution, addressing a significant methodological gap in the field. Third, we propose two novel evaluation paradigms for assessing the faithfulness of attributions in learning-to-rank models, measuring the correctness and completeness of the explanation with respect to different aspects. Through experiments on standard learning-to-rank datasets, we demonstrate RankingSHAP's practical application while identifying the constraints of selection-based explanations. We further employ a simulated study with an interpretable model to showcase how listwise ranking attributions can be used to examine model decisions and conduct a qualitative evaluation of explanations. Due to the contrastive nature of the ranking task, our understanding of ranking model decisions can substantially benefit from feature attribution explanations like RankingSHAP.

Related papers

• On the Evaluation Consistency of Attribution-based Explanations [42.1421504321572]
  We introduce Meta-Rank, an open platform for benchmarking attribution methods in the image domain. Our benchmark reveals three insights in attribution evaluation endeavors: 1) evaluating attribution methods under disparate settings can yield divergent performance rankings; 2) although inconsistent across numerous cases, the performance rankings exhibit remarkable consistency across distinct checkpoints along the same training trajectory; and 3) prior attempts at consistent evaluation fare no better than baselines when extended to more heterogeneous models and datasets.
  arXiv  Detail & Related papers  (2024-07-28T11:49:06Z)
• Evaluating Human Alignment and Model Faithfulness of LLM Rationale [66.75309523854476]
  We study how well large language models (LLMs) explain their generations through rationales. We show that prompting-based methods are less "faithful" than attribution-based explanations.
  arXiv  Detail & Related papers  (2024-06-28T20:06:30Z)
• RankSHAP: Shapley Value Based Feature Attributions for Learning to Rank [28.438428292619577]
  We adopt an axiomatic game-theoretic approach, popular in the feature attribution community, to identify a set of fundamental axioms that every ranking-based feature attribution method should satisfy.<n>We then introduce Rank-SHAP, extending classical Shapley values to ranking.<n>We also perform an axiomatic analysis of existing rank attribution algorithms to determine their compliance with our proposed axioms.
  arXiv  Detail & Related papers  (2024-05-03T04:43:24Z)
• Evaluating Generative Language Models in Information Extraction as Subjective Question Correction [49.729908337372436]
  We propose a new evaluation method, SQC-Score. Inspired by the principles in subjective question correction, we propose a new evaluation method, SQC-Score. Results on three information extraction tasks show that SQC-Score is more preferred by human annotators than the baseline metrics.
  arXiv  Detail & Related papers  (2024-04-04T15:36:53Z)
• Rank-LIME: Local Model-Agnostic Feature Attribution for Learning to Rank [16.780058676633914]
  Rank-LIME is a model-agnostic, local, post-hoc linear feature attribution method for the task of learning to rank. We employ novel correlation-based perturbations, differentiable ranking loss functions and introduce new metrics to evaluate ranking based additive feature attribution models.
  arXiv  Detail & Related papers  (2022-12-24T12:14:32Z)
• Learning List-Level Domain-Invariant Representations for Ranking [59.3544317373004]
  We propose list-level alignment -- learning domain-invariant representations at the higher level of lists. The benefits are twofold: it leads to the first domain adaptation generalization bound for ranking, in turn providing theoretical support for the proposed method.
  arXiv  Detail & Related papers  (2022-12-21T04:49:55Z)
• Discover, Explanation, Improvement: An Automatic Slice Detection Framework for Natural Language Processing [72.14557106085284]
  slice detection models (SDM) automatically identify underperforming groups of datapoints. This paper proposes a benchmark named "Discover, Explain, improve (DEIM)" for classification NLP tasks. Our evaluation shows that Edisa can accurately select error-prone datapoints with informative semantic features.
  arXiv  Detail & Related papers  (2022-11-08T19:00:00Z)
• Few-Shot Named Entity Recognition: A Comprehensive Study [92.40991050806544]
  We investigate three schemes to improve the model generalization ability for few-shot settings. We perform empirical comparisons on 10 public NER datasets with various proportions of labeled data. We create new state-of-the-art results on both few-shot and training-free settings.
  arXiv  Detail & Related papers  (2020-12-29T23:43:16Z)
• Towards Unifying Feature Attribution and Counterfactual Explanations: Different Means to the Same End [17.226134854746267]
  We present a method to generate feature attribution explanations from a set of counterfactual examples. We show how counterfactual examples can be used to evaluate the goodness of an attribution-based explanation in terms of its necessity and sufficiency.
  arXiv  Detail & Related papers  (2020-11-10T05:41:43Z)
• Interpretable Learning-to-Rank with Generalized Additive Models [78.42800966500374]
  Interpretability of learning-to-rank models is a crucial yet relatively under-examined research area. Recent progress on interpretable ranking models largely focuses on generating post-hoc explanations for existing black-box ranking models. We lay the groundwork for intrinsically interpretable learning-to-rank by introducing generalized additive models (GAMs) into ranking tasks.
  arXiv  Detail & Related papers  (2020-05-06T01:51:30Z)
• Valid Explanations for Learning to Rank Models [5.320400771224103]
  We propose a model agnostic local explanation method that seeks to identify a small subset of input features as explanation to a ranking decision. We introduce new notions of validity and completeness of explanations specifically for rankings, based on the presence or absence of selected features.
  arXiv  Detail & Related papers  (2020-04-29T06:21:56Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.