Evaluating SAE interpretability without explanations

• URL: http://arxiv.org/abs/2507.08473v1
• Date: Fri, 11 Jul 2025 10:31:53 GMT
• Title: Evaluating SAE interpretability without explanations
• Authors: Gonçalo Paulo, Nora Belrose,
• Abstract summary: We adapt existing methods to assess the interpretability of sparse coders, with the advantage that they do not require generating natural language explanations as an intermediate step.<n>We compare the scores produced by our interpretability metrics with human evaluations across similar tasks and varying setups, offering suggestions for the community on improving the evaluation of these techniques.
• Score: 0.7234862895932991
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Sparse autoencoders (SAEs) and transcoders have become important tools for machine learning interpretability. However, measuring how interpretable they are remains challenging, with weak consensus about which benchmarks to use. Most evaluation procedures start by producing a single-sentence explanation for each latent. These explanations are then evaluated based on how well they enable an LLM to predict the activation of a latent in new contexts. This method makes it difficult to disentangle the explanation generation and evaluation process from the actual interpretability of the latents discovered. In this work, we adapt existing methods to assess the interpretability of sparse coders, with the advantage that they do not require generating natural language explanations as an intermediate step. This enables a more direct and potentially standardized assessment of interpretability. Furthermore, we compare the scores produced by our interpretability metrics with human evaluations across similar tasks and varying setups, offering suggestions for the community on improving the evaluation of these techniques.

Related papers

• Do LLMs Understand Your Translations? Evaluating Paragraph-level MT with Question Answering [68.3400058037817]
  We introduce TREQA (Translation Evaluation via Question-Answering), a framework that extrinsically evaluates translation quality.<n>We show that TREQA is competitive with and, in some cases, outperforms state-of-the-art neural and LLM-based metrics in ranking alternative paragraph-level translations.
  arXiv  Detail & Related papers  (2025-04-10T09:24:54Z)
• The BiGGen Bench: A Principled Benchmark for Fine-grained Evaluation of Language Models with Language Models [94.31327813151208]
  BiGGen Bench is a principled generation benchmark designed to thoroughly evaluate nine distinct capabilities of LMs across 77 diverse tasks.<n>A key feature of the BiGGen Bench is its use of instance-specific evaluation criteria, closely mirroring the nuanced discernment of human evaluation.
  arXiv  Detail & Related papers  (2024-06-09T12:30:30Z)
• DnA-Eval: Enhancing Large Language Model Evaluation through Decomposition and Aggregation [75.81096662788254]
  Large Language Models (LLMs) are scalable and economical evaluators.<n>The question of how reliable these evaluators are has emerged as a crucial research question.<n>We propose Decompose and Aggregate, which breaks down the evaluation process into different stages based on pedagogical practices.
  arXiv  Detail & Related papers  (2024-05-24T08:12:30Z)
• Goodhart's Law Applies to NLP's Explanation Benchmarks [57.26445915212884]
  We critically examine two sets of metrics: the ERASER metrics (comprehensiveness and sufficiency) and the EVAL-X metrics. We show that we can inflate a model's comprehensiveness and sufficiency scores dramatically without altering its predictions or explanations on in-distribution test inputs. Our results raise doubts about the ability of current metrics to guide explainability research, underscoring the need for a broader reassessment of what precisely these metrics are intended to capture.
  arXiv  Detail & Related papers  (2023-08-28T03:03:03Z)
• DecompEval: Evaluating Generated Texts as Unsupervised Decomposed Question Answering [95.89707479748161]
  Existing evaluation metrics for natural language generation (NLG) tasks face the challenges on generalization ability and interpretability. We propose a metric called DecompEval that formulates NLG evaluation as an instruction-style question answering task. We decompose our devised instruction-style question about the quality of generated texts into the subquestions that measure the quality of each sentence. The subquestions with their answers generated by PLMs are then recomposed as evidence to obtain the evaluation result.
  arXiv  Detail & Related papers  (2023-07-13T16:16:51Z)
• Counterfactuals of Counterfactuals: a back-translation-inspired approach to analyse counterfactual editors [3.4253416336476246]
  We focus on the analysis of counterfactual, contrastive explanations. We propose a new back translation-inspired evaluation methodology. We show that by iteratively feeding the counterfactual to the explainer we can obtain valuable insights into the behaviour of both the predictor and the explainer models.
  arXiv  Detail & Related papers  (2023-05-26T16:04:28Z)
• The Generalizability of Explanations [0.0]
  This work proposes a novel evaluation methodology from the perspective of generalizability. We employ an Autoencoder to learn the distributions of the generated explanations and observe their learnability as well as the plausibility of the learned distributional features.
  arXiv  Detail & Related papers  (2023-02-23T12:25:59Z)
• ROSCOE: A Suite of Metrics for Scoring Step-by-Step Reasoning [63.77667876176978]
  Large language models show improved downstream task interpretability when prompted to generate step-by-step reasoning to justify their final answers. These reasoning steps greatly improve model interpretability and verification, but objectively studying their correctness is difficult. We present ROS, a suite of interpretable, unsupervised automatic scores that improve and extend previous text generation evaluation metrics.
  arXiv  Detail & Related papers  (2022-12-15T15:52:39Z)
• Evaluation and Improvement of Interpretability for Self-Explainable Part-Prototype Networks [43.821442711496154]
  Part-prototype networks have attracted broad research interest for their intrinsic interpretability and comparable accuracy to non-interpretable counterparts. We make the first attempt to quantitatively and objectively evaluate the interpretability of the part-prototype networks. We propose an elaborated part-prototype network with a shallow-deep feature alignment module and a score aggregation module to improve the interpretability of prototypes.
  arXiv  Detail & Related papers  (2022-12-12T14:59:11Z)
• Towards Human-Centred Explainability Benchmarks For Text Classification [4.393754160527062]
  We propose to extend text classification benchmarks to evaluate the explainability of text classifiers. We review challenges associated with objectively evaluating the capabilities to produce valid explanations. We propose to ground these benchmarks in human-centred applications.
  arXiv  Detail & Related papers  (2022-11-10T09:52:31Z)
• A Fine-grained Interpretability Evaluation Benchmark for Neural NLP [44.08113828762984]
  This benchmark covers three representative NLP tasks: sentiment analysis, textual similarity and reading comprehension. We provide token-level rationales that are carefully annotated to be sufficient, compact and comprehensive. We conduct experiments on three typical models with three saliency methods, and unveil their strengths and weakness in terms of interpretability.
  arXiv  Detail & Related papers  (2022-05-23T07:37:04Z)
• Evaluation of post-hoc interpretability methods in time-series classification [0.6249768559720122]
  We propose a framework with quantitative metrics to assess the performance of existing post-hoc interpretability methods.<n>We show that several drawbacks identified in the literature are addressed, namely dependence on human judgement, retraining, and shift in the data distribution when occluding samples.<n>The proposed methodology and quantitative metrics can be used to understand the reliability of interpretability methods results obtained in practical applications.
  arXiv  Detail & Related papers  (2022-02-11T14:55:56Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.