Attention Mechanisms Don't Learn Additive Models: Rethinking Feature Importance for Transformers

• URL: http://arxiv.org/abs/2405.13536v1
• Date: Wed, 22 May 2024 11:14:00 GMT
• Title: Attention Mechanisms Don't Learn Additive Models: Rethinking Feature Importance for Transformers
• Authors: Tobias Leemann, Alina Fastowski, Felix Pfeiffer, Gjergji Kasneci,
• Abstract summary: We introduce the Softmax-Linked Additive Log-Odds Model (SLALOM), a novel surrogate model specifically designed to align with the transformer framework. SLALOM demonstrates the capacity to deliver a range of faithful and insightful explanations across both synthetic and real-world datasets.
• Score: 12.986126243018452
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We address the critical challenge of applying feature attribution methods to the transformer architecture, which dominates current applications in natural language processing and beyond. Traditional attribution methods to explainable AI (XAI) explicitly or implicitly rely on linear or additive surrogate models to quantify the impact of input features on a model's output. In this work, we formally prove an alarming incompatibility: transformers are structurally incapable to align with popular surrogate models for feature attribution, undermining the grounding of these conventional explanation methodologies. To address this discrepancy, we introduce the Softmax-Linked Additive Log-Odds Model (SLALOM), a novel surrogate model specifically designed to align with the transformer framework. Unlike existing methods, SLALOM demonstrates the capacity to deliver a range of faithful and insightful explanations across both synthetic and real-world datasets. Showing that diverse explanations computed from SLALOM outperform common surrogate explanations on different tasks, we highlight the need for task-specific feature attributions rather than a one-size-fits-all approach.

Related papers

• Disentanglement with Factor Quantized Variational Autoencoders [11.086500036180222]
  We propose a discrete variational autoencoder (VAE) based model where the ground truth information about the generative factors are not provided to the model. We demonstrate the advantages of learning discrete representations over learning continuous representations in facilitating disentanglement. Our method called FactorQVAE is the first method that combines optimization based disentanglement approaches with discrete representation learning.
  arXiv  Detail & Related papers  (2024-09-23T09:33:53Z)
• Explaining Modern Gated-Linear RNNs via a Unified Implicit Attention Formulation [54.50526986788175]
  Recent advances in efficient sequence modeling have led to attention-free layers, such as Mamba, RWKV, and various gated RNNs. We present a unified view of these models, formulating such layers as implicit causal self-attention layers. Our framework compares the underlying mechanisms on similar grounds for different layers and provides a direct means for applying explainability methods.
  arXiv  Detail & Related papers  (2024-05-26T09:57:45Z)
• MergeNet: Knowledge Migration across Heterogeneous Models, Tasks, and Modalities [72.68829963458408]
  We present MergeNet, which learns to bridge the gap of parameter spaces of heterogeneous models. The core mechanism of MergeNet lies in the parameter adapter, which operates by querying the source model's low-rank parameters. MergeNet is learned alongside both models, allowing our framework to dynamically transfer and adapt knowledge relevant to the current stage.
  arXiv  Detail & Related papers  (2024-04-20T08:34:39Z)
• AttnLRP: Attention-Aware Layer-Wise Relevance Propagation for Transformers [14.147646140595649]
  Large Language Models are prone to biased predictions and hallucinations. achieving faithful attributions for the entirety of a black-box transformer model and maintaining computational efficiency is an unsolved challenge.
  arXiv  Detail & Related papers  (2024-02-08T12:01:24Z)
• Disentanglement via Latent Quantization [60.37109712033694]
  In this work, we construct an inductive bias towards encoding to and decoding from an organized latent space. We demonstrate the broad applicability of this approach by adding it to both basic data-re (vanilla autoencoder) and latent-reconstructing (InfoGAN) generative models.
  arXiv  Detail & Related papers  (2023-05-28T06:30:29Z)
• Relational Reasoning via Set Transformers: Provable Efficiency and Applications to MARL [154.13105285663656]
  A cooperative Multi-A gent R einforcement Learning (MARL) with permutation invariant agents framework has achieved tremendous empirical successes in real-world applications. Unfortunately, the theoretical understanding of this MARL problem is lacking due to the curse of many agents and the limited exploration of the relational reasoning in existing works. We prove that the suboptimality gaps of the model-free and model-based algorithms are independent of and logarithmic in the number of agents respectively, which mitigates the curse of many agents.
  arXiv  Detail & Related papers  (2022-09-20T16:42:59Z)
• MACE: An Efficient Model-Agnostic Framework for Counterfactual Explanation [132.77005365032468]
  We propose a novel framework of Model-Agnostic Counterfactual Explanation (MACE) In our MACE approach, we propose a novel RL-based method for finding good counterfactual examples and a gradient-less descent method for improving proximity. Experiments on public datasets validate the effectiveness with better validity, sparsity and proximity.
  arXiv  Detail & Related papers  (2022-05-31T04:57:06Z)
• XAI for Transformers: Better Explanations through Conservative Propagation [60.67748036747221]
  We show that the gradient in a Transformer reflects the function only locally, and thus fails to reliably identify the contribution of input features to the prediction. Our proposal can be seen as a proper extension of the well-established LRP method to Transformers.
  arXiv  Detail & Related papers  (2022-02-15T10:47:11Z)
• Embedded-model flows: Combining the inductive biases of model-free deep learning and explicit probabilistic modeling [8.405013085269976]
  We propose embedded-model flows which alternate general-purpose transformations with structured layers that embed domain-specific inductive biases. We demonstrate that EMFs can be used to induce desirable properties such as multimodality, hierarchical coupling and continuity. In experiments, we show that this approach outperforms state-of-the-art methods in common structured inference problems.
  arXiv  Detail & Related papers  (2021-10-12T14:12:16Z)
• CARE: Coherent Actionable Recourse based on Sound Counterfactual Explanations [0.0]
  This paper introduces CARE, a modular explanation framework that addresses the model- and user-level desiderata. As a model-agnostic approach, CARE generates multiple, diverse explanations for any black-box model.
  arXiv  Detail & Related papers  (2021-08-18T15:26:59Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.