Attention Mechanisms Don't Learn Additive Models: Rethinking Feature Importance for Transformers

• URL: http://arxiv.org/abs/2405.13536v2
• Date: Thu, 09 Jan 2025 17:58:44 GMT
• Title: Attention Mechanisms Don't Learn Additive Models: Rethinking Feature Importance for Transformers
• Authors: Tobias Leemann, Alina Fastowski, Felix Pfeiffer, Gjergji Kasneci,
• Abstract summary: transformers are structurally incapable of representing linear or additive surrogate models used for feature attribution. We introduce the Softmax-Linked Additive Log Odds Model (SLALOM), a novel surrogate model specifically designed to align with the transformer framework. We highlight SLALOM's unique efficiency-quality curve by showing that SLALOM can produce explanations with substantially higher fidelity than competing surrogate models.
• Score: 12.986126243018452
• License:
• Abstract: We address the critical challenge of applying feature attribution methods to the transformer architecture, which dominates current applications in natural language processing and beyond. Traditional attribution methods to explainable AI (XAI) explicitly or implicitly rely on linear or additive surrogate models to quantify the impact of input features on a model's output. In this work, we formally prove an alarming incompatibility: transformers are structurally incapable of representing linear or additive surrogate models used for feature attribution, undermining the grounding of these conventional explanation methodologies. To address this discrepancy, we introduce the Softmax-Linked Additive Log Odds Model (SLALOM), a novel surrogate model specifically designed to align with the transformer framework. SLALOM demonstrates the capacity to deliver a range of insightful explanations with both synthetic and real-world datasets. We highlight SLALOM's unique efficiency-quality curve by showing that SLALOM can produce explanations with substantially higher fidelity than competing surrogate models or provide explanations of comparable quality at a fraction of their computational costs. We release code for SLALOM as an open-source project online at https://github.com/tleemann/slalom_explanations.

Related papers

• Disentanglement with Factor Quantized Variational Autoencoders [11.086500036180222]
  We propose a discrete variational autoencoder (VAE) based model where the ground truth information about the generative factors are not provided to the model. We demonstrate the advantages of learning discrete representations over learning continuous representations in facilitating disentanglement. Our method called FactorQVAE combines optimization based disentanglement approaches with discrete representation learning.
  arXiv  Detail & Related papers  (2024-09-23T09:33:53Z)
• SMILE: Zero-Shot Sparse Mixture of Low-Rank Experts Construction From Pre-Trained Foundation Models [85.67096251281191]
  We present an innovative approach to model fusion called zero-shot Sparse MIxture of Low-rank Experts (SMILE) construction. SMILE allows for the upscaling of source models into an MoE model without extra data or further training. We conduct extensive experiments across diverse scenarios, such as image classification and text generation tasks, using full fine-tuning and LoRA fine-tuning.
  arXiv  Detail & Related papers  (2024-08-19T17:32:15Z)
• Explaining Modern Gated-Linear RNNs via a Unified Implicit Attention Formulation [54.50526986788175]
  Recent advances in efficient sequence modeling have led to attention-free layers, such as Mamba, RWKV, and various gated RNNs. We present a unified view of these models, formulating such layers as implicit causal self-attention layers. Our framework compares the underlying mechanisms on similar grounds for different layers and provides a direct means for applying explainability methods.
  arXiv  Detail & Related papers  (2024-05-26T09:57:45Z)
• Disentanglement via Latent Quantization [60.37109712033694]
  In this work, we construct an inductive bias towards encoding to and decoding from an organized latent space. We demonstrate the broad applicability of this approach by adding it to both basic data-re (vanilla autoencoder) and latent-reconstructing (InfoGAN) generative models.
  arXiv  Detail & Related papers  (2023-05-28T06:30:29Z)
• Distributional Learning of Variational AutoEncoder: Application to Synthetic Data Generation [0.7614628596146602]
  We propose a new approach that expands the model capacity without sacrificing the computational advantages of the VAE framework. Our VAE model's decoder is composed of an infinite mixture of asymmetric Laplace distribution. We apply the proposed model to synthetic data generation, and particularly, our model demonstrates superiority in easily adjusting the level of data privacy.
  arXiv  Detail & Related papers  (2023-02-22T11:26:50Z)
• MACE: An Efficient Model-Agnostic Framework for Counterfactual Explanation [132.77005365032468]
  We propose a novel framework of Model-Agnostic Counterfactual Explanation (MACE) In our MACE approach, we propose a novel RL-based method for finding good counterfactual examples and a gradient-less descent method for improving proximity. Experiments on public datasets validate the effectiveness with better validity, sparsity and proximity.
  arXiv  Detail & Related papers  (2022-05-31T04:57:06Z)
• Automatic Rule Induction for Efficient Semi-Supervised Learning [56.91428251227253]
  Semi-supervised learning has shown promise in allowing NLP models to generalize from small amounts of labeled data. Pretrained transformer models act as black-box correlation engines that are difficult to explain and sometimes behave unreliably. We propose tackling both of these challenges via Automatic Rule Induction (ARI), a simple and general-purpose framework.
  arXiv  Detail & Related papers  (2022-05-18T16:50:20Z)
• Learning Deep Implicit Fourier Neural Operators (IFNOs) with Applications to Heterogeneous Material Modeling [3.9181541460605116]
  We propose to use data-driven modeling to predict a material's response without using conventional models. The material response is modeled by learning the implicit mappings between loading conditions and the resultant displacement and/or damage fields. We demonstrate the performance of our proposed method for a number of examples, including hyperelastic, anisotropic and brittle materials.
  arXiv  Detail & Related papers  (2022-03-15T19:08:13Z)
• XAI for Transformers: Better Explanations through Conservative Propagation [60.67748036747221]
  We show that the gradient in a Transformer reflects the function only locally, and thus fails to reliably identify the contribution of input features to the prediction. Our proposal can be seen as a proper extension of the well-established LRP method to Transformers.
  arXiv  Detail & Related papers  (2022-02-15T10:47:11Z)
• Causality-aware counterfactual confounding adjustment for feature representations learned by deep models [14.554818659491644]
  Causal modeling has been recognized as a potential solution to many challenging problems in machine learning (ML) We describe how a recently proposed counterfactual approach can still be used to deconfound the feature representations learned by deep neural network (DNN) models.
  arXiv  Detail & Related papers  (2020-04-20T17:37:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.