Fast Axiomatic Attribution for Neural Networks

• URL: http://arxiv.org/abs/2111.07668v1
• Date: Mon, 15 Nov 2021 10:51:01 GMT
• Title: Fast Axiomatic Attribution for Neural Networks
• Authors: Robin Hesse, Simone Schaub-Meyer, Stefan Roth
• Abstract summary: Recent approaches include priors on the feature attribution of a deep neural network (DNN) into the training process to reduce the dependence on unwanted features. We consider a special class of efficiently axiomatically attributable DNNs for which an axiomatic feature attribution can be computed with only a single forward/backward pass. Various experiments demonstrate the advantages of $mathcalX$-DNNs, beating state-of-the-art generic attribution methods on regular DNNs for training with attribution priors.
• Score: 44.527672563424545
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Mitigating the dependence on spurious correlations present in the training dataset is a quickly emerging and important topic of deep learning. Recent approaches include priors on the feature attribution of a deep neural network (DNN) into the training process to reduce the dependence on unwanted features. However, until now one needed to trade off high-quality attributions, satisfying desirable axioms, against the time required to compute them. This in turn either led to long training times or ineffective attribution priors. In this work, we break this trade-off by considering a special class of efficiently axiomatically attributable DNNs for which an axiomatic feature attribution can be computed with only a single forward/backward pass. We formally prove that nonnegatively homogeneous DNNs, here termed $\mathcal{X}$-DNNs, are efficiently axiomatically attributable and show that they can be effortlessly constructed from a wide range of regular DNNs by simply removing the bias term of each layer. Various experiments demonstrate the advantages of $\mathcal{X}$-DNNs, beating state-of-the-art generic attribution methods on regular DNNs for training with attribution priors.

Related papers

• Theoretical Analysis of Robust Overfitting for Wide DNNs: An NTK Approach [8.994430921243767]
  Adversarial training (AT) is a canonical method for enhancing the robustness of deep neural networks (DNNs) We non-trivially extend the neural tangent kernel (NTK) theory to AT and prove that an adversarially trained wide DNN can be well approximated by a linearized DNN. For squared loss, closed-form AT dynamics for the linearized DNN can be derived, which reveals a new AT degeneration phenomenon.
  arXiv  Detail & Related papers  (2023-10-09T19:40:25Z)
• Benign Overfitting in Deep Neural Networks under Lazy Training [72.28294823115502]
  We show that when the data distribution is well-separated, DNNs can achieve Bayes-optimal test error for classification. Our results indicate that interpolating with smoother functions leads to better generalization.
  arXiv  Detail & Related papers  (2023-05-30T19:37:44Z)
• Taming Reachability Analysis of DNN-Controlled Systems via Abstraction-Based Training [14.787056022080625]
  This paper presents a novel abstraction-based approach to bypass the crux of over-approximating DNNs in reachability analysis. We extend conventional DNNs by inserting an additional abstraction layer, which abstracts a real number to an interval for training. We devise the first black-box reachability analysis approach for DNN-controlled systems, where trained DNNs are only queried as black-box oracles for the actions on abstract states.
  arXiv  Detail & Related papers  (2022-11-21T00:11:50Z)
• Comparative Analysis of Interval Reachability for Robust Implicit and Feedforward Neural Networks [64.23331120621118]
  We use interval reachability analysis to obtain robustness guarantees for implicit neural networks (INNs) INNs are a class of implicit learning models that use implicit equations as layers. We show that our approach performs at least as well as, and generally better than, applying state-of-the-art interval bound propagation methods to INNs.
  arXiv  Detail & Related papers  (2022-04-01T03:31:27Z)
• Rethinking Nearest Neighbors for Visual Classification [56.00783095670361]
  k-NN is a lazy learning method that aggregates the distance between the test image and top-k neighbors in a training set. We adopt k-NN with pre-trained visual representations produced by either supervised or self-supervised methods in two steps. Via extensive experiments on a wide range of classification tasks, our study reveals the generality and flexibility of k-NN integration.
  arXiv  Detail & Related papers  (2021-12-15T20:15:01Z)
• Understanding and Improving Early Stopping for Learning with Noisy Labels [63.0730063791198]
  The memorization effect of deep neural network (DNN) plays a pivotal role in many state-of-the-art label-noise learning methods. Current methods generally decide the early stopping point by considering a DNN as a whole. We propose to separate a DNN into different parts and progressively train them to address this problem.
  arXiv  Detail & Related papers  (2021-06-30T07:18:00Z)
• S2-BNN: Bridging the Gap Between Self-Supervised Real and 1-bit Neural Networks via Guided Distribution Calibration [74.5509794733707]
  We present a novel guided learning paradigm from real-valued to distill binary networks on the final prediction distribution. Our proposed method can boost the simple contrastive learning baseline by an absolute gain of 5.515% on BNNs. Our method achieves substantial improvement over the simple contrastive learning baseline, and is even comparable to many mainstream supervised BNN methods.
  arXiv  Detail & Related papers  (2021-02-17T18:59:28Z)
• Implicit Bias of Linear RNNs [27.41989861342218]
  Linear recurrent neural networks (RNNs) do not perform well on tasks requiring long-term memory. This paper provides a rigorous explanation of this property in the special case of linear RNNs. Using recently-developed kernel regime analysis, our main result shows that linear RNNs are functionally equivalent to a certain weighted 1D-convolutional network.
  arXiv  Detail & Related papers  (2021-01-19T19:39:28Z)
• Fractional Deep Neural Network via Constrained Optimization [0.0]
  This paper introduces a novel algorithmic framework for a deep neural network (DNN) Fractional-DNN can be viewed as a time-discretization of a fractional in time nonlinear ordinary differential equation (ODE)
  arXiv  Detail & Related papers  (2020-04-01T21:58:21Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.