Attributing Learned Concepts in Neural Networks to Training Data

• URL: http://arxiv.org/abs/2310.03149v4
• Date: Thu, 28 Dec 2023 18:03:12 GMT
• Title: Attributing Learned Concepts in Neural Networks to Training Data
• Authors: Nicholas Konz, Charles Godfrey, Madelyn Shapiro, Jonathan Tu, Henry Kvinge, Davis Brown
• Abstract summary: We find evidence for convergence, where removing the 10,000 top attributing images for a concept and retraining the model does not change the location of the concept in the network. This suggests that the features that inform the development of a concept are spread in a more diffuse manner across its exemplars, implying robustness in concept formation.
• Score: 5.930268338525991
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: By now there is substantial evidence that deep learning models learn certain human-interpretable features as part of their internal representations of data. As having the right (or wrong) concepts is critical to trustworthy machine learning systems, it is natural to ask which inputs from the model's original training set were most important for learning a concept at a given layer. To answer this, we combine data attribution methods with methods for probing the concepts learned by a model. Training network and probe ensembles for two concept datasets on a range of network layers, we use the recently developed TRAK method for large-scale data attribution. We find some evidence for convergence, where removing the 10,000 top attributing images for a concept and retraining the model does not change the location of the concept in the network nor the probing sparsity of the concept. This suggests that rather than being highly dependent on a few specific examples, the features that inform the development of a concept are spread in a more diffuse manner across its exemplars, implying robustness in concept formation.

Related papers

• Restyling Unsupervised Concept Based Interpretable Networks with Generative Models [14.604305230535026]
  We propose a novel method that relies on mapping the concept features to the latent space of a pretrained generative model. We quantitatively ascertain the efficacy of our method in terms of accuracy of the interpretable prediction network, fidelity of reconstruction, as well as faithfulness and consistency of learnt concepts.
  arXiv  Detail & Related papers  (2024-07-01T14:39:41Z)
• Concept Distillation: Leveraging Human-Centered Explanations for Model Improvement [3.026365073195727]
  Concept Activation Vectors (CAVs) estimate a model's sensitivity and possible biases to a given concept. We extend CAVs from post-hoc analysis to ante-hoc training in order to reduce model bias through fine-tuning. We show applications of concept-sensitive training to debias several classification problems.
  arXiv  Detail & Related papers  (2023-11-26T14:00:14Z)
• A Recursive Bateson-Inspired Model for the Generation of Semantic Formal Concepts from Spatial Sensory Data [77.34726150561087]
  This paper presents a new symbolic-only method for the generation of hierarchical concept structures from complex sensory data. The approach is based on Bateson's notion of difference as the key to the genesis of an idea or a concept. The model is able to produce fairly rich yet human-readable conceptual representations without training.
  arXiv  Detail & Related papers  (2023-07-16T15:59:13Z)
• Uncovering Unique Concept Vectors through Latent Space Decomposition [0.0]
  Concept-based explanations have emerged as a superior approach that is more interpretable than feature attribution estimates. We propose a novel post-hoc unsupervised method that automatically uncovers the concepts learned by deep models during training. Our experiments reveal that the majority of our concepts are readily understandable to humans, exhibit coherency, and bear relevance to the task at hand.
  arXiv  Detail & Related papers  (2023-07-13T17:21:54Z)
• Hierarchical Semantic Tree Concept Whitening for Interpretable Image Classification [19.306487616731765]
  Post-hoc analysis can only discover the patterns or rules that naturally exist in models. We proactively instill knowledge to alter the representation of human-understandable concepts in hidden layers. Our method improves model interpretability, showing better disentanglement of semantic concepts, without negatively affecting model classification performance.
  arXiv  Detail & Related papers  (2023-07-10T04:54:05Z)
• Neural networks trained with SGD learn distributions of increasing complexity [78.30235086565388]
  We show that neural networks trained using gradient descent initially classify their inputs using lower-order input statistics. We then exploit higher-order statistics only later during training. We discuss the relation of DSB to other simplicity biases and consider its implications for the principle of universality in learning.
  arXiv  Detail & Related papers  (2022-11-21T15:27:22Z)
• Concept-Based Explanations for Tabular Data [0.0]
  We propose a concept-based explainability for Deep Neural Networks (DNNs) We show the validity of our method in generating interpretability results that match the human-level intuitions. We also propose a notion of fairness based on TCAV that quantifies what layer of DNN has learned representations that lead to biased predictions.
  arXiv  Detail & Related papers  (2022-09-13T02:19:29Z)
• Human-Centered Concept Explanations for Neural Networks [47.71169918421306]
  We introduce concept explanations including the class of Concept Activation Vectors (CAV) We then discuss approaches to automatically extract concepts, and approaches to address some of their caveats. Finally, we discuss some case studies that showcase the utility of such concept-based explanations in synthetic settings and real world applications.
  arXiv  Detail & Related papers  (2022-02-25T01:27:31Z)
• Reasoning-Modulated Representations [85.08205744191078]
  We study a common setting where our task is not purely opaque. Our approach paves the way for a new class of data-efficient representation learning.
  arXiv  Detail & Related papers  (2021-07-19T13:57:13Z)
• Adversarially-Trained Deep Nets Transfer Better: Illustration on Image Classification [53.735029033681435]
  Transfer learning is a powerful methodology for adapting pre-trained deep neural networks on image recognition tasks to new domains. In this work, we demonstrate that adversarially-trained models transfer better than non-adversarially-trained models.
  arXiv  Detail & Related papers  (2020-07-11T22:48:42Z)
• The large learning rate phase of deep learning: the catapult mechanism [50.23041928811575]
  We present a class of neural networks with solvable training dynamics. We find good agreement between our model's predictions and training dynamics in realistic deep learning settings. We believe our results shed light on characteristics of models trained at different learning rates.
  arXiv  Detail & Related papers  (2020-03-04T17:52:48Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.