Does a Neural Network Really Encode Symbolic Concepts?
- URL: http://arxiv.org/abs/2302.13080v3
- Date: Fri, 13 Sep 2024 15:38:23 GMT
- Title: Does a Neural Network Really Encode Symbolic Concepts?
- Authors: Mingjie Li, Quanshi Zhang,
- Abstract summary: In this paper, we examine the trustworthiness of interaction concepts from four perspectives.
Extensive empirical studies have verified that a well-trained DNN usually encodes sparse, transferable, and discriminative concepts.
- Score: 24.099892982101398
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Recently, a series of studies have tried to extract interactions between input variables modeled by a DNN and define such interactions as concepts encoded by the DNN. However, strictly speaking, there still lacks a solid guarantee whether such interactions indeed represent meaningful concepts. Therefore, in this paper, we examine the trustworthiness of interaction concepts from four perspectives. Extensive empirical studies have verified that a well-trained DNN usually encodes sparse, transferable, and discriminative concepts, which is partially aligned with human intuition.
Related papers
- Two-Phase Dynamics of Interactions Explains the Starting Point of a DNN Learning Over-Fitted Features [68.3512123520931]
We investigate the dynamics of a deep neural network (DNN) learning interactions.
In this paper, we discover the DNN learns interactions in two phases.
The first phase mainly penalizes interactions of medium and high orders, and the second phase mainly learns interactions of gradually increasing orders.
arXiv Detail & Related papers (2024-05-16T17:13:25Z) - Defining and Extracting generalizable interaction primitives from DNNs [22.79131582164054]
We develop a new method to extract interactions that are shared by different deep neural networks (DNNs)
Experiments show that the extracted interactions can better reflect common knowledge shared by different DNNs.
arXiv Detail & Related papers (2024-01-29T17:21:41Z) - Deep Neural Networks Tend To Extrapolate Predictably [51.303814412294514]
neural network predictions tend to be unpredictable and overconfident when faced with out-of-distribution (OOD) inputs.
We observe that neural network predictions often tend towards a constant value as input data becomes increasingly OOD.
We show how one can leverage our insights in practice to enable risk-sensitive decision-making in the presence of OOD inputs.
arXiv Detail & Related papers (2023-10-02T03:25:32Z) - Technical Note: Defining and Quantifying AND-OR Interactions for Faithful and Concise Explanation of DNNs [24.099892982101398]
We aim to explain a deep neural network (DNN) by quantifying the encoded interactions between input variables.
Specifically, we first rethink the definition of interactions, and then formally define faithfulness and conciseness for interaction-based explanation.
arXiv Detail & Related papers (2023-04-26T06:33:31Z) - Bayesian Neural Networks Avoid Encoding Complex and
Perturbation-Sensitive Concepts [22.873523599349326]
In this paper, we focus on mean-field variational Bayesian Neural Networks (BNNs) and explore the representation capacity of such BNNs.
It has been observed and studied that a relatively small set of interactive concepts usually emerge in the knowledge representation of a sufficiently-trained neural network.
Our study proves that compared to standard deep neural networks (DNNs), it is less likely for BNNs to encode complex concepts.
arXiv Detail & Related papers (2023-02-25T14:56:35Z) - Explaining Generalization Power of a DNN Using Interactive Concepts [24.712192363947096]
This paper explains the generalization power of a deep neural network (DNN) from the perspective of interactions.
We also discover the detouring dynamics of learning complex concepts, which explains both the high learning difficulty and the low generalization power of complex concepts.
arXiv Detail & Related papers (2023-02-25T14:44:40Z) - Emergence of Concepts in DNNs? [0.0]
It is examined, first, how existing methods actually identify concepts that are supposedly represented in DNNs.
Second, it is discussed how conceptual spaces are shaped by a tradeoff between predictive accuracy and compression.
arXiv Detail & Related papers (2022-11-11T11:25:39Z) - Neural Networks with Recurrent Generative Feedback [61.90658210112138]
We instantiate this design on convolutional neural networks (CNNs)
In the experiments, CNN-F shows considerably improved adversarial robustness over conventional feedforward CNNs on standard benchmarks.
arXiv Detail & Related papers (2020-07-17T19:32:48Z) - A Chain Graph Interpretation of Real-World Neural Networks [58.78692706974121]
We propose an alternative interpretation that identifies NNs as chain graphs (CGs) and feed-forward as an approximate inference procedure.
The CG interpretation specifies the nature of each NN component within the rich theoretical framework of probabilistic graphical models.
We demonstrate with concrete examples that the CG interpretation can provide novel theoretical support and insights for various NN techniques.
arXiv Detail & Related papers (2020-06-30T14:46:08Z) - Neural Additive Models: Interpretable Machine Learning with Neural Nets [77.66871378302774]
Deep neural networks (DNNs) are powerful black-box predictors that have achieved impressive performance on a wide variety of tasks.
We propose Neural Additive Models (NAMs) which combine some of the expressivity of DNNs with the inherent intelligibility of generalized additive models.
NAMs learn a linear combination of neural networks that each attend to a single input feature.
arXiv Detail & Related papers (2020-04-29T01:28:32Z) - Architecture Disentanglement for Deep Neural Networks [174.16176919145377]
We introduce neural architecture disentanglement (NAD) to explain the inner workings of deep neural networks (DNNs)
NAD learns to disentangle a pre-trained DNN into sub-architectures according to independent tasks, forming information flows that describe the inference processes.
Results show that misclassified images have a high probability of being assigned to task sub-architectures similar to the correct ones.
arXiv Detail & Related papers (2020-03-30T08:34:33Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.