An Information-theoretic Progressive Framework for Interpretation

• URL: http://arxiv.org/abs/2101.02879v1
• Date: Fri, 8 Jan 2021 06:59:48 GMT
• Title: An Information-theoretic Progressive Framework for Interpretation
• Authors: Zhengqi He, Taro Toyoizumi
• Abstract summary: This paper proposes an information-theoretic progressive framework to synthesize interpretation. We build the framework with an information map splitting idea and implement it with the variational information bottleneck technique. The framework is shown to be able to split information maps and synthesize interpretation in the form of meta-information.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Both brain science and the deep learning communities have the problem of interpreting neural activity. For deep learning, even though we can access all neurons' activity data, interpretation of how the deep network solves the task is still challenging. Although a large amount of effort has been devoted to interpreting a deep network, there is still no consensus of what interpretation is. This paper tries to push the discussion in this direction and proposes an information-theoretic progressive framework to synthesize interpretation. Firstly, we discuss intuitions of interpretation: interpretation is meta-information; interpretation should be at the right level; inducing independence is helpful to interpretation; interpretation is naturally progressive; interpretation doesn't have to involve a human. Then, we build the framework with an information map splitting idea and implement it with the variational information bottleneck technique. After that, we test the framework with the CLEVR dataset. The framework is shown to be able to split information maps and synthesize interpretation in the form of meta-information.

Related papers

• How Do Transformers Learn Topic Structure: Towards a Mechanistic Understanding [56.222097640468306]
  We provide mechanistic understanding of how transformers learn "semantic structure" We show, through a combination of mathematical analysis and experiments on Wikipedia data, that the embedding layer and the self-attention layer encode the topical structure.
  arXiv  Detail & Related papers  (2023-03-07T21:42:17Z)
• Toward Transparent AI: A Survey on Interpreting the Inner Structures of Deep Neural Networks [8.445831718854153]
  We review over 300 works with a focus on inner interpretability tools. We introduce a taxonomy that classifies methods by what part of the network they help to explain. We argue that the status quo in interpretability research is largely unproductive.
  arXiv  Detail & Related papers  (2022-07-27T01:59:13Z)
• Human Interpretation of Saliency-based Explanation Over Text [65.29015910991261]
  We study saliency-based explanations over textual data. We find that people often mis-interpret the explanations. We propose a method to adjust saliencies based on model estimates of over- and under-perception.
  arXiv  Detail & Related papers  (2022-01-27T15:20:32Z)
• What do End-to-End Speech Models Learn about Speaker, Language and Channel Information? A Layer-wise and Neuron-level Analysis [16.850888973106706]
  We conduct a post-hoc functional interpretability analysis of pretrained speech models using the probing framework. We analyze utterance-level representations of speech models trained for various tasks such as speaker recognition and dialect identification. Our results reveal several novel findings, including: i) channel and gender information are distributed across the network, ii) the information is redundantly available in neurons with respect to a task, and iv) complex properties such as dialectal information are encoded only in the task-oriented pretrained network.
  arXiv  Detail & Related papers  (2021-07-01T13:32:55Z)
• Compositional Processing Emerges in Neural Networks Solving Math Problems [100.80518350845668]
  Recent progress in artificial neural networks has shown that when large models are trained on enough linguistic data, grammatical structure emerges in their representations. We extend this work to the domain of mathematical reasoning, where it is possible to formulate precise hypotheses about how meanings should be composed. Our work shows that neural networks are not only able to infer something about the structured relationships implicit in their training data, but can also deploy this knowledge to guide the composition of individual meanings into composite wholes.
  arXiv  Detail & Related papers  (2021-05-19T07:24:42Z)
• Did they answer? Subjective acts and intents in conversational discourse [48.63528550837949]
  We present the first discourse dataset with multiple and subjective interpretations of English conversation. We show disagreements are nuanced and require a deeper understanding of the different contextual factors.
  arXiv  Detail & Related papers  (2021-04-09T16:34:19Z)
• Interpretable Deep Learning: Interpretations, Interpretability, Trustworthiness, and Beyond [49.93153180169685]
  We introduce and clarify two basic concepts-interpretations and interpretability-that people usually get confused. We elaborate the design of several recent interpretation algorithms, from different perspectives, through proposing a new taxonomy. We summarize the existing work in evaluating models' interpretability using "trustworthy" interpretation algorithms.
  arXiv  Detail & Related papers  (2021-03-19T08:40:30Z)
• A Survey on Neural Network Interpretability [25.27545364222555]
  interpretability is a desired property for deep networks to become powerful tools in other research fields. We propose a novel taxonomy organized along three dimensions: type of engagement (passive vs. active interpretation approaches), the type of explanation, and the focus.
  arXiv  Detail & Related papers  (2020-12-28T15:09:50Z)
• Are Interpretations Fairly Evaluated? A Definition Driven Pipeline for Post-Hoc Interpretability [54.85658598523915]
  We propose to have a concrete definition of interpretation before we could evaluate faithfulness of an interpretation. We find that although interpretation methods perform differently under a certain evaluation metric, such a difference may not result from interpretation quality or faithfulness.
  arXiv  Detail & Related papers  (2020-09-16T06:38:03Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.