Contextual Semantic Interpretability
- URL: http://arxiv.org/abs/2009.08720v1
- Date: Fri, 18 Sep 2020 09:47:05 GMT
- Title: Contextual Semantic Interpretability
- Authors: Diego Marcos, Ruth Fong, Sylvain Lobry, Remi Flamary, Nicolas Courty
and Devis Tuia
- Abstract summary: We look into semantic bottlenecks that capture context.
We use a two-layer semantic bottleneck that gathers attributes into interpretable, sparse groups.
Our model yields in predictions as accurate as a non-interpretable baseline when applied to a real-world test set of Flickr images.
- Score: 16.18912769522768
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: Convolutional neural networks (CNN) are known to learn an image
representation that captures concepts relevant to the task, but do so in an
implicit way that hampers model interpretability. However, one could argue that
such a representation is hidden in the neurons and can be made explicit by
teaching the model to recognize semantically interpretable attributes that are
present in the scene. We call such an intermediate layer a \emph{semantic
bottleneck}. Once the attributes are learned, they can be re-combined to reach
the final decision and provide both an accurate prediction and an explicit
reasoning behind the CNN decision. In this paper, we look into semantic
bottlenecks that capture context: we want attributes to be in groups of a few
meaningful elements and participate jointly to the final decision. We use a
two-layer semantic bottleneck that gathers attributes into interpretable,
sparse groups, allowing them contribute differently to the final output
depending on the context. We test our contextual semantic interpretable
bottleneck (CSIB) on the task of landscape scenicness estimation and train the
semantic interpretable bottleneck using an auxiliary database (SUN Attributes).
Our model yields in predictions as accurate as a non-interpretable baseline
when applied to a real-world test set of Flickr images, all while providing
clear and interpretable explanations for each prediction.
Related papers
- Linking in Style: Understanding learned features in deep learning models [0.0]
Convolutional neural networks (CNNs) learn abstract features to perform object classification.
We propose an automatic method to visualize and systematically analyze learned features in CNNs.
arXiv Detail & Related papers (2024-09-25T12:28:48Z) - Semantics Meets Temporal Correspondence: Self-supervised Object-centric Learning in Videos [63.94040814459116]
Self-supervised methods have shown remarkable progress in learning high-level semantics and low-level temporal correspondence.
We propose a novel semantic-aware masked slot attention on top of the fused semantic features and correspondence maps.
We adopt semantic- and instance-level temporal consistency as self-supervision to encourage temporally coherent object-centric representations.
arXiv Detail & Related papers (2023-08-19T09:12:13Z) - Seeing in Words: Learning to Classify through Language Bottlenecks [59.97827889540685]
Humans can explain their predictions using succinct and intuitive descriptions.
We show that a vision model whose feature representations are text can effectively classify ImageNet images.
arXiv Detail & Related papers (2023-06-29T00:24:42Z) - NeSyFOLD: Neurosymbolic Framework for Interpretable Image Classification [1.3812010983144802]
We present a novel framework called NeSyFOLD to create a neurosymbolic (NeSy) model for image classification tasks.
A rule-based machine learning algorithm called FOLD-SE-M is used to derive the stratified answer set program.
A justification for the predictions made by the NeSy model can be obtained using an ASP interpreter.
arXiv Detail & Related papers (2023-01-30T05:08:05Z) - Measuring the Interpretability of Unsupervised Representations via
Quantized Reverse Probing [97.70862116338554]
We investigate the problem of measuring interpretability of self-supervised representations.
We formulate the latter as estimating the mutual information between the representation and a space of manually labelled concepts.
We use our method to evaluate a large number of self-supervised representations, ranking them by interpretability.
arXiv Detail & Related papers (2022-09-07T16:18:50Z) - Dynamic Inference with Neural Interpreters [72.90231306252007]
We present Neural Interpreters, an architecture that factorizes inference in a self-attention network as a system of modules.
inputs to the model are routed through a sequence of functions in a way that is end-to-end learned.
We show that Neural Interpreters perform on par with the vision transformer using fewer parameters, while being transferrable to a new task in a sample efficient manner.
arXiv Detail & Related papers (2021-10-12T23:22:45Z) - Interpretable Compositional Convolutional Neural Networks [20.726080433723922]
We propose a method to modify a traditional convolutional neural network (CNN) into an interpretable compositional CNN.
In a compositional CNN, each filter is supposed to consistently represent a specific compositional object part or image region with a clear meaning.
Our method can be broadly applied to different types of CNNs.
arXiv Detail & Related papers (2021-07-09T15:01:24Z) - Learning Semantically Meaningful Features for Interpretable
Classifications [17.88784870849724]
SemCNN learns associations between visual features and word phrases.
Experiment results on multiple benchmark datasets demonstrate that SemCNN can learn features with clear semantic meaning.
arXiv Detail & Related papers (2021-01-11T14:35:16Z) - Towards Efficient Scene Understanding via Squeeze Reasoning [71.1139549949694]
We propose a novel framework called Squeeze Reasoning.
Instead of propagating information on the spatial map, we first learn to squeeze the input feature into a channel-wise global vector.
We show that our approach can be modularized as an end-to-end trained block and can be easily plugged into existing networks.
arXiv Detail & Related papers (2020-11-06T12:17:01Z) - Interpreting Graph Neural Networks for NLP With Differentiable Edge
Masking [63.49779304362376]
Graph neural networks (GNNs) have become a popular approach to integrating structural inductive biases into NLP models.
We introduce a post-hoc method for interpreting the predictions of GNNs which identifies unnecessary edges.
We show that we can drop a large proportion of edges without deteriorating the performance of the model.
arXiv Detail & Related papers (2020-10-01T17:51:19Z) - A Disentangling Invertible Interpretation Network for Explaining Latent
Representations [19.398202091883366]
We formulate interpretation as a translation of hidden representations onto semantic concepts that are comprehensible to the user.
The proposed invertible interpretation network can be transparently applied on top of existing architectures.
We present an efficient approach to define semantic concepts by only sketching two images and also an unsupervised strategy.
arXiv Detail & Related papers (2020-04-27T20:43:20Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.