Decomposing and Interpreting Image Representations via Text in ViTs Beyond CLIP
- URL: http://arxiv.org/abs/2406.01583v1
- Date: Mon, 3 Jun 2024 17:58:43 GMT
- Title: Decomposing and Interpreting Image Representations via Text in ViTs Beyond CLIP
- Authors: Sriram Balasubramanian, Samyadeep Basu, Soheil Feizi,
- Abstract summary: We introduce a framework which can identify the roles of various components in arbitrary vision transformers (ViTs)
Specifically, we automate the decomposition of the final representation into contributions from different model components.
We also introduce a novel scoring function to rank components by their importance with respect to specific features.
- Score: 53.18562650350898
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: Recent works have explored how individual components of the CLIP-ViT model contribute to the final representation by leveraging the shared image-text representation space of CLIP. These components, such as attention heads and MLPs, have been shown to capture distinct image features like shape, color or texture. However, understanding the role of these components in arbitrary vision transformers (ViTs) is challenging. To this end, we introduce a general framework which can identify the roles of various components in ViTs beyond CLIP. Specifically, we (a) automate the decomposition of the final representation into contributions from different model components, and (b) linearly map these contributions to CLIP space to interpret them via text. Additionally, we introduce a novel scoring function to rank components by their importance with respect to specific features. Applying our framework to various ViT variants (e.g. DeiT, DINO, DINOv2, Swin, MaxViT), we gain insights into the roles of different components concerning particular image features.These insights facilitate applications such as image retrieval using text descriptions or reference images, visualizing token importance heatmaps, and mitigating spurious correlations.
Related papers
- Understanding the Effect of using Semantically Meaningful Tokens for Visual Representation Learning [41.81009725976217]
We provide semantically-meaningful visual tokens to transformer encoders within a vision-language pre-training framework.
We demonstrate notable improvements over ViTs in learned representation quality across text-to-image and image-to-text retrieval tasks.
arXiv Detail & Related papers (2024-05-26T01:46:22Z) - Improving Referring Image Segmentation using Vision-Aware Text Features [26.768147543628096]
We present VATEX to improve referring image segmentation by enhancing object and context understanding with Vision-Aware Text Feature.
Our method achieves a significant performance improvement on three benchmark datasets RefCOCO, RefCO+ and G-Ref. Code.
arXiv Detail & Related papers (2024-04-12T16:38:48Z) - Interpreting CLIP's Image Representation via Text-Based Decomposition [73.54377859089801]
We investigate the CLIP image encoder by analyzing how individual model components affect the final representation.
We decompose the image representation as a sum across individual image patches, model layers, and attention heads.
We use this understanding to remove spurious features from CLIP and to create a strong zero-shot image segmenter.
arXiv Detail & Related papers (2023-10-09T17:59:04Z) - Parts of Speech-Grounded Subspaces in Vision-Language Models [32.497303059356334]
We propose to separate representations of different visual modalities in CLIP's joint vision-language space.
We learn subspaces capturing variability corresponding to a specific part of speech, while minimising variability to the rest.
We show the proposed model additionally facilitates learning subspaces corresponding to specific visual appearances.
arXiv Detail & Related papers (2023-05-23T13:32:19Z) - Splicing ViT Features for Semantic Appearance Transfer [10.295754142142686]
We present a method for semantically transferring the visual appearance of one natural image to another.
Specifically, our goal is to generate an image in which objects in a source structure image are "painted" with the visual appearance of their semantically related objects in a target appearance image.
arXiv Detail & Related papers (2022-01-02T22:00:34Z) - CRIS: CLIP-Driven Referring Image Segmentation [71.56466057776086]
We propose an end-to-end CLIP-Driven Referring Image framework (CRIS)
CRIS resorts to vision-language decoding and contrastive learning for achieving the text-to-pixel alignment.
Our proposed framework significantly outperforms the state-of-the-art performance without any post-processing.
arXiv Detail & Related papers (2021-11-30T07:29:08Z) - Matching Visual Features to Hierarchical Semantic Topics for Image
Paragraph Captioning [50.08729005865331]
This paper develops a plug-and-play hierarchical-topic-guided image paragraph generation framework.
To capture the correlations between the image and text at multiple levels of abstraction, we design a variational inference network.
To guide the paragraph generation, the learned hierarchical topics and visual features are integrated into the language model.
arXiv Detail & Related papers (2021-05-10T06:55:39Z) - Dense Relational Image Captioning via Multi-task Triple-Stream Networks [95.0476489266988]
We introduce dense captioning, a novel task which aims to generate captions with respect to information between objects in a visual scene.
This framework is advantageous in both diversity and amount of information, leading to a comprehensive image understanding.
arXiv Detail & Related papers (2020-10-08T09:17:55Z) - Learning Physical Graph Representations from Visual Scenes [56.7938395379406]
Physical Scene Graphs (PSGs) represent scenes as hierarchical graphs with nodes corresponding intuitively to object parts at different scales, and edges to physical connections between parts.
PSGNet augments standard CNNs by including: recurrent feedback connections to combine low and high-level image information; graph pooling and vectorization operations that convert spatially-uniform feature maps into object-centric graph structures.
We show that PSGNet outperforms alternative self-supervised scene representation algorithms at scene segmentation tasks.
arXiv Detail & Related papers (2020-06-22T16:10:26Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.