Inverting Visual Representations with Detection Transformers

• URL: http://arxiv.org/abs/2412.06534v1
• Date: Mon, 09 Dec 2024 14:43:06 GMT
• Title: Inverting Visual Representations with Detection Transformers
• Authors: Jan Rathjens, Shirin Reyhanian, David Kappel, Laurenz Wiskott,
• Abstract summary: We apply the approach of training inverse models to reconstruct input images from intermediate layers within a Detection Transformer.<n>We demonstrate critical properties of Detection Transformers, including contextual shape robustness, inter-layer correlation, and preservation to color perturbations.
• Score: 0.8124699127636158
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Understanding the mechanisms underlying deep neural networks in computer vision remains a fundamental challenge. While many prior approaches have focused on visualizing intermediate representations within deep neural networks, particularly convolutional neural networks, these techniques have yet to be thoroughly explored in transformer-based vision models. In this study, we apply the approach of training inverse models to reconstruct input images from intermediate layers within a Detection Transformer, showing that this approach is efficient and feasible for transformer-based vision models. Through qualitative and quantitative evaluations of reconstructed images across model stages, we demonstrate critical properties of Detection Transformers, including contextual shape preservation, inter-layer correlation, and robustness to color perturbations, illustrating how these characteristics emerge within the model's architecture. Our findings contribute to a deeper understanding of transformer-based vision models. The code for reproducing our experiments will be made available at github.com/wiskott-lab/inverse-detection-transformer.

Related papers

• Image Segmentation with transformers: An Overview, Challenges and Future [0.0]
  This paper explores the shortcomings of CNN-based models and the shift towards transformer architectures. The paper discusses current challenges in transformer-based segmentation and outlines promising future trends.
  arXiv  Detail & Related papers  (2025-01-16T08:34:39Z)
• Causal Transformer for Fusion and Pose Estimation in Deep Visual Inertial Odometry [1.2289361708127877]
  We propose a causal visual-inertial fusion transformer (VIFT) for pose estimation in deep visual-inertial odometry. The proposed method is end-to-end trainable and requires only a monocular camera and IMU during inference.
  arXiv  Detail & Related papers  (2024-09-13T12:21:25Z)
• A Review of Transformer-Based Models for Computer Vision Tasks: Capturing Global Context and Spatial Relationships [0.5639904484784127]
  Transformer-based models have transformed the landscape of natural language processing (NLP) These models are renowned for their ability to capture long-range dependencies and contextual information. We discuss potential research directions and applications of transformer-based models in computer vision.
  arXiv  Detail & Related papers  (2024-08-27T16:22:18Z)
• Automatic Discovery of Visual Circuits [66.99553804855931]
  We explore scalable methods for extracting the subgraph of a vision model's computational graph that underlies recognition of a specific visual concept. We find that our approach extracts circuits that causally affect model output, and that editing these circuits can defend large pretrained models from adversarial attacks.
  arXiv  Detail & Related papers  (2024-04-22T17:00:57Z)
• T-TAME: Trainable Attention Mechanism for Explaining Convolutional Networks and Vision Transformers [9.284740716447342]
  "Black box" nature of neural networks is a barrier to adoption in applications where explainability is essential. This paper presents T-TAME, Transformer-compatible Trainable Attention Mechanism for Explanations. Proposed architecture and training technique can be easily applied to any convolutional or Vision Transformer-like neural network.
  arXiv  Detail & Related papers  (2024-03-07T14:25:03Z)
• Visual Analytics for Generative Transformer Models [28.251218916955125]
  We present a novel visual analytical framework to support the analysis of transformer-based generative networks. Our framework is one of the first dedicated to supporting the analysis of transformer-based encoder-decoder models.
  arXiv  Detail & Related papers  (2023-11-21T08:15:01Z)
• Rethinking Attention: Exploring Shallow Feed-Forward Neural Networks as an Alternative to Attention Layers in Transformers [5.356051655680145]
  This work presents an analysis of the effectiveness of using standard shallow feed-forward networks to mimic the behavior of the attention mechanism in the original Transformer model. We substitute key elements of the attention mechanism in the Transformer with simple feed-forward networks, trained using the original components via knowledge distillation. Our experiments, conducted on the IWSLT 2017 dataset, reveal the capacity of these "attentionless Transformers" to rival the performance of the original architecture.
  arXiv  Detail & Related papers  (2023-11-17T16:58:52Z)
• Blind Image Quality Assessment via Transformer Predicted Error Map and Perceptual Quality Token [19.67014524146261]
  No-reference image quality assessment (NR-IQA) has gained increasing attention recently. We propose a Transformer based NR-IQA model using a predicted objective error map and perceptual quality token. Our proposed method outperforms the current state-of-the-art in both authentic and synthetic image databases.
  arXiv  Detail & Related papers  (2023-05-16T11:17:54Z)
• AbHE: All Attention-based Homography Estimation [0.0]
  We propose a strong-baseline model based on the Swin Transformer, which combines convolution neural network for local features and transformer module for global features. In the homography regression stage, we adopt an attention layer for the channels of correlation volume, which can drop out some weak correlation feature points. The experiment shows that in 8 Degree-of-Freedoms(DOFs) homography estimation our method overperforms the state-of-the-art method.
  arXiv  Detail & Related papers  (2022-12-06T15:00:00Z)
• Vision Transformer with Convolutions Architecture Search [72.70461709267497]
  We propose an architecture search method-Vision Transformer with Convolutions Architecture Search (VTCAS) The high-performance backbone network searched by VTCAS introduces the desirable features of convolutional neural networks into the Transformer architecture. It enhances the robustness of the neural network for object recognition, especially in the low illumination indoor scene.
  arXiv  Detail & Related papers  (2022-03-20T02:59:51Z)
• Visformer: The Vision-friendly Transformer [105.52122194322592]
  We propose a new architecture named Visformer, which is abbreviated from the Vision-friendly Transformer' With the same computational complexity, Visformer outperforms both the Transformer-based and convolution-based models in terms of ImageNet classification accuracy.
  arXiv  Detail & Related papers  (2021-04-26T13:13:03Z)
• Transformers Solve the Limited Receptive Field for Monocular Depth Prediction [82.90445525977904]
  We propose TransDepth, an architecture which benefits from both convolutional neural networks and transformers. This is the first paper which applies transformers into pixel-wise prediction problems involving continuous labels.
  arXiv  Detail & Related papers  (2021-03-22T18:00:13Z)
• Transformers in Vision: A Survey [101.07348618962111]
  Transformers enable modeling long dependencies between input sequence elements and support parallel processing of sequence. Transformers require minimal inductive biases for their design and are naturally suited as set-functions. This survey aims to provide a comprehensive overview of the Transformer models in the computer vision discipline.
  arXiv  Detail & Related papers  (2021-01-04T18:57:24Z)
• A Survey on Visual Transformer [126.56860258176324]
  Transformer is a type of deep neural network mainly based on the self-attention mechanism. In this paper, we review these vision transformer models by categorizing them in different tasks and analyzing their advantages and disadvantages.
  arXiv  Detail & Related papers  (2020-12-23T09:37:54Z)
• SIR: Self-supervised Image Rectification via Seeing the Same Scene from Multiple Different Lenses [82.56853587380168]
  We propose a novel self-supervised image rectification (SIR) method based on an important insight that the rectified results of distorted images of the same scene from different lens should be the same. We leverage a differentiable warping module to generate the rectified images and re-distorted images from the distortion parameters. Our method achieves comparable or even better performance than the supervised baseline method and representative state-of-the-art methods.
  arXiv  Detail & Related papers  (2020-11-30T08:23:25Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.