SVD-ViT: Does SVD Make Vision Transformers Attend More to the Foreground?

• URL: http://arxiv.org/abs/2602.02765v2
• Date: Sun, 08 Feb 2026 14:29:35 GMT
• Title: SVD-ViT: Does SVD Make Vision Transformers Attend More to the Foreground?
• Authors: Haruhiko Murata, Kazuhiro Hotta,
• Abstract summary: Vision Transformers (ViT) have been established as large-scale foundation models.<n>We propose SVD-ViT, which prioritizes the learning of foreground features.<n> Experimental results demonstrate that our method improves classification accuracy and effectively learns informative foreground representations.
• Score: 17.159633200689225
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Vision Transformers (ViT) have been established as large-scale foundation models. However, because self-attention operates globally, they lack an explicit mechanism to distinguish foreground from background. As a result, ViT may learn unnecessary background features and artifacts, leading to degraded classification performance. To address this issue, we propose SVD-ViT, which leverages singular value decomposition (SVD) to prioritize the learning of foreground features. SVD-ViT consists of three components-\textbf{SPC module}, \textbf{SSVA}, and \textbf{ID-RSVD}-and suppresses task-irrelevant factors such as background noise and artifacts by extracting and aggregating singular vectors that capture object foreground information. Experimental results demonstrate that our method improves classification accuracy and effectively learns informative foreground representations while reducing the impact of background noise.

Related papers

• Which Direction to Choose? An Analysis on the Representation Power of Self-Supervised ViTs in Downstream Tasks [43.473390101413166]
  Self-Supervised Learning for Vision Transformers (ViTs) has recently demonstrated considerable potential as a pre-training strategy for a variety of computer vision tasks.<n>This study aims to bridge the gap by systematically evaluating the use of unmodified features across image classification and segmentation tasks.
  arXiv  Detail & Related papers  (2025-09-18T11:46:07Z)
• LetheViT: Selective Machine Unlearning for Vision Transformers via Attention-Guided Contrastive Learning [8.104991333199264]
  Vision Transformers (ViTs) have revolutionized computer vision tasks with their exceptional performance.<n>This work addresses the particularly challenging scenario of random data forgetting in ViTs.<n>We propose LetheViT, a contrastive unlearning method tailored for ViTs.
  arXiv  Detail & Related papers  (2025-08-03T03:37:31Z)
• Exploring Self-Supervised Vision Transformers for Deepfake Detection: A Comparative Analysis [38.074487843137064]
  This paper investigates the effectiveness of self-supervised pre-trained vision transformers (ViTs) compared to supervised pre-trained ViTs and conventional neural networks (ConvNets) for detecting facial deepfake images and videos. It examines their potential for improved generalization and explainability, especially with limited training data. By leveraging SSL ViTs for deepfake detection with modest data and partial fine-tuning, we find comparable adaptability to deepfake detection and explainability via the attention mechanism.
  arXiv  Detail & Related papers  (2024-05-01T07:16:49Z)
• Denoising Vision Transformers [43.03068202384091]
  We propose a two-stage denoising approach, termed Denoising Vision Transformers (DVT) In the first stage, we separate the clean features from those contaminated by positional artifacts by enforcing cross-view feature consistency with neural fields on a per-image basis. In the second stage, we train a lightweight transformer block to predict clean features from raw ViT outputs, leveraging the derived estimates of the clean features as supervision.
  arXiv  Detail & Related papers  (2024-01-05T18:59:52Z)
• VST++: Efficient and Stronger Visual Saliency Transformer [74.26078624363274]
  We develop an efficient and stronger VST++ model to explore global long-range dependencies. We evaluate our model across various transformer-based backbones on RGB, RGB-D, and RGB-T SOD benchmark datasets.
  arXiv  Detail & Related papers  (2023-10-18T05:44:49Z)
• Generalized Face Forgery Detection via Adaptive Learning for Pre-trained Vision Transformer [54.32283739486781]
  We present a textbfForgery-aware textbfAdaptive textbfVision textbfTransformer (FA-ViT) under the adaptive learning paradigm. FA-ViT achieves 93.83% and 78.32% AUC scores on Celeb-DF and DFDC datasets in the cross-dataset evaluation.
  arXiv  Detail & Related papers  (2023-09-20T06:51:11Z)
• SGDViT: Saliency-Guided Dynamic Vision Transformer for UAV Tracking [12.447854608181833]
  This work presents a novel saliency-guided dynamic vision Transformer (SGDViT) for UAV tracking. The proposed method designs a new task-specific object saliency mining network to refine the cross-correlation operation. A lightweight saliency filtering Transformer further refines saliency information and increases the focus on appearance information.
  arXiv  Detail & Related papers  (2023-03-08T05:01:00Z)
• Self-Promoted Supervision for Few-Shot Transformer [178.52948452353834]
  Self-promoted sUpervisioN (SUN) is a few-shot learning framework for vision transformers (ViTs) SUN pretrains the ViT on the few-shot learning dataset and then uses it to generate individual location-specific supervision for guiding each patch token. Experiments show that SUN using ViTs significantly surpasses other few-shot learning frameworks with ViTs and is the first one that achieves higher performance than those CNN state-of-the-arts.
  arXiv  Detail & Related papers  (2022-03-14T12:53:27Z)
• PreViTS: Contrastive Pretraining with Video Tracking Supervision [53.73237606312024]
  PreViTS is an unsupervised SSL framework for selecting clips containing the same object. PreViTS spatially constrains the frame regions to learn from and trains the model to locate meaningful objects. We train a momentum contrastive (MoCo) encoder on VGG-Sound and Kinetics-400 datasets with PreViTS.
  arXiv  Detail & Related papers  (2021-12-01T19:49:57Z)
• Vision Transformers are Robust Learners [65.91359312429147]
  We study the robustness of the Vision Transformer (ViT) against common corruptions and perturbations, distribution shifts, and natural adversarial examples. We present analyses that provide both quantitative and qualitative indications to explain why ViTs are indeed more robust learners.
  arXiv  Detail & Related papers  (2021-05-17T02:39:22Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.