Leveraging Registers in Vision Transformers for Robust Adaptation

• URL: http://arxiv.org/abs/2501.04784v1
• Date: Wed, 08 Jan 2025 19:02:32 GMT
• Title: Leveraging Registers in Vision Transformers for Robust Adaptation
• Abstract summary: Vision Transformers (ViTs) have shown success across a variety of tasks due to their ability to capture global image representations.<n>Recent studies have identified the existence of high-norm tokens in ViTs, which can interfere with unsupervised object discovery.<n>To address this, the use of "registers" which are additional tokens that isolate high norm patch tokens while capturing global image-level information has been proposed.
• Score: 36.30125926550378
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Vision Transformers (ViTs) have shown success across a variety of tasks due to their ability to capture global image representations. Recent studies have identified the existence of high-norm tokens in ViTs, which can interfere with unsupervised object discovery. To address this, the use of "registers" which are additional tokens that isolate high norm patch tokens while capturing global image-level information has been proposed. While registers have been studied extensively for object discovery, their generalization properties particularly in out-of-distribution (OOD) scenarios, remains underexplored. In this paper, we examine the utility of register token embeddings in providing additional features for improving generalization and anomaly rejection. To that end, we propose a simple method that combines the special CLS token embedding commonly employed in ViTs with the average-pooled register embeddings to create feature representations which are subsequently used for training a downstream classifier. We find that this enhances OOD generalization and anomaly rejection, while maintaining in-distribution (ID) performance. Extensive experiments across multiple ViT backbones trained with and without registers reveal consistent improvements of 2-4\% in top-1 OOD accuracy and a 2-3\% reduction in false positive rates for anomaly detection. Importantly, these gains are achieved without additional computational overhead.

Related papers

• Test-Time Registers as Global Priors for Tokenized Image Generation [25.626464690706484]
  We show that testtime register features exhibit stronger low-frequency concentration than both pixel-space readouts and patch-mean features.<n>Motivated by these diagnostics, we introduce RegToken, a training procedure that converts register structure into a small set of global prior tokens.
  arXiv  Detail & Related papers  (2026-07-18T13:56:19Z)
• UniRefiner: Teaching Pre-trained ViTs to Self-Dispose Dross via Contrastive Register [30.077265994339303]
  We argue that any token failing to encode location-aligned semantics should be treated as a spurious artifact.<n>We propose UniRefiner, a universal refinement framework that teaches pre-trained Vision Transformers to self-dispose of these artifacts.<n>Our method requires only a few epochs of fine-tuning on 5k images to refine diverse ViTs, including massive models like EVA-CLIP-8B and InternViT-6B.
  arXiv  Detail & Related papers  (2026-05-19T10:00:14Z)
• Taming Outlier Tokens in Diffusion Transformers [55.42341508886889]
  We study outlier tokens in Diffusion Transformers (DiTs) for image generation.<n>We show that this phenomenon appears in both the encoder and denoiser of modern Representation Autoencoder (RAE)-DiT pipelines.<n>We introduce Dual-Stage Registers (DSR), a register-based intervention for both components.
  arXiv  Detail & Related papers  (2026-05-06T17:59:42Z)
• Exploiting Inter-Sample Information for Long-tailed Out-of-Distribution Detection [7.0229899259286945]
  We show that exploiting inter-sample relationships can significantly improve OOD detection in long-tailed recognition of vision datasets.<n>Our method outperforms the state-of-the-art approaches by a large margin in terms of FPR and tail-class ID classification accuracy.
  arXiv  Detail & Related papers  (2025-11-20T03:31:37Z)
• Vision Transformers Don't Need Trained Registers [17.412430704896455]
  A sparse set of neurons is responsible for concentrating high-norm activations on outlier tokens.<n>We create a training-free approach to mitigate these artifacts.<n>Our results suggest that test-time registers effectively take on the role of register tokens at test-time.
  arXiv  Detail & Related papers  (2025-06-09T17:59:57Z)
• Learning Binarized Representations with Pseudo-positive Sample Enhancement for Efficient Graph Collaborative Filtering [35.82405808653398]
  We study the problem of graph representation binarization for efficient collaborative filtering.<n>Our findings indicate that explicitly mitigating information loss at various stages of embedding binarization has a significant positive impact on performance.<n>Compared to its predecessor BiGeaR, BiGeaR++ introduces a fine-grained inference distillation mechanism and an effective embedding sample synthesis approach.
  arXiv  Detail & Related papers  (2025-06-03T11:11:43Z)
• SINDER: Repairing the Singular Defects of DINOv2 [61.98878352956125]
  Vision Transformer models trained on large-scale datasets often exhibit artifacts in the patch token they extract. We propose a novel fine-tuning smooth regularization that rectifies structural deficiencies using only a small dataset.
  arXiv  Detail & Related papers  (2024-07-23T20:34:23Z)
• ADFQ-ViT: Activation-Distribution-Friendly Post-Training Quantization for Vision Transformers [7.155242379236052]
  Quantization of Vision Transformers (ViTs) has emerged as a promising solution to mitigate these challenges. Existing methods still suffer from significant accuracy loss at low-bit. ADFQ-ViT provides significant improvements over various baselines in image classification, object detection, and instance segmentation tasks at 4-bit.
  arXiv  Detail & Related papers  (2024-07-03T02:41:59Z)
• TPC-ViT: Token Propagation Controller for Efficient Vision Transformer [6.341420717393898]
  Vision transformers (ViTs) have achieved promising results on a variety of Computer Vision tasks. Previous approaches that employ gradual token reduction to address this challenge assume that token redundancy in one layer implies redundancy in all the following layers. We propose a novel token propagation controller (TPC) that incorporates two different token-distributions.
  arXiv  Detail & Related papers  (2024-01-03T00:10:33Z)
• Generalized Face Forgery Detection via Adaptive Learning for Pre-trained Vision Transformer [54.32283739486781]
  We present a textbfForgery-aware textbfAdaptive textbfVision textbfTransformer (FA-ViT) under the adaptive learning paradigm. FA-ViT achieves 93.83% and 78.32% AUC scores on Celeb-DF and DFDC datasets in the cross-dataset evaluation.
  arXiv  Detail & Related papers  (2023-09-20T06:51:11Z)
• RegFormer: An Efficient Projection-Aware Transformer Network for Large-Scale Point Cloud Registration [73.69415797389195]
  We propose an end-to-end transformer network (RegFormer) for large-scale point cloud alignment. Specifically, a projection-aware hierarchical transformer is proposed to capture long-range dependencies and filter outliers. Our transformer has linear complexity, which guarantees high efficiency even for large-scale scenes.
  arXiv  Detail & Related papers  (2023-03-22T08:47:37Z)
• A Theoretical Understanding of Shallow Vision Transformers: Learning, Generalization, and Sample Complexity [71.11795737362459]
  ViTs with self-attention modules have recently achieved great empirical success in many tasks. However, theoretical learning generalization analysis is mostly noisy and elusive. This paper provides the first theoretical analysis of a shallow ViT for a classification task.
  arXiv  Detail & Related papers  (2023-02-12T22:12:35Z)
• Self-Supervised Training with Autoencoders for Visual Anomaly Detection [61.62861063776813]
  We focus on a specific use case in anomaly detection where the distribution of normal samples is supported by a lower-dimensional manifold. We adapt a self-supervised learning regime that exploits discriminative information during training but focuses on the submanifold of normal examples. We achieve a new state-of-the-art result on the MVTec AD dataset -- a challenging benchmark for visual anomaly detection in the manufacturing domain.
  arXiv  Detail & Related papers  (2022-06-23T14:16:30Z)
• No Shifted Augmentations (NSA): compact distributions for robust self-supervised Anomaly Detection [4.243926243206826]
  Unsupervised Anomaly detection (AD) requires building a notion of normalcy, distinguishing in-distribution (ID) and out-of-distribution (OOD) data. We investigate how the emph geometrical compactness of the ID feature distribution makes isolating and detecting outliers easier. We propose novel architectural modifications to the self-supervised feature learning step, that enable such compact distributions for ID data to be learned.
  arXiv  Detail & Related papers  (2022-03-19T15:55:32Z)
• Self-Promoted Supervision for Few-Shot Transformer [178.52948452353834]
  Self-promoted sUpervisioN (SUN) is a few-shot learning framework for vision transformers (ViTs) SUN pretrains the ViT on the few-shot learning dataset and then uses it to generate individual location-specific supervision for guiding each patch token. Experiments show that SUN using ViTs significantly surpasses other few-shot learning frameworks with ViTs and is the first one that achieves higher performance than those CNN state-of-the-arts.
  arXiv  Detail & Related papers  (2022-03-14T12:53:27Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.