ViT Registers and Fractal ViT

• URL: http://arxiv.org/abs/2601.15506v1
• Date: Wed, 21 Jan 2026 22:24:53 GMT
• Title: ViT Registers and Fractal ViT
• Abstract summary: We invent and test a variant of large vision transformers (ViTs) called fractal ViT.<n>These models do not improve upon ViT with registers, highlighting the fact that these findings may be scale, domain, or application-specific.
• Score: 7.970267053011483
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Drawing inspiration from recent findings including surprisingly decent performance of transformers without positional encoding (NoPE) in the domain of language models and how registers (additional throwaway tokens not tied to input) may improve the performance of large vision transformers (ViTs), we invent and test a variant of ViT called fractal ViT that breaks permutation invariance among the tokens by applying an attention mask between the regular tokens and ``summary tokens'' similar to registers, in isolation or in combination with various positional encodings. These models do not improve upon ViT with registers, highlighting the fact that these findings may be scale, domain, or application-specific.

Related papers

• Vision Transformers for Face Recognition Need More Registers [16.250163286204593]
  Recent advances in Vision Transformers (ViTs) for face recognition (FR) have moved beyond the standard CLS-based paradigm.<n>In this paradigm, a special classification token (token) is prepended to the patch embeddings and used as a representation of the input for downstream tasks.<n>We propose ConChenated Patch Embeddings (CPE), instead leveraging all patch tokens by concatenating them into a single vector, which is then projected into a compact face representation.
  arXiv  Detail & Related papers  (2026-06-10T12:58:23Z)
• Registers Matter for Pixel-Space Diffusion Transformers [22.836387612443037]
  Vision Transformers (ViTs) are known to exhibit high-norm patch-token outliers that degrade feature map quality.<n>We show that DiTs differ from ViTs in a key respect: they do not exhibit patch-token outliers.<n>We find that register tokens significantly improve convergence and generation quality of pixel-space DiTs.
  arXiv  Detail & Related papers  (2026-05-15T16:27:10Z)
• Taming Outlier Tokens in Diffusion Transformers [55.42341508886889]
  We study outlier tokens in Diffusion Transformers (DiTs) for image generation.<n>We show that this phenomenon appears in both the encoder and denoiser of modern Representation Autoencoder (RAE)-DiT pipelines.<n>We introduce Dual-Stage Registers (DSR), a register-based intervention for both components.
  arXiv  Detail & Related papers  (2026-05-06T17:59:42Z)
• WriteViT: Handwritten Text Generation with Vision Transformer [7.10052009802944]
  We introduce WriteViT, a one-shot handwritten text synthesis framework that incorporates Vision Transformers (ViT)<n>WriteViT produces high-quality, style-consistent handwriting while maintaining strong recognition performance in low-resource scenarios.<n>These results highlight the promise of transformer-based designs for multilingual handwriting generation and efficient style adaptation.
  arXiv  Detail & Related papers  (2025-05-19T15:17:53Z)
• VORTEX: Challenging CNNs at Texture Recognition by using Vision Transformers with Orderless and Randomized Token Encodings [1.6594406786473057]
  Vision Transformers (ViTs) were introduced a few years ago, but little is known about their texture recognition ability.<n>We introduce VORTEX, a novel method that enables the effective use of ViTs for texture analysis.<n>We evaluate VORTEX on nine diverse texture datasets, demonstrating its ability to achieve or surpass SOTA performance.
  arXiv  Detail & Related papers  (2025-03-09T00:36:02Z)
• MADTP: Multimodal Alignment-Guided Dynamic Token Pruning for Accelerating Vision-Language Transformer [66.71930982549028]
  Vision-Language Transformers (VLTs) have shown great success recently, but are accompanied by heavy computation costs. We propose a novel framework named Multimodal Alignment-Guided Dynamic Token Pruning (MADTP) for accelerating various VLTs.
  arXiv  Detail & Related papers  (2024-03-05T14:13:50Z)
• Multi-Tailed Vision Transformer for Efficient Inference [44.43126137573205]
  Vision Transformer (ViT) has achieved promising performance in image recognition. We propose a Multi-Tailed Vision Transformer (MT-ViT) in the paper. MT-ViT adopts multiple tails to produce visual sequences of different lengths for the following Transformer encoder.
  arXiv  Detail & Related papers  (2022-03-03T09:30:55Z)
• Self-slimmed Vision Transformer [52.67243496139175]
  Vision transformers (ViTs) have become the popular structures and outperformed convolutional neural networks (CNNs) on various vision tasks. We propose a generic self-slimmed learning approach for vanilla ViTs, namely SiT. Specifically, we first design a novel Token Slimming Module (TSM), which can boost the inference efficiency of ViTs.
  arXiv  Detail & Related papers  (2021-11-24T16:48:57Z)
• Discrete Representations Strengthen Vision Transformer Robustness [43.821734467553554]
  Vision Transformer (ViT) is emerging as the state-of-the-art architecture for image recognition. We present a simple and effective architecture modification to ViT's input layer by adding discrete tokens produced by a vector-quantized encoder. Experimental results demonstrate that adding discrete representation on four architecture variants strengthens ViT robustness by up to 12% across seven ImageNet robustness benchmarks.
  arXiv  Detail & Related papers  (2021-11-20T01:49:56Z)
• Vision Transformer with Progressive Sampling [73.60630716500154]
  We propose an iterative and progressive sampling strategy to locate discriminative regions. When trained from scratch on ImageNet, PS-ViT performs 3.8% higher than the vanilla ViT in terms of top-1 accuracy.
  arXiv  Detail & Related papers  (2021-08-03T18:04:31Z)
• On Improving Adversarial Transferability of Vision Transformers [97.17154635766578]
  Vision transformers (ViTs) process input images as sequences of patches via self-attention. We study the adversarial feature space of ViT models and their transferability. We introduce two novel strategies specific to the architecture of ViT models.
  arXiv  Detail & Related papers  (2021-06-08T08:20:38Z)
• Intriguing Properties of Vision Transformers [114.28522466830374]
  Vision transformers (ViT) have demonstrated impressive performance across various machine vision problems. We systematically study this question via an extensive set of experiments and comparisons with a high-performing convolutional neural network (CNN) We show effective features of ViTs are due to flexible receptive and dynamic fields possible via the self-attention mechanism.
  arXiv  Detail & Related papers  (2021-05-21T17:59:18Z)
• Vision Transformers are Robust Learners [65.91359312429147]
  We study the robustness of the Vision Transformer (ViT) against common corruptions and perturbations, distribution shifts, and natural adversarial examples. We present analyses that provide both quantitative and qualitative indications to explain why ViTs are indeed more robust learners.
  arXiv  Detail & Related papers  (2021-05-17T02:39:22Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.