Registers Matter for Pixel-Space Diffusion Transformers

• URL: http://arxiv.org/abs/2605.16147v1
• Date: Fri, 15 May 2026 16:27:10 GMT
• Title: Registers Matter for Pixel-Space Diffusion Transformers
• Abstract summary: Vision Transformers (ViTs) are known to exhibit high-norm patch-token outliers that degrade feature map quality.<n>We show that DiTs differ from ViTs in a key respect: they do not exhibit patch-token outliers.<n>We find that register tokens significantly improve convergence and generation quality of pixel-space DiTs.
• Score: 22.836387612443037
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Vision Transformers (ViTs) are known to exhibit high-norm patch-token outliers that degrade feature map quality, a problem effectively mitigated by \textit{register tokens}. As diffusion models increasingly adopt transformer architectures and move toward pixel-space training, they become closer in form to ViTs, raising the question of whether register tokens are also useful for Diffusion Transformers (DiTs). In this work, we show that DiTs differ from ViTs in a key respect: they do not exhibit patch-token outliers. Interestingly, register tokens significantly improve convergence and generation quality of pixel-space DiTs. By analyzing intermediate representations, we find that register tokens produce cleaner feature maps at high noise levels, which may contribute to their effectiveness in pixel-space generation. We further observe that recent pixel-space DiT architectures implicitly incorporate register-like mechanisms, which may partially account for their strong empirical performance. Motivated by these insights, we investigate a parameter-efficient dual-stream architecture that specializes processing for register tokens and improves pixel-space generation quality with negligible runtime overhead.

Related papers

• Vision Transformers for Face Recognition Need More Registers [16.250163286204593]
  Recent advances in Vision Transformers (ViTs) for face recognition (FR) have moved beyond the standard CLS-based paradigm.<n>In this paradigm, a special classification token (token) is prepended to the patch embeddings and used as a representation of the input for downstream tasks.<n>We propose ConChenated Patch Embeddings (CPE), instead leveraging all patch tokens by concatenating them into a single vector, which is then projected into a compact face representation.
  arXiv  Detail & Related papers  (2026-06-10T12:58:23Z)
• Taming Outlier Tokens in Diffusion Transformers [55.42341508886889]
  We study outlier tokens in Diffusion Transformers (DiTs) for image generation.<n>We show that this phenomenon appears in both the encoder and denoiser of modern Representation Autoencoder (RAE)-DiT pipelines.<n>We introduce Dual-Stage Registers (DSR), a register-based intervention for both components.
  arXiv  Detail & Related papers  (2026-05-06T17:59:42Z)
• ViT Registers and Fractal ViT [7.970267053011483]
  We invent and test a variant of large vision transformers (ViTs) called fractal ViT.<n>These models do not improve upon ViT with registers, highlighting the fact that these findings may be scale, domain, or application-specific.
  arXiv  Detail & Related papers  (2026-01-21T22:24:53Z)
• THAT: Token-wise High-frequency Augmentation Transformer for Hyperspectral Pansharpening [11.026691195419453]
  Transformer-based methods have demonstrated strong potential in hyperspectral pansharpening.<n>Their effectiveness is often limited by redundant token representations and a lack of multi-scale feature modeling.<n>We propose the Token-wise High-frequency Augmentation Transformer (THAT), a novel framework designed to enhance hyperspectral pansharpening.
  arXiv  Detail & Related papers  (2025-08-11T17:03:10Z)
• Superpixel Tokenization for Vision Transformers: Preserving Semantic Integrity in Visual Tokens [38.31045722878938]
  We propose to substitute the grid-based tokenization in Vision Transformer with superpixel tokenization.<n>Our approach, which exhibits strong compatibility with existing frameworks, enhances the accuracy and robustness of ViT on various downstream tasks.
  arXiv  Detail & Related papers  (2024-12-06T00:38:36Z)
• Sub-token ViT Embedding via Stochastic Resonance Transformers [51.12001699637727]
  Vision Transformer (ViT) architectures represent images as collections of high-dimensional vectorized tokens, each corresponding to a rectangular non-overlapping patch. We propose a training-free method inspired by "stochastic resonance" The resulting "Stochastic Resonance Transformer" (SRT) retains the rich semantic information of the original representation, but grounds it on a finer-scale spatial domain, partly mitigating the coarse effect of spatial tokenization.
  arXiv  Detail & Related papers  (2023-10-06T01:53:27Z)
• Accurate Image Restoration with Attention Retractable Transformer [50.05204240159985]
  We propose Attention Retractable Transformer (ART) for image restoration. ART presents both dense and sparse attention modules in the network. We conduct extensive experiments on image super-resolution, denoising, and JPEG compression artifact reduction tasks.
  arXiv  Detail & Related papers  (2022-10-04T07:35:01Z)
• Attribute Surrogates Learning and Spectral Tokens Pooling in Transformers for Few-shot Learning [50.95116994162883]
  Vision transformers have been thought of as a promising alternative to convolutional neural networks for visual recognition. This paper presents hierarchically cascaded transformers that exploit intrinsic image structures through spectral tokens pooling. HCTransformers surpass the DINO baseline by a large margin of 9.7% 5-way 1-shot accuracy and 9.17% 5-way 5-shot accuracy on miniImageNet.
  arXiv  Detail & Related papers  (2022-03-17T03:49:58Z)
• Less is More: Pay Less Attention in Vision Transformers [61.05787583247392]
  Less attention vIsion Transformer builds upon the fact that convolutions, fully-connected layers, and self-attentions have almost equivalent mathematical expressions for processing image patch sequences. The proposed LIT achieves promising performance on image recognition tasks, including image classification, object detection and instance segmentation.
  arXiv  Detail & Related papers  (2021-05-29T05:26:07Z)
• Intriguing Properties of Vision Transformers [114.28522466830374]
  Vision transformers (ViT) have demonstrated impressive performance across various machine vision problems. We systematically study this question via an extensive set of experiments and comparisons with a high-performing convolutional neural network (CNN) We show effective features of ViTs are due to flexible receptive and dynamic fields possible via the self-attention mechanism.
  arXiv  Detail & Related papers  (2021-05-21T17:59:18Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.