Vision Transformers for Face Recognition Need More Registers

• URL: http://arxiv.org/abs/2606.12036v1
• Date: Wed, 10 Jun 2026 12:58:23 GMT
• Title: Vision Transformers for Face Recognition Need More Registers
• Abstract summary: Recent advances in Vision Transformers (ViTs) for face recognition (FR) have moved beyond the standard CLS-based paradigm.<n>In this paradigm, a special classification token (token) is prepended to the patch embeddings and used as a representation of the input for downstream tasks.<n>We propose ConChenated Patch Embeddings (CPE), instead leveraging all patch tokens by concatenating them into a single vector, which is then projected into a compact face representation.
• Score: 16.250163286204593
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Recent advances in Vision Transformers (ViTs) for face recognition (FR) have moved beyond the standard CLS-token paradigm. In this paradigm, a special classification token (CLS) is prepended to the patch embeddings and used as a representation of the input for downstream tasks. An alternative approach, Concatenated Patch Embeddings (CPE), instead leverages all patch tokens by concatenating them into a single vector, which is then projected into a compact face representation. CPE has been shown to improve recognition performance in comparison to CLS-based ones, but our qualitative analysis of attention maps showed the presence of artifacts that limit their interpretability. To address this issue, we incorporate register tokens, learnable tokens concatenated to the initial patch embeddings, and processed jointly through the ViT encoder blocks. This mechanism has been shown to produce more structured and interpretable attention maps compared to baseline ViT. We empirically demonstrate that these artifacts consistently appear across various ViT backbones, including small and large models, and that introducing register tokens effectively mitigates them. Adding four or eight registers significantly enhances interpretability, with eight registers providing the highest verification accuracies and smoothest attention structures. Our resulting model, ViT-8R, corresponds to a CPE-based ViT-B architecture augmented with eight register tokens achieves state-of-the-art performance among ViT-based FR models on large-scale IJB-B and IJB-C benchmarks. Also, ViT-8R produces substantially clearer attention maps compared with the baseline model, which offer deeper insight into the model's attention behavior (https://github.com/TaharChettaoui/ViT-FR-Registers)

Related papers

• UniRefiner: Teaching Pre-trained ViTs to Self-Dispose Dross via Contrastive Register [30.077265994339303]
  We argue that any token failing to encode location-aligned semantics should be treated as a spurious artifact.<n>We propose UniRefiner, a universal refinement framework that teaches pre-trained Vision Transformers to self-dispose of these artifacts.<n>Our method requires only a few epochs of fine-tuning on 5k images to refine diverse ViTs, including massive models like EVA-CLIP-8B and InternViT-6B.
  arXiv  Detail & Related papers  (2026-05-19T10:00:14Z)
• Registers Matter for Pixel-Space Diffusion Transformers [22.836387612443037]
  Vision Transformers (ViTs) are known to exhibit high-norm patch-token outliers that degrade feature map quality.<n>We show that DiTs differ from ViTs in a key respect: they do not exhibit patch-token outliers.<n>We find that register tokens significantly improve convergence and generation quality of pixel-space DiTs.
  arXiv  Detail & Related papers  (2026-05-15T16:27:10Z)
• Taming Outlier Tokens in Diffusion Transformers [55.42341508886889]
  We study outlier tokens in Diffusion Transformers (DiTs) for image generation.<n>We show that this phenomenon appears in both the encoder and denoiser of modern Representation Autoencoder (RAE)-DiT pipelines.<n>We introduce Dual-Stage Registers (DSR), a register-based intervention for both components.
  arXiv  Detail & Related papers  (2026-05-06T17:59:42Z)
• SPOT: Sparsification with Attention Dynamics via Token Relevance in Vision Transformers [9.24617274894642]
  We present SParsification with attentiOn dynamics via Token relevance (SPOT) as a framework for early detection of redundant tokens within Vision Transformers (ViTs)<n>SPOT informs token sparsification and facilitates the elimination of such tokens, improving computational efficiency without sacrificing performance.<n> Empirical evaluations demonstrate significant efficiency gains of up to 40% compared to standard ViTs.
  arXiv  Detail & Related papers  (2025-11-13T16:56:24Z)
• Tensor Product Attention Is All You Need [61.3442269053374]
  Product Attention (TPA) is a novel attention mechanism that uses tensor decompositions to represent queries, keys, and values compactly.<n>TPA achieves improved model quality alongside memory efficiency.<n>Based on TPA, we introduce the ProducT ATTion Transformer (T6), a new model architecture for sequence modeling.
  arXiv  Detail & Related papers  (2025-01-11T03:37:10Z)
• Patch-level Representation Learning for Self-supervised Vision Transformers [68.8862419248863]
  Vision Transformers (ViTs) have gained much attention recently as a better architectural choice, often outperforming convolutional networks for various visual tasks. Inspired by this, we design a simple yet effective visual pretext task, coined SelfPatch, for learning better patch-level representations. We demonstrate that SelfPatch can significantly improve the performance of existing SSL methods for various visual tasks.
  arXiv  Detail & Related papers  (2022-06-16T08:01:19Z)
• PaCa-ViT: Learning Patch-to-Cluster Attention in Vision Transformers [9.63371509052453]
  This paper proposes to learn Patch-to-Cluster attention (PaCa) in Vision Transformers (ViT) The proposed PaCa module is used in designing efficient and interpretable ViT backbones and semantic segmentation head networks. It is significantly more efficient than PVT models in MS-COCO and MIT-ADE20k due to the linear complexity.
  arXiv  Detail & Related papers  (2022-03-22T18:28:02Z)
• Self-slimmed Vision Transformer [52.67243496139175]
  Vision transformers (ViTs) have become the popular structures and outperformed convolutional neural networks (CNNs) on various vision tasks. We propose a generic self-slimmed learning approach for vanilla ViTs, namely SiT. Specifically, we first design a novel Token Slimming Module (TSM), which can boost the inference efficiency of ViTs.
  arXiv  Detail & Related papers  (2021-11-24T16:48:57Z)
• PSViT: Better Vision Transformer via Token Pooling and Attention Sharing [114.8051035856023]
  We propose a PSViT: a ViT with token Pooling and attention Sharing to reduce the redundancy. Experimental results show that the proposed scheme can achieve up to 6.6% accuracy improvement in ImageNet classification.
  arXiv  Detail & Related papers  (2021-08-07T11:30:54Z)
• On Improving Adversarial Transferability of Vision Transformers [97.17154635766578]
  Vision transformers (ViTs) process input images as sequences of patches via self-attention. We study the adversarial feature space of ViT models and their transferability. We introduce two novel strategies specific to the architecture of ViT models.
  arXiv  Detail & Related papers  (2021-06-08T08:20:38Z)
• So-ViT: Mind Visual Tokens for Vision Transformer [27.243241133304785]
  We propose a new classification paradigm, where the second-order, cross-covariance pooling of visual tokens is combined with class token for final classification. We develop a light-weight, hierarchical module based on off-the-shelf convolutions for visual token embedding. The results show our models, when trained from scratch, outperform the competing ViT variants, while being on par with or better than state-of-the-art CNN models.
  arXiv  Detail & Related papers  (2021-04-22T09:05:09Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.