UniRefiner: Teaching Pre-trained ViTs to Self-Dispose Dross via Contrastive Register

• URL: http://arxiv.org/abs/2605.19622v1
• Date: Tue, 19 May 2026 10:00:14 GMT
• Title: UniRefiner: Teaching Pre-trained ViTs to Self-Dispose Dross via Contrastive Register
• Abstract summary: We argue that any token failing to encode location-aligned semantics should be treated as a spurious artifact.<n>We propose UniRefiner, a universal refinement framework that teaches pre-trained Vision Transformers to self-dispose of these artifacts.<n>Our method requires only a few epochs of fine-tuning on 5k images to refine diverse ViTs, including massive models like EVA-CLIP-8B and InternViT-6B.
• Score: 30.077265994339303
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Representation learning with Vision Transformers (ViTs) has advanced rapidly, yet the utility of large-scale models in spatially sensitive tasks is hindered by spurious tokens. Prior efforts to mitigate this have been limited, often defining these artifacts narrowly, for example, as simple high-norm outliers. We argue that this scope is insufficient. For dense prediction tasks, we posit that any token failing to encode location-aligned semantics should be treated as a spurious artifact. This broader definition reveals a more complex problem, leading us to systematically categorize and characterize three fundamental types of spurious tokens that corrupt spatial representations. Based on this comprehensive diagnosis, we propose UniRefiner, a universal refinement framework that teaches pre-trained ViTs to self-dispose of these artifacts. UniRefiner uses contrastive registers to explicitly isolate and redistribute spurious tokens via a dual objective: (i) it aligns image tokens with filtered regular tokens to preserve semantics, and (ii) it aligns register tokens with detected spurious tokens to capture the spurious signals. Our method requires only a few epochs of fine-tuning on ~5k images to refine diverse ViTs, including massive models like EVA-CLIP-8B and InternViT-6B. Experiments demonstrate consistent and significant improvements: notably, the refined EVA-CLIP-8B achieves 51.9\% mIoU on ADE20K (+9.4\%), surpassing specialized vision models like DINOv2 (49.1\%), while zero-shot segmentation accuracy improves by up to 22\%. UniRefiner unlocks the latent spatial potential of existing large-scale foundation models, paving the way for their broader application.

Related papers

• Learning Fingerprints for Medical Time Series with Redundancy-Constrained Information Maximization [11.233142317726971]
  We propose a novel framework that compresses a variable-length MedTS into a fixed-size set of $k$ latent Fingerprint Tokens.<n>Our architecture employs a cross-attention bottleneck to generate these tokens and is trained with a dual-objective function.<n>This approach produces a low-dimensional, interpretable, and sample-efficient representation, where each token is encouraged to capture an independent factor of variation.
  arXiv  Detail & Related papers  (2026-04-30T18:33:40Z)
• UniWeTok: An Unified Binary Tokenizer with Codebook Size $\mathit{2^{128}}$ for Unified Multimodal Large Language Model [50.68870074090426]
  We introduce UniWeTok, a unified discrete tokenizer for Unified Multimodal Large Language Models.<n>For training framework, we introduce Pre-Post Distillation and a Generative-Aware Prior to enhance the semantic extraction and generative prior of the discrete tokens.<n>We propose a three-stage training framework designed to enhance UniWeTok's adaptability cross various image resolutions and perception-sensitive scenarios.
  arXiv  Detail & Related papers  (2026-02-15T15:07:19Z)
• D2Pruner: Debiased Importance and Structural Diversity for MLLM Token Pruning [49.16227597771663]
  D2Pruner is a framework that combines debiased importance with a structural pruning mechanism.<n>It reduces FLOPs by 74.2% while retaining 99.2% of its original performance.<n>It marks a significant advancement with up to 63. 53% improvement over existing methods.
  arXiv  Detail & Related papers  (2025-12-22T14:42:31Z)
• Vision Transformers with Self-Distilled Registers [11.649023403110528]
  Post Hoc Registers (PH-Reg) is an efficient self-distillation method that integrates registers into an existing ViT without requiring additional labeled data and full retraining.<n>We show that our approach can effectively reduce the number of artifact tokens, improving the segmentation and depth prediction of the student ViT under zero-shot and linear probing.
  arXiv  Detail & Related papers  (2025-05-27T17:59:41Z)
• UniTok: A Unified Tokenizer for Visual Generation and Understanding [63.23796622553615]
  Visual generative and understanding models typically rely on distinct tokenizers to process images.<n>We introduce UniTok, a unified tokenizer featuring a novel multi-codebook quantization mechanism.<n>In terms of final performance, UniTok sets a new record of 0.38 rFID and 78.6% zero-shot accuracy on ImageNet.
  arXiv  Detail & Related papers  (2025-02-27T17:47:01Z)
• Leveraging Registers in Vision Transformers for Robust Adaptation [36.30125926550378]
  Vision Transformers (ViTs) have shown success across a variety of tasks due to their ability to capture global image representations.<n>Recent studies have identified the existence of high-norm tokens in ViTs, which can interfere with unsupervised object discovery.<n>To address this, the use of "registers" which are additional tokens that isolate high norm patch tokens while capturing global image-level information has been proposed.
  arXiv  Detail & Related papers  (2025-01-08T19:02:32Z)
• AiluRus: A Scalable ViT Framework for Dense Prediction [95.1313839257891]
  Vision transformers (ViTs) have emerged as a prevalent architecture for vision tasks owing to their impressive performance. We propose to apply adaptive resolution for different regions in the image according to their importance. We evaluate our proposed method on three different datasets and observe promising performance.
  arXiv  Detail & Related papers  (2023-11-02T12:48:43Z)
• Self-Promoted Supervision for Few-Shot Transformer [178.52948452353834]
  Self-promoted sUpervisioN (SUN) is a few-shot learning framework for vision transformers (ViTs) SUN pretrains the ViT on the few-shot learning dataset and then uses it to generate individual location-specific supervision for guiding each patch token. Experiments show that SUN using ViTs significantly surpasses other few-shot learning frameworks with ViTs and is the first one that achieves higher performance than those CNN state-of-the-arts.
  arXiv  Detail & Related papers  (2022-03-14T12:53:27Z)
• On Improving Adversarial Transferability of Vision Transformers [97.17154635766578]
  Vision transformers (ViTs) process input images as sequences of patches via self-attention. We study the adversarial feature space of ViT models and their transferability. We introduce two novel strategies specific to the architecture of ViT models.
  arXiv  Detail & Related papers  (2021-06-08T08:20:38Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.