ViT-Up: Faithful Feature Upsampling for Vision Transformers

• URL: http://arxiv.org/abs/2606.14024v1
• Date: Fri, 12 Jun 2026 01:55:17 GMT
• Title: ViT-Up: Faithful Feature Upsampling for Vision Transformers
• Abstract summary: Vision Transformers (ViTs) have become a dominant architecture for visual representation learning.<n>ViTs are commonly operated on relatively small patch-token grids due to the quadratic cost of global self-attention.<n>We introduce ViT-Up, a framework that replaces external image guidance with layer-wise query construction from intermediate ViT hidden states.
• Score: 19.84545321813943
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Vision Transformers (ViTs) have become a dominant architecture for visual representation learning, providing exceptionally strong and broadly reusable backbone features. However, ViTs are commonly operated on relatively small patch-token grids due to the quadratic cost of global self-attention, which creates a persistent bottleneck for dense prediction tasks such as semantic segmentation and depth estimation. This has motivated the development of task-agnostic feature upsamplers. While recent state-of-the-art methods produce visually sharp dense representations, their reliance on shallow image encoders for guided upsampling can introduce feature leakage, fragmentation, and blur. We introduce ViT-Up, an implicit feature upsampling framework that replaces external image guidance with layer-wise query construction from intermediate ViT hidden states. This enables feature prediction at arbitrary continuous image coordinates while preserving alignment with the backbone feature space. Experiments demonstrate that ViT-Up consistently outperforms state-of-the-art image-guided upsamplers across dense prediction and semantic correspondence. On DINOv3-S+, ViT-Up improves over prior methods by up to +2.07 mIoU on Cityscapes and +4.17 PCK@0.10 on SPair-71k. With the larger DINOv3-B backbone, these gains increase to +3.36 mIoU and +8.09 PCK@0.10, demonstrating that ViT-Up scales favorably with backbone capacity.

Related papers

• Same Semantics, Different Paths: Self-Improving Alignment for Vision-Text Compression [7.944386069136441]
  Vision-Text Compression (VTC) renders long texts into images and encodes them through the vision encoder (ViT)<n>ViT captures visual attributes (glyphs, font sizes, layout) rather than linguistic semantics, causing rendered-image representations to diverge from native-text representations.<n>We propose SPIRAL, a self-supervised alignment framework that closes this gap using only the model's own text-path behavior as supervision.
  arXiv  Detail & Related papers  (2026-08-03T12:08:56Z)
• PointSplat: Compact Gaussian Splatting via Human-Centric Prediction [70.45838533838104]
  We propose PointSplat, a novel human-centric approach to producing 3D human representations from input views on the fly.<n>The proposed method first estimates a coarse geometric proxy and performs ray casting to prune redundant points.<n>It then employs a Point-Image Transformer to fuse appearance and geometry features, predicting Gaussian attributes in a single forward pass.
  arXiv  Detail & Related papers  (2026-06-30T17:59:05Z)
• Beyond Static Frames: Temporal Aggregate-and-Restore Vision Transformer for Human Pose Estimation [7.63512442757446]
  Vision Transformers (ViTs) have recently achieved state-of-the-art performance in 2D human pose estimation.<n> TAR-ViTPose enhances static ViT representations by aggregating temporal cues across frames in a plug-and-play manner.<n>Our approach outperforms existing state-of-the-art video-based methods, while also achieving a noticeably higher runtime frame rate in real-world applications.
  arXiv  Detail & Related papers  (2026-03-06T05:45:04Z)
• One-Shot Refiner: Boosting Feed-forward Novel View Synthesis via One-Step Diffusion [57.824020826432815]
  We present a novel framework for high-fidelity novel view synthesis (NVS) from sparse images.<n>We design a Dual-Domain Detail Perception Module, which enables handling high-resolution images without being limited by the ViT backbone.<n>We develop a feature-guided diffusion network, which can preserve high-frequency details during the restoration process.
  arXiv  Detail & Related papers  (2026-01-20T17:11:55Z)
• Denoising Vision Transformers [43.03068202384091]
  We propose a two-stage denoising approach, termed Denoising Vision Transformers (DVT) In the first stage, we separate the clean features from those contaminated by positional artifacts by enforcing cross-view feature consistency with neural fields on a per-image basis. In the second stage, we train a lightweight transformer block to predict clean features from raw ViT outputs, leveraging the derived estimates of the clean features as supervision.
  arXiv  Detail & Related papers  (2024-01-05T18:59:52Z)
• CAIT: Triple-Win Compression towards High Accuracy, Fast Inference, and Favorable Transferability For ViTs [89.79139531731637]
  Vision Transformers (ViTs) have emerged as state-of-the-art models for various vision tasks.<n>We propose a joint underlinecompression method for ViTs that achieves a harmonious blend of high underlineaccuracy, fast underlineinference speed, and favorable underlinetransferability to downstream tasks.
  arXiv  Detail & Related papers  (2023-09-27T16:12:07Z)
• RangeViT: Towards Vision Transformers for 3D Semantic Segmentation in Autonomous Driving [80.14669385741202]
  Vision transformers (ViTs) have achieved state-of-the-art results in many image-based benchmarks. ViTs are notoriously hard to train and require a lot of training data to learn powerful representations. We show that our method, called RangeViT, outperforms existing projection-based methods on nuScenes and Semantic KITTI.
  arXiv  Detail & Related papers  (2023-01-24T18:50:48Z)
• Intriguing Properties of Vision Transformers [114.28522466830374]
  Vision transformers (ViT) have demonstrated impressive performance across various machine vision problems. We systematically study this question via an extensive set of experiments and comparisons with a high-performing convolutional neural network (CNN) We show effective features of ViTs are due to flexible receptive and dynamic fields possible via the self-attention mechanism.
  arXiv  Detail & Related papers  (2021-05-21T17:59:18Z)
• Scalable Visual Transformers with Hierarchical Pooling [61.05787583247392]
  We propose a Hierarchical Visual Transformer (HVT) which progressively pools visual tokens to shrink the sequence length. It brings a great benefit by scaling dimensions of depth/width/resolution/patch size without introducing extra computational complexity. Our HVT outperforms the competitive baselines on ImageNet and CIFAR-100 datasets.
  arXiv  Detail & Related papers  (2021-03-19T03:55:58Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.