DiveUp: Learning Feature Upsampling from Diverse Vision Foundation Models

• URL: http://arxiv.org/abs/2603.13571v1
• Date: Fri, 13 Mar 2026 20:20:22 GMT
• Title: DiveUp: Learning Feature Upsampling from Diverse Vision Foundation Models
• Abstract summary: DiveUp is a novel framework that breaks away from single-model dependency by introducing multi-VFM relational guidance.<n>We propose a universal relational feature representation, formulated as a local center-of-mass (COM) field, that extracts intrinsic geometric structures.<n>DiveUp is a unified, encoder-agnostic framework; a jointly-trained model can universally upsample features from diverse VFMs.
• Score: 11.659303031631723
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Recently, feature upsampling has gained increasing attention owing to its effectiveness in enhancing vision foundation models (VFMs) for pixel-level understanding tasks. Existing methods typically rely on high-resolution features from the same foundation model to achieve upsampling via self-reconstruction. However, relying solely on intra-model features forces the upsampler to overfit to the source model's inherent location misalignment and high-norm artifacts. To address this fundamental limitation, we propose DiveUp, a novel framework that breaks away from single-model dependency by introducing multi-VFM relational guidance. Instead of naive feature fusion, DiveUp leverages diverse VFMs as a panel of experts, utilizing their structural consensus to regularize the upsampler's learning process, effectively preventing the propagation of inaccurate spatial structures from the source model. To reconcile the unaligned feature spaces across different VFMs, we propose a universal relational feature representation, formulated as a local center-of-mass (COM) field, that extracts intrinsic geometric structures, enabling seamless cross-model interaction. Furthermore, we introduce a spikiness-aware selection strategy that evaluates the spatial reliability of each VFM, effectively filtering out high-norm artifacts to aggregate guidance from only the most reliable expert at each local region. DiveUp is a unified, encoder-agnostic framework; a jointly-trained model can universally upsample features from diverse VFMs without requiring per-model retraining. Extensive experiments demonstrate that DiveUp achieves state-of-the-art performance across various downstream dense prediction tasks, validating the efficacy of multi-expert relational guidance. Our code and models are available at: https://github.com/Xiaoqiong-Liu/DiveUp

Related papers

• FoundationSLAM: Unleashing the Power of Depth Foundation Models for End-to-End Dense Visual SLAM [50.9765003472032]
  FoundationSLAM is a learning-based monocular dense SLAM system for accurate and robust tracking and mapping.<n>Our core idea is to bridge flow estimation with reasoning by leveraging the guidance from foundation depth models.
  arXiv  Detail & Related papers  (2025-12-31T17:57:45Z)
• Rethinking Infrared Small Target Detection: A Foundation-Driven Efficient Paradigm [17.63632082331749]
  Large-scale visual foundation models (VFMs) exhibit strong generalization across diverse visual domains, but their potential for single-frame infrared small target (SIRST) detection remains largely unexplored.<n>We propose a Foundation-Driven Efficient Paradigm (FDEP) which can seamlessly adapt to existing encoder-decoder-based methods and significantly improve accuracy without additional inference overhead.
  arXiv  Detail & Related papers  (2025-12-05T08:12:35Z)
• NAF: Zero-Shot Feature Upsampling via Neighborhood Attention Filtering [80.55691420311616]
  Neighborhood Attention Filtering (NAF) learns adaptive spatial-and-content weights through Cross-Scale Neighborhood Attention and Rotary Position Embeddings (RoPE)<n>NAF operates zero-shot: it upsamples features from any Vision Foundation Models (VFMs) without retraining.<n>It maintains high efficiency, scaling to 2K feature maps and reconstructing intermediate-resolution maps at 18 FPS.
  arXiv  Detail & Related papers  (2025-11-23T13:43:52Z)
• Beyond Boundaries: Leveraging Vision Foundation Models for Source-Free Object Detection [34.292554427633505]
  Source-Free Object Detection (SFOD) aims to adapt a source-pretrained object detector to a target domain without access to source data.<n> Vision Foundation Models (VFMs), pretrained on massive and diverse data, exhibit strong perception capabilities and broad generalization.<n>We propose a novel SFOD framework that leverages VFMs as external knowledge sources to jointly enhance feature alignment and label quality.
  arXiv  Detail & Related papers  (2025-11-10T17:06:01Z)
• Consistent World Models via Foresight Diffusion [56.45012929930605]
  We argue that a key bottleneck in learning consistent diffusion-based world models lies in the suboptimal predictive ability.<n>We propose Foresight Diffusion (ForeDiff), a diffusion-based world modeling framework that enhances consistency by decoupling condition understanding from target denoising.
  arXiv  Detail & Related papers  (2025-05-22T10:01:59Z)
• Robust Federated Learning on Edge Devices with Domain Heterogeneity [13.362209980631876]
  Federated Learning (FL) allows collaborative training while ensuring data privacy across distributed edge devices.<n>We introduce a new framework to address this challenge by improving the generalization ability of the FL global model.<n>We introduce FedAPC, a prototype-based FL framework designed to enhance feature diversity and model robustness.
  arXiv  Detail & Related papers  (2025-05-15T09:53:14Z)
• Benchmarking Feature Upsampling Methods for Vision Foundation Models using Interactive Segmentation [24.531539125814877]
  Vision Foundation Models (VFMs) are large-scale, pre-trained models that serve as general-purpose backbones for various computer vision tasks.<n>One way to tackle this limitation is by employing a task-agnostic feature upsampling module that refines VFM features resolution.<n>Our benchmarking experiments show that selecting appropriate upsampling strategies significantly improves VFM features quality.
  arXiv  Detail & Related papers  (2025-05-04T11:59:26Z)
• A Refreshed Similarity-based Upsampler for Direct High-Ratio Feature Upsampling [54.05517338122698]
  A popular similarity-based feature upsampling pipeline has been proposed, which utilizes a high-resolution feature as guidance.<n>We propose an explicitly controllable query-key feature alignment from both semantic-aware and detail-aware perspectives.<n>We develop a fine-grained neighbor selection strategy on HR features, which is simple yet effective for alleviating mosaic artifacts.
  arXiv  Detail & Related papers  (2024-07-02T14:12:21Z)
• Consistency Regularization for Generalizable Source-free Domain Adaptation [62.654883736925456]
  Source-free domain adaptation (SFDA) aims to adapt a well-trained source model to an unlabelled target domain without accessing the source dataset. Existing SFDA methods ONLY assess their adapted models on the target training set, neglecting the data from unseen but identically distributed testing sets. We propose a consistency regularization framework to develop a more generalizable SFDA method.
  arXiv  Detail & Related papers  (2023-08-03T07:45:53Z)
• T1: Scaling Diffusion Probabilistic Fields to High-Resolution on Unified Visual Modalities [69.16656086708291]
  Diffusion Probabilistic Field (DPF) models the distribution of continuous functions defined over metric spaces. We propose a new model comprising of a view-wise sampling algorithm to focus on local structure learning. The model can be scaled to generate high-resolution data while unifying multiple modalities.
  arXiv  Detail & Related papers  (2023-05-24T03:32:03Z)
• Global Relation Modeling and Refinement for Bottom-Up Human Pose Estimation [4.24515544235173]
  We propose a convolutional neural network for bottom-up human pose estimation. Our model has the ability to focus on different granularity from local to global regions. Our results on the COCO and CrowdPose datasets demonstrate that it is an efficient framework for multi-person pose estimation.
  arXiv  Detail & Related papers  (2023-03-27T02:54:08Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.