Lightweight Prompt-Guided CLIP Adaptation for Monocular Depth Estimation

• URL: http://arxiv.org/abs/2604.01118v1
• Date: Wed, 01 Apr 2026 16:41:04 GMT
• Title: Lightweight Prompt-Guided CLIP Adaptation for Monocular Depth Estimation
• Abstract summary: MoA-DepthCLIP adapts pretrained CLIP representations for monocular depth estimation with minimal supervision.<n>Our method integrates a lightweight Mixture-of-Adapters (MoA) module into the pretrained Vision Transformer (ViT-B/32) backbone.<n>On the NYU Depth V2 benchmark, MoA-DepthCLIP achieves competitive results, significantly outperforming the DepthCLIP baseline.
• Score: 3.0739122323721766
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Leveraging the rich semantic features of vision-language models (VLMs) like CLIP for monocular depth estimation tasks is a promising direction, yet often requires extensive fine-tuning or lacks geometric precision. We present a parameter-efficient framework, named MoA-DepthCLIP, that adapts pretrained CLIP representations for monocular depth estimation with minimal supervision. Our method integrates a lightweight Mixture-of-Adapters (MoA) module into the pretrained Vision Transformer (ViT-B/32) backbone combined with selective fine-tuning of the final layers. This design enables spatially-aware adaptation, guided by a global semantic context vector and a hybrid prediction architecture that synergizes depth bin classification with direct regression. To enhance structural accuracy, we employ a composite loss function that enforces geometric constraints. On the NYU Depth V2 benchmark, MoA-DepthCLIP achieves competitive results, significantly outperforming the DepthCLIP baseline by improving the $δ_1$ accuracy from 0.390 to 0.745 and reducing the RMSE from 1.176 to 0.520. These results are achieved while requiring substantially few trainable parameters, demonstrating that lightweight, prompt-guided MoA is a highly effective strategy for transferring VLM knowledge to fine-grained monocular depth estimation tasks.

Related papers

• Predict and Reconstruct: Joint Objectives for Self-Supervised Language Representation Learning [0.0]
  Masked language modelling (MLM) has been the dominant pre-training objective for text encoders since BERT.<n>Inspired by the success of Joint Embedding Predictive Architectures (JEPA) in vision and audio, we propose a hybrid objective.<n>We pre-train both a hybrid model and a pure-MLM baseline on English Wikipedia using identical architectures and compute budgets.
  arXiv  Detail & Related papers  (2026-04-16T22:26:47Z)
• $D^3$-RSMDE: 40$\times$ Faster and High-Fidelity Remote Sensing Monocular Depth Estimation [72.9912717963138]
  Real-time, high-fidelity monocular depth estimation from remote sensing imagery is crucial for numerous applications.<n>Although using Vision Transformer (ViT) backbones for dense prediction is fast, they often exhibit poor perceptual quality.<n>We propose Depth Detail Diffusion for Remote Sensing Monocular Depth Estimation ($D3$-RSMDE) to overcome these limitations.<n>$D3$-RSMDE achieves a notable 11.85% reduction in the Learned Perceptual Image Patch Similarity (LPIPS) perceptual metric.
  arXiv  Detail & Related papers  (2026-03-17T10:50:36Z)
• How to Set the Learning Rate for Large-Scale Pre-training? [73.03133634525635]
  We formalize this investigation into two distinct research paradigms: Fitting and Transfer.<n>Within the Fitting Paradigm, we introduce a Scaling Law for search factor, effectively reducing the search complexity from O(n3) to O(n*C_D*C_) via predictive modeling.<n>We extend the principles of $$Transfer to the Mixture of Experts (MoE) architecture, broadening its applicability to encompass model depth, weight decay, and token horizons.
  arXiv  Detail & Related papers  (2026-01-08T15:55:13Z)
• IMSE: Efficient U-Net-based Speech Enhancement using Inception Depthwise Convolution and Amplitude-Aware Linear Attention [2.3959703715401903]
  This paper proposes IMSE, a systematically optimized and ultra-lightweight network.<n>We introduce two core innovations: 1) replacing the MET module with Amplitude-Aware Linear Attention (MALA) and 2) replacing the Deformable Embedding (DE) module with Inception Depthwise Convolution (IDConv)<n>In experiments, IMSE significantly reduces the parameter count by 16.8% (from 0.513M to 0.427M) while achieving competitive performance comparable to the state-of-the-art on the PESQ metric (3.373)
  arXiv  Detail & Related papers  (2025-11-18T14:11:54Z)
• MM-HELIX: Boosting Multimodal Long-Chain Reflective Reasoning with Holistic Platform and Adaptive Hybrid Policy Optimization [103.74675519953898]
  Long-chain reflective reasoning is a prerequisite for solving complex real-world problems.<n>We build a benchmark consisting 1,260 samples of 42 challenging synthetic tasks.<n>We generate post-training data and explore learning paradigms for exploiting such data.
  arXiv  Detail & Related papers  (2025-10-09T17:53:58Z)
• CLAMP-ViT: Contrastive Data-Free Learning for Adaptive Post-Training Quantization of ViTs [6.456189487006878]
  We present CLAMP-ViT, a data-free post-training quantization method for vision transformers (ViTs) We identify the limitations of recent techniques, notably their inability to leverage meaningful inter-patch relationships. CLAMP-ViT employs a two-stage approach, cyclically adapting between data generation and model quantization.
  arXiv  Detail & Related papers  (2024-07-07T05:39:25Z)
• EndoDepthL: Lightweight Endoscopic Monocular Depth Estimation with CNN-Transformer [0.0]
  We propose a novel lightweight solution named EndoDepthL that integrates CNN and Transformers to predict multi-scale depth maps. Our approach includes optimizing the network architecture, incorporating multi-scale dilated convolution, and a multi-channel attention mechanism. To better evaluate the performance of monocular depth estimation in endoscopic imaging, we propose a novel complexity evaluation metric.
  arXiv  Detail & Related papers  (2023-08-04T21:38:29Z)
• Monocular Visual-Inertial Depth Estimation [66.71452943981558]
  We present a visual-inertial depth estimation pipeline that integrates monocular depth estimation and visual-inertial odometry. Our approach performs global scale and shift alignment against sparse metric depth, followed by learning-based dense alignment. We evaluate on the TartanAir and VOID datasets, observing up to 30% reduction in RMSE with dense scale alignment.
  arXiv  Detail & Related papers  (2023-03-21T18:47:34Z)
• Unifying Synergies between Self-supervised Learning and Dynamic Computation [53.66628188936682]
  We present a novel perspective on the interplay between SSL and DC paradigms. We show that it is feasible to simultaneously learn a dense and gated sub-network from scratch in a SSL setting. The co-evolution during pre-training of both dense and gated encoder offers a good accuracy-efficiency trade-off.
  arXiv  Detail & Related papers  (2023-01-22T17:12:58Z)
• Rethinking Lightweight Salient Object Detection via Network Depth-Width Tradeoff [26.566339984225756]
  Existing salient object detection methods often adopt deeper and wider networks for better performance. We propose a novel trilateral decoder framework by decoupling the U-shape structure into three complementary branches. We show that our method achieves better efficiency-accuracy balance across five benchmarks.
  arXiv  Detail & Related papers  (2023-01-17T03:43:25Z)
• DepthFormer: Exploiting Long-Range Correlation and Local Information for Accurate Monocular Depth Estimation [50.08080424613603]
  Long-range correlation is essential for accurate monocular depth estimation. We propose to leverage the Transformer to model this global context with an effective attention mechanism. Our proposed model, termed DepthFormer, surpasses state-of-the-art monocular depth estimation methods with prominent margins.
  arXiv  Detail & Related papers  (2022-03-27T05:03:56Z)
• Towards 3D Scene Reconstruction from Locally Scale-Aligned Monocular Video Depth [90.33296913575818]
  In some video-based scenarios such as video depth estimation and 3D scene reconstruction from a video, the unknown scale and shift residing in per-frame prediction may cause the depth inconsistency. We propose a locally weighted linear regression method to recover the scale and shift with very sparse anchor points. Our method can boost the performance of existing state-of-the-art approaches by 50% at most over several zero-shot benchmarks.
  arXiv  Detail & Related papers  (2022-02-03T08:52:54Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.