ER-LoRA: Effective-Rank Guided Adaptation for Weather-Generalized Depth Estimation

• URL: http://arxiv.org/abs/2509.00665v2
• Date: Sat, 06 Sep 2025 16:03:21 GMT
• Title: ER-LoRA: Effective-Rank Guided Adaptation for Weather-Generalized Depth Estimation
• Authors: Weilong Yan, Xin Zhang, Robby T. Tan,
• Abstract summary: We propose weather-generalized depth estimation using only a small amount of high-visibility (normal) data.<n>We introduce the Selecting-Tuning-Maintaining (STM) strategy, which structurally decomposes the pretrained weights of Vision Foundation Models.<n>In the tuning phase, we adaptively select the proper rank number as well as the task-aware singular directions.<n>While in the maintaining stage, we enforce a principal direction regularization based on the stable-rank.
• Score: 33.632587382356824
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Monocular depth estimation under adverse weather conditions (e.g.\ rain, fog, snow, and nighttime) remains highly challenging due to the lack of reliable ground truth and the difficulty of learning from unlabeled real-world data. Existing methods often rely on synthetic adverse data with pseudo-labels, which suffer from domain gaps, or employ self-supervised learning, which violates photometric assumptions in adverse scenarios. In this work, we propose to achieve weather-generalized depth estimation by Parameter-Efficient Fine-Tuning (PEFT) of Vision Foundation Models (VFMs), using only a small amount of high-visibility (normal) data. While PEFT has shown strong performance in semantic tasks such as segmentation, it remains underexplored for geometry -- centric tasks like depth estimation -- especially in terms of balancing effective adaptation with the preservation of pretrained knowledge. To this end, we introduce the Selecting-Tuning-Maintaining (STM) strategy, which structurally decomposes the pretrained weights of VFMs based on two kinds of effective ranks (entropy-rank and stable-rank). In the tuning phase, we adaptively select the proper rank number as well as the task-aware singular directions for initialization, based on the entropy-rank and full-tuned weight; while in the maintaining stage, we enforce a principal direction regularization based on the stable-rank. This design guarantees flexible task adaptation while preserving the strong generalization capability of the pretrained VFM. Extensive experiments on four real-world benchmarks across diverse weather conditions demonstrate that STM not only outperforms existing PEFT methods and full fine-tuning but also surpasses methods trained with adverse synthetic data, and even the depth foundation model

Related papers

• Prequential posteriors [2.831395148295604]
  We introduce prequential posteriors, based upon a predictive-sequential (prequential) loss function.<n>We prove that, under mild conditions, both the prequential loss minimizer and the prequential posterior concentrate around parameters with optimal predictive performance.<n>We validate our method on both a synthetic multi-dimensional time series and a real-world meteorological dataset.
  arXiv  Detail & Related papers  (2025-11-21T19:18:19Z)
• Task-Adaptive Parameter-Efficient Fine-Tuning for Weather Foundation Models [28.79157031663951]
  We introduce WeatherPEFT, a novel PEFT framework for Weather Foundation Models (WFMs)<n>Task-Tu Dynamic Prompting (TTu) injects the embedding weights within the encoder to the input tokens of the backbone of the pre-trained via internal and external pattern extraction.<n>Fisher-Guided Adaptive Selection (SFAS) identifies the most task-critical randomness, preserving in pre-trained knowledge, but also stabilizing the selection.<n>We demonstrate the effectiveness and efficiency of WeatherPEFT on three downstream tasks, where existing PEFT methods show significant gaps versus Full-Tuning, and WeatherPEFT performance parity
  arXiv  Detail & Related papers  (2025-09-26T07:54:05Z)
• Weight Spectra Induced Efficient Model Adaptation [54.8615621415845]
  Fine-tuning large-scale foundation models incurs prohibitive computational costs.<n>We show that fine-tuning predominantly amplifies the top singular values while leaving the remainder largely intact.<n>We propose a novel method that leverages learnable rescaling of top singular directions.
  arXiv  Detail & Related papers  (2025-05-29T05:03:29Z)
• Synthetic-to-Real Self-supervised Robust Depth Estimation via Learning with Motion and Structure Priors [22.831281986234988]
  We present the first synthetic-to-real robust depth estimation framework, incorporating motion and structure priors to capture real-world knowledge effectively.<n>We achieve improvements of 7.5% and 4.3% in AbsRel and RMSE on average for nuScenes and Robotcar datasets (daytime, nighttime, rain)<n>In zero-shot evaluation of DrivingStereo (rain, fog), our method generalizes better than the previous ones.
  arXiv  Detail & Related papers  (2025-03-26T04:12:54Z)
• Large Continual Instruction Assistant [59.585544987096974]
  Continual Instruction Tuning (CIT) is adopted to instruct Large Models to follow human intent data by data.<n>Existing update gradient would heavily destroy the performance on previous datasets during CIT process.<n>We propose a general continual instruction tuning framework to address the challenge.
  arXiv  Detail & Related papers  (2024-10-08T11:24:59Z)
• Uncertainty Aware Learning for Language Model Alignment [97.36361196793929]
  We propose uncertainty-aware learning (UAL) to improve the model alignment of different task scenarios. We implement UAL in a simple fashion -- adaptively setting the label smoothing value of training according to the uncertainty of individual samples. Experiments on widely used benchmarks demonstrate that our UAL significantly and consistently outperforms standard supervised fine-tuning.
  arXiv  Detail & Related papers  (2024-06-07T11:37:45Z)
• Impact of Noisy Supervision in Foundation Model Learning [91.56591923244943]
  This paper is the first work to comprehensively understand and analyze the nature of noise in pre-training datasets.<n>We propose a tuning method (NMTune) to affine the feature space to mitigate the malignant effect of noise and improve generalization.
  arXiv  Detail & Related papers  (2024-03-11T16:22:41Z)
• DSAF: A Dual-Stage Adaptive Framework for Numerical Weather Prediction Downscaling [6.990912650604992]
  We propose a novel framework to address regional NWP downscaling and bias correction tasks. Dual-Stage Adaptive Framework (DSAF) incorporates adaptive elements in its design to ensure a flexible response to evolving weather conditions.
  arXiv  Detail & Related papers  (2023-12-19T13:13:17Z)
• Sparse is Enough in Fine-tuning Pre-trained Large Language Models [98.46493578509039]
  We propose a gradient-based sparse fine-tuning algorithm, named Sparse Increment Fine-Tuning (SIFT) We validate its effectiveness on a range of tasks including the GLUE Benchmark and Instruction-tuning.
  arXiv  Detail & Related papers  (2023-12-19T06:06:30Z)
• Test-Time Training for Semantic Segmentation with Output Contrastive Loss [12.535720010867538]
  Deep learning-based segmentation models have achieved impressive performance on public benchmarks, but generalizing well to unseen environments remains a major challenge. This paper introduces Contrastive Loss (OCL), known for its capability to learn robust and generalized representations, to stabilize the adaptation process. Our method excels even when applied to models initially pre-trained using domain adaptation methods on test domain data, showcasing its resilience and adaptability.
  arXiv  Detail & Related papers  (2023-11-14T03:13:47Z)
• Enhancing Plasticity for First Session Adaptation Continual Learning [20.62749699589017]
  We introduce Plasticity-Enhanced Test-Time Adaptation in Class-Incremental Learning (PLASTIC)<n>PLASTIC reinstates plasticity in CIL while preserving model stability.<n>It consistently outperforms both conventional and state-of-the-art PTM-based CIL approaches.
  arXiv  Detail & Related papers  (2023-10-17T13:06:39Z)
• Strong Baselines for Parameter Efficient Few-Shot Fine-tuning [50.83426196335385]
  Few-shot classification (FSC) entails learning novel classes given only a few examples per class after a pre-training (or meta-training) phase. Recent works have shown that simply fine-tuning a pre-trained Vision Transformer (ViT) on new test classes is a strong approach for FSC. Fine-tuning ViTs, however, is expensive in time, compute and storage. This has motivated the design of parameter efficient fine-tuning (PEFT) methods which fine-tune only a fraction of the Transformer's parameters.
  arXiv  Detail & Related papers  (2023-04-04T16:14:39Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.