Mechanistic Interpretability-Guided Selective Fine-Tuning of Vision-Language Models for Centimeter-Level Flood Depth Estimation

• URL: http://arxiv.org/abs/2608.07562v1
• Date: Sun, 02 Aug 2026 22:38:25 GMT
• Title: Mechanistic Interpretability-Guided Selective Fine-Tuning of Vision-Language Models for Centimeter-Level Flood Depth Estimation
• Abstract summary: Urban flooding poses an escalating threat to transportation infrastructure, yet no operational system provides real-time, street-level flood-depth estimates at centimeter resolution.<n>This paper presents three vision-language models fine-tuned for continuous flood-depth estimation from street-level imagery.<n>FloodLlama-Dense achieves an MAE of 0.40 cm, an RMSE of 1.97 cm, and an Acc@5cm of 97.59%.
• Score: 4.766239287853751
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Urban flooding poses an escalating threat to transportation infrastructure, yet no operational system provides real-time, street-level flood-depth estimates at centimeter resolution. This paper presents three vision-language models fine-tuned for continuous flood-depth estimation from street-level imagery: FloodLlama-Dense, a fully fine-tuned QLoRA baseline, and FloodLlama-MI5 and FloodLlama-MI6, interpretability-guided sparse variants that fine-tune only the top five and six causally relevant cross-attention layers identified through mechanistic interpretability analysis, respectively. Training uses an approximately 610,000-image subset of a 2.81-million-image synthetic corpus generated in Unreal Engine 5. The dataset combines single-vehicle subsets with 5 cm depth increments and mixed-vehicle subsets with 1 cm depth increments, spanning seven vehicle types, four weather conditions, and flood depths from 0 to 40 cm. FloodLlama-Dense achieves an MAE of 0.40 cm, an RMSE of 1.97 cm, and an Acc@5cm of 97.59%. Mechanistic interpretability analysis combining linear probing, logit lens, centered kernel alignment (CKA), and cross-attention entropy reveals a two-stage adaptation pattern: layers L13-L22 restructure visual representations, while depth first becomes linearly decodable at layer L23. FloodLlama-MI5 and FloodLlama-MI6 leverage this insight by fine-tuning only five or six of the eight cross-attention layers, achieving an 86-88% reduction in trainable parameters (6.55-7.86 million versus 54.4 million) with minimal accuracy loss. On a real-world benchmark, FloodLlama-MI6 achieves 98.62% accuracy, compared with 86.61% for the published STURM-FloodDepth baseline.

Related papers

• ScopeMamba-YOLO: Widening the Perceptual Scope Inward and Outward for Small Object Detection in Remote Sensing Imagery [2.3496832983179807]
  Small object detection in unmanned aerial vehicle (UAV) and remote sensing imagery requires preserving high-resolution detail while modeling long-range context.<n>We propose ScopeMamba-YOLO, built around an off-path, zero-gated selective-scanning principle that decouples contextual modeling from the convolutional stream.<n>On VisDrone 2019, ScopeMamba-S achieves 50.8% mAP50 with 3.57M parameters, exceeding YOLOv8s by 10.8 pp while using 32% of its parameters; ScopeMamba-M reaches 52.6% mAP50 with 6.48M parameters.
  arXiv  Detail & Related papers  (2026-09-09T13:32:46Z)
• Learning to Navigate Efficiently with Only 0.58M Trainable Parameters [12.68304107799412]
  We propose a navigation model in which operations with known closed-form structure are computed analytically and serve as interfaces between three small learned modules.<n>The system trains only 0.58M out of a total of 22.7M parameters, on 44k frames in under one GPU-hour, yet approaches the performance of state-of-the-art models on navigation tasks.
  arXiv  Detail & Related papers  (2026-07-13T02:49:55Z)
• Scaling Closed-Loop Feature Channel Configuration with LLMs [48.83701310501069]
  The analysis covers 2000 generated candidates from 8 complete cycles, yielding 462 verified CIFAR-100 evaluations after task and metadata filtering.<n>The best model reaches 0.3676 with 11.8M parameters, compared with an early high-performing model at 0.3144 with 166.5M parameters.<n>Non-power-of-two channel widths occur in 41.8% of verified candidates, and the strongest models share structured channelallocation patterns.
  arXiv  Detail & Related papers  (2026-07-07T22:31:53Z)
• LLM-Powered Flood Depth Estimation from Social Media Imagery: A Vision-Language Model Framework with Mechanistic Interpretability for Transportation Resilience [0.0]
  FloodLlama is a fine-tuned vision-language model for continuous flood depth estimation from single street-level images.<n>A TikTok-based data pipeline, validated on 676 annotated flood frames from Detroit, demonstrates the feasibility of real-time, crowd-sourced flood sensing.
  arXiv  Detail & Related papers  (2026-03-17T19:59:25Z)
• Do Foundation Models Know Geometry? Probing Frozen Features for Continuous Physical Measurement [0.0]
  Vision-language models encode geometry that their text pathway fails to express.<n>LoRA fine-tuning (r=16, 2,000 images) narrows this gap to 6.5 degrees.<n>These findings enable a single frozen backbone to function as a multi-task geometric sensor.
  arXiv  Detail & Related papers  (2026-03-06T16:48:27Z)
• MUOT_3M: A 3 Million Frame Multimodal Underwater Benchmark and the MUTrack Tracking Method [49.121704780656366]
  Underwater Object Tracking (UOT) is crucial for efficient marine robotics, large scale ecological monitoring, and ocean exploration.<n>Existing benchmarks remain small and RGB only, limiting robustness under severe color distortion, turbidity, and low visibility conditions.<n>We introduce MUOT_3M, the first pseudo multimodal UOT benchmark comprising 3 million frames from 3,030 videos (27.8h) annotated with 32 tracking attributes, 677 fine grained classes, and synchronized RGB.<n>We propose MUTrack, a SAM-based multimodal to unimodal tracker featuring visual geometric alignment, vision language fusion, and four level knowledge distillation.
  arXiv  Detail & Related papers  (2026-02-20T05:43:47Z)
• Structure-Informed Estimation for Pilot-Limited MIMO Channels via Tensor Decomposition [51.56484100374058]
  This paper formulates pilot-limited channel estimation as low-rank tensor completion from sparse observations.<n>Experiments on synthetic channels demonstrate 10-20,dB normalized mean-square error (NMSE) improvement over least-squares (LS)<n> evaluations on DeepMIMO ray-tracing channels show 24-44% additional NMSE reduction over pure tensor-based methods.
  arXiv  Detail & Related papers  (2026-02-03T23:38:05Z)
• Automated river gauge plate reading using a hybrid object detection and generative AI framework in the Limpopo River Basin [0.0]
  This study presents a hybrid framework integrating vision based waterline detection, YOLOv8 pose scale extraction, and large multimodal language models for automated river gauge plate reading.<n>Experiments demonstrate that waterline detection achieved high precision of 94.24 percent and an F1 score of 83.64 percent, while scale gap detection provided accurate geometric calibration for subsequent reading extraction.<n>Results highlight the sensitivity of LLMs to image quality, with degraded images producing higher errors, and underscore the importance of combining geometric metadata with multimodal artificial intelligence for robust water level estimation.
  arXiv  Detail & Related papers  (2025-12-29T13:26:59Z)
• ReconMOST: Multi-Layer Sea Temperature Reconstruction with Observations-Guided Diffusion [48.540756751934836]
  ReconMOST is a data-driven guided diffusion model framework for multi-layer sea temperature reconstruction.<n>Our method extends ML-based SST reconstruction to a global, multi-layer setting, handling over 92.5% missing data.
  arXiv  Detail & Related papers  (2025-06-12T06:27:22Z)
• Mixture-of-Transformers: A Sparse and Scalable Architecture for Multi-Modal Foundation Models [111.97026994761254]
  Mixture-of-Transformers (MoT) is a sparse multi-modal transformer architecture.<n>MoT decouples non-embedding parameters of the model by modality.<n>We evaluate MoT across multiple settings and model scales.
  arXiv  Detail & Related papers  (2024-11-07T18:59:06Z)
• Ultra-low Power Deep Learning-based Monocular Relative Localization Onboard Nano-quadrotors [64.68349896377629]
  This work presents a novel autonomous end-to-end system that addresses the monocular relative localization, through deep neural networks (DNNs), of two peer nano-drones. To cope with the ultra-constrained nano-drone platform, we propose a vertically-integrated framework, including dataset augmentation, quantization, and system optimizations. Experimental results show that our DNN can precisely localize a 10cm-size target nano-drone by employing only low-resolution monochrome images, up to 2m distance.
  arXiv  Detail & Related papers  (2023-03-03T14:14:08Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.