Grounding Agentic VLMs with Dedicated Segmentation for Fine-Grained Vehicle Damage Assessment

• URL: http://arxiv.org/abs/2608.02470v1
• Date: Mon, 03 Aug 2026 16:37:49 GMT
• Title: Grounding Agentic VLMs with Dedicated Segmentation for Fine-Grained Vehicle Damage Assessment
• Abstract summary: Vision-language models (VLMs) are increasingly deployed as reasoning agents in real-world visual assessment pipelines.<n>We show that a state-of-the-art VLM (Qwen-VL) achieves strong semantic classification accuracy (87.3%) on this task but is systematically ungrounded at the spatial level.<n>We propose TinyDamage, a hybrid architecture that delegates spatial grounding to a dedicated multi-task segmentation model.<n>We show that this grounding reduces the report hallucination rate from 92% (text-only) and 78% (image-only) to 31% in a controlled evaluation on 100 human-verified reports
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Vision-language models (VLMs) are increasingly deployed as reasoning agents in real-world visual assessment pipelines, yet their spatial grounding remains unreliable for fine-grained, visually ambiguous targets. We study this gap in the context of automated vehicle damage assessment, where fine-grained defects such as scratches and hairline cracks occupy few pixels, produce weak gradient signal, and are easily confused with reflections and surface texture. We show that a state-of-the-art VLM (Qwen-VL) achieves strong semantic classification accuracy (87.3%) on this task but is systematically ungrounded at the spatial level: it hallucinates damage in reflective regions, misses elongated scratches entirely, and produces spatially inconsistent outputs when prompted for localization. We propose TinyDamage, a hybrid architecture that delegates spatial grounding to a dedicated multi-task segmentation model while reserving the VLM for semantic reasoning and report generation. On the segmentation side, we find that the choice of loss function has an outsized and underexplored effect on tiny-object grounding: focal loss, widely used for class imbalance, collapses tiny-damage detection to zero, while a supervised contrastive objective measurably improves damage/background separability. We integrate the segmentation model into a 7-node LangGraph agent pipeline that grounds every VLM generation step in the segmentation output, and show that this grounding reduces the report hallucination rate from 92% (text-only) and 78% (image-only) to 31% in a controlled evaluation on 100 human-verified reports. We introduce DET_l, a permissive per-category detection metric for evaluating tiny-object grounding under class imbalance, and report latency and reliability characteristics of the deployed pipeline.

Related papers

• Assessing the Benefits of Combining Advanced Deep Learning Techniques for Post-Disaster Building Damage Assessment from UAV Imagery [3.016488960368601]
  We propose a hybrid framework that decouples detection from damage assessment.<n>A CV model first detects buildings and generates bounding boxes on the image that are then passed to an LVLM for damage classification and contextual interpretation.<n>We evaluate our framework on two real-world benchmarks: RescueNet and FloodNet.
  arXiv  Detail & Related papers  (2026-08-03T08:39:59Z)
• Active Adversarial Perturbation-driven Associative Memory Retrieval for RGB-Event Visual Object Tracking [58.338848435042536]
  RGB-Event tracking improves localization by fusing RGB appearance textures and dense temporal motion cues from event sensors.<n>Real-world scenes suffer diverse structured signal degradations that hinder traditional multi-modal fusion.<n>We present a hierarchical perturbation and retrieval framework tailored for RGB-Event tracking with robustness against partial target missing and modal degradation.
  arXiv  Detail & Related papers  (2026-06-24T23:34:06Z)
• PASTA: Vision Transformer Patch Aggregation for Weakly Supervised Target and Anomaly Segmentation [4.14197005718384]
  PASTA identifies Target and Anomaly objects through distribution analysis in self-supervised Vision Transformer feature spaces.<n>Our pipeline utilizes semantic text-prompts via the Segment Anything Model 3 to guide zero-shot object segmentation.<n>While being domain-agnostic, our method achieves superior Target (up to 88.3% IoU) and Anomaly (up to 3.5% 6 IoU) segmentation performance in the industrial and agricultural domain.
  arXiv  Detail & Related papers  (2026-04-07T08:29:11Z)
• VLM-RobustBench: A Comprehensive Benchmark for Robustness of Vision-Language Models [64.56065206447788]
  Vision-language models (VLMs) achieve strong performance on standard, high-quality datasets.<n>We present VLM-RobustBench, a benchmark spanning 49 augmentation types across noise, blur, weather, digital, and geometrics.<n>We show that low-severity spatial perturbations often degrade performance more than visually severe photometric corruptions.
  arXiv  Detail & Related papers  (2026-03-06T10:58:02Z)
• Deep Global Clustering for Hyperspectral Image Segmentation: Concepts, Applications, and Open Challenges [1.9116784879310027]
  Hyperspectral imaging (HSI) analysis faces computational bottlenecks due to massive data volumes that exceed available memory.<n>This report presents Deep Global Clustering (DGC), a conceptual framework for memory-efficient HSI segmentation.<n>DGC operates on small patches with overlapping regions to enforce consistency, enabling training in under 30 minutes on consumer hardware.
  arXiv  Detail & Related papers  (2025-12-30T12:10:43Z)
• SINDER: Repairing the Singular Defects of DINOv2 [61.98878352956125]
  Vision Transformer models trained on large-scale datasets often exhibit artifacts in the patch token they extract. We propose a novel fine-tuning smooth regularization that rectifies structural deficiencies using only a small dataset.
  arXiv  Detail & Related papers  (2024-07-23T20:34:23Z)
• GeneralAD: Anomaly Detection Across Domains by Attending to Distorted Features [68.14842693208465]
  GeneralAD is an anomaly detection framework designed to operate in semantic, near-distribution, and industrial settings. We propose a novel self-supervised anomaly generation module that employs straightforward operations like noise addition and shuffling to patch features. We extensively evaluated our approach on ten datasets, achieving state-of-the-art results in six and on-par performance in the remaining.
  arXiv  Detail & Related papers  (2024-07-17T09:27:41Z)
• 2D Feature Distillation for Weakly- and Semi-Supervised 3D Semantic Segmentation [92.17700318483745]
  We propose an image-guidance network (IGNet) which builds upon the idea of distilling high level feature information from a domain adapted synthetically trained 2D semantic segmentation network. IGNet achieves state-of-the-art results for weakly-supervised LiDAR semantic segmentation on ScribbleKITTI, boasting up to 98% relative performance to fully supervised training with only 8% labeled points.
  arXiv  Detail & Related papers  (2023-11-27T07:57:29Z)
• Semi-signed neural fitting for surface reconstruction from unoriented point clouds [53.379712818791894]
  We propose SSN-Fitting to reconstruct a better signed distance field. SSN-Fitting consists of a semi-signed supervision and a loss-based region sampling strategy. We conduct experiments to demonstrate that SSN-Fitting achieves state-of-the-art performance under different settings.
  arXiv  Detail & Related papers  (2022-06-14T09:40:17Z)
• FSNet: A Failure Detection Framework for Semantic Segmentation [9.453250122818808]
  We propose a failure detection framework to identify pixel-level misclassification. We do so by exploiting internal features of the segmentation model and training it simultaneously with a failure detection network. We evaluate the proposed approach against state-of-the-art methods and achieve 12.30%, 9.46%, and 9.65% performance improvement.
  arXiv  Detail & Related papers  (2021-08-19T15:26:52Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.