VLMaterial: Vision-Language Model-Based Camera-Radar Fusion for Physics-Grounded Material Identification

• URL: http://arxiv.org/abs/2604.11671v2
• Date: Wed, 15 Apr 2026 01:48:41 GMT
• Title: VLMaterial: Vision-Language Model-Based Camera-Radar Fusion for Physics-Grounded Material Identification
• Abstract summary: We introduce a training-free framework that fuses vision-language models (VLMs) with domain-specific radar knowledge for physics-grounded material identification.<n> Experimental results demonstrate that VLMaterial achieves a recognition accuracy of 96.08%, delivering performance on par with state-of-the-art closed-set benchmarks.
• Score: 2.934626037282987
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Accurate material recognition is a fundamental capability for intelligent perception systems to interact safely and effectively with the physical world. For instance, distinguishing visually similar objects like glass and plastic cups is critical for safety but challenging for vision-based methods due to specular reflections, transparency, and visual deception. While millimeter-wave (mmWave) radar offers robust material sensing regardless of lighting, existing camera-radar fusion methods are limited to closed-set categories and lack semantic interpretability. In this paper, we introduce VLMaterial, a training-free framework that fuses vision-language models (VLMs) with domain-specific radar knowledge for physics-grounded material identification. First, we propose a dual-pipeline architecture: an optical pipeline uses the segment anything model and VLM for material candidate proposals, while an electromagnetic characterization pipeline extracts the intrinsic dielectric constant from radar signals via an effective peak reflection cell area (PRCA) method and weighted vector synthesis. Second, we employ a context-augmented generation (CAG) strategy to equip the VLM with radar-specific physical knowledge, enabling it to interpret electromagnetic parameters as stable references. Third, an adaptive fusion mechanism is introduced to intelligently integrate outputs from both sensors by resolving cross-modal conflicts based on uncertainty estimation. We evaluated VLMaterial in over 120 real-world experiments involving 41 diverse everyday objects and 4 typical visually deceptive counterfeits across varying environments. Experimental results demonstrate that VLMaterial achieves a recognition accuracy of 96.08%, delivering performance on par with state-of-the-art closed-set benchmarks while eliminating the need for extensive task-specific data collection and training.

Related papers

• Towards Generalizable Visually Grounded Exploration of Household Devices [74.66117923766632]
  Key bottleneck of realizing general autonomous embodied agents lies in Generalizable Visually Grounded Exploration.<n>We introduce VGEBench, a benchmark designed to evaluate the generalizable visually grounded exploration capabilities of Vision-Language Models.<n>Unlike static datasets, we construct a Logic-Driven State Machine framework. This framework simulates multi-turn interaction loops, compelling agents to achieve goals by active visual perception and feedback-driven correction.
  arXiv  Detail & Related papers  (2026-09-01T07:42:45Z)
• NavEYE: Vision-Centered Multi-Sensor Fusion-Based Situational Awareness System for Intelligent Surface Vehicles [13.095893937693802]
  We develop a vision-centered multi-sensor fusion system, named NavEYE, to achieve high-quality perception.<n>We first propose a multi-constrained gated data association method (MCGA) to accurately match low-temporal-resolution AIS data with high-temporal-resolution radar data.
  arXiv  Detail & Related papers  (2026-07-04T15:18:22Z)
• End-to-End Unmixing with Material Prompts for Hyperspectral Object Tracking [10.366723514972815]
  Hyperspectral imagery encodes rich material properties that can improve tracking robustness under appearance ambiguity, illumination change, and background clutter.<n>We propose a material representation decomposition module for deep learning-based spectral unmixing with adaptive frequency decomposition.<n>We also introduce a dual-branch wavelet-enhanced material prompt module that learns low- and high-frequency material prompts.
  arXiv  Detail & Related papers  (2026-05-20T00:01:05Z)
• Multi-Modal Landslide Detection from Sentinel-1 SAR and Sentinel-2 Optical Imagery Using Multi-Encoder Vision Transformers and Ensemble Learning [0.0]
  Landslides represent a major geohazard with severe impacts on human life, infrastructure, and ecosystems.<n>This study proposes a modular, multi-model framework that fuses Sentinel-2 optical imagery with Sentinel-1 Synthetic Aperture Radar (SAR) data.<n>The proposed methodology achieves a state-of-the-art F1 score of 0.919 on landslide detection.
  arXiv  Detail & Related papers  (2026-04-07T14:51:05Z)
• TranX-Adapter: Bridging Artifacts and Semantics within MLLMs for Robust AI-generated Image Detection [70.42796551833946]
  incorporating texture-level artifact features alongside semantic features into multimodal large language models (MLLMs) can enhance their AIGI detection capability.<n>We propose a lightweight fusion adapter, TranX-Adapter, which integrates a Task-aware Optimal-Transport Fusion.<n>Experiments on standard AIGI detection benchmarks upon several advanced MLLMs, show that our TranX-Adapter brings consistent and significant improvements.
  arXiv  Detail & Related papers  (2026-02-25T09:22:46Z)
• Zero-shot HOI Detection with MLLM-based Detector-agnostic Interaction Recognition [71.5328300638085]
  Zero-shot Human-object interaction (HOI) detection aims to locate humans and objects in images and recognize their interactions.<n>Existing methods, including two-stage methods, tightly couple interaction recognition with a specific detector.<n>We propose a decoupled framework that separates object detection from IR and leverages multi-modal large language models (MLLMs) for zero-shot IR.
  arXiv  Detail & Related papers  (2026-02-16T19:01:31Z)
• From Words to Wavelengths: VLMs for Few-Shot Multispectral Object Detection [7.459632891054827]
  Multispectral object detection is critical for safety-sensitive applications such as autonomous driving and surveillance.<n>Motivated by the recent success of Vision-Language Models in computer vision, we explore their potential for few-shot multispectral object detection.
  arXiv  Detail & Related papers  (2025-12-17T21:06:36Z)
• SpectralCA: Bi-Directional Cross-Attention for Next-Generation UAV Hyperspectral Vision [0.0]
  The relevance of this research lies in the growing demand for unmanned aerial vehicles capable of operating reliably in complex environments.<n>Hyperspectral imaging (HSI) provides unique opportunities for UAV-based computer vision.<n>The aim of this work is to develop a deep learning architecture integrating HSI into UAV perception for navigation, object detection, and terrain classification.
  arXiv  Detail & Related papers  (2025-10-10T22:53:28Z)
• Graph-Based Multi-Modal Sensor Fusion for Autonomous Driving [3.770103075126785]
  We introduce a novel approach to multi-modal sensor fusion, focusing on developing a graph-based state representation. We present a Sensor-Agnostic Graph-Aware Kalman Filter, the first online state estimation technique designed to fuse multi-modal graphs. We validate the effectiveness of our proposed framework through extensive experiments conducted on both synthetic and real-world driving datasets.
  arXiv  Detail & Related papers  (2024-11-06T06:58:17Z)
• Adaptive Multi-source Predictor for Zero-shot Video Object Segmentation [68.56443382421878]
  We propose a novel adaptive multi-source predictor for zero-shot video object segmentation (ZVOS) In the static object predictor, the RGB source is converted to depth and static saliency sources, simultaneously. Experiments show that the proposed model outperforms the state-of-the-art methods on three challenging ZVOS benchmarks.
  arXiv  Detail & Related papers  (2023-03-18T10:19:29Z)
• Multimodal Object Detection via Bayesian Fusion [59.31437166291557]
  We study multimodal object detection with RGB and thermal cameras, since the latter can provide much stronger object signatures under poor illumination. Our key contribution is a non-learned late-fusion method that fuses together bounding box detections from different modalities. We apply our approach to benchmarks containing both aligned (KAIST) and unaligned (FLIR) multimodal sensor data.
  arXiv  Detail & Related papers  (2021-04-07T04:03:20Z)
• SSTN: Self-Supervised Domain Adaptation Thermal Object Detection for Autonomous Driving [6.810856082577402]
  We have proposed a deep neural network Self Supervised Thermal Network (SSTN) for learning the feature embedding to maximize the information between visible and infrared spectrum domain by contrastive learning. The proposed method is extensively evaluated on the two publicly available datasets: the FLIR-ADAS dataset and the KAIST Multi-Spectral dataset.
  arXiv  Detail & Related papers  (2021-03-04T16:42:49Z)
• Semantics-aware Adaptive Knowledge Distillation for Sensor-to-Vision Action Recognition [131.6328804788164]
  We propose a framework, named Semantics-aware Adaptive Knowledge Distillation Networks (SAKDN), to enhance action recognition in vision-sensor modality (videos) The SAKDN uses multiple wearable-sensors as teacher modalities and uses RGB videos as student modality.
  arXiv  Detail & Related papers  (2020-09-01T03:38:31Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.