PolarVLM: Bridging the Semantic-Physical Gap in Vision-Language Models

• URL: http://arxiv.org/abs/2605.07574v2
• Date: Mon, 11 May 2026 13:10:17 GMT
• Title: PolarVLM: Bridging the Semantic-Physical Gap in Vision-Language Models
• Abstract summary: We introduce PolarVLM, the first framework integrating polarimetric physical parameters into vision-language models.<n>By employing a dual-stream architecture and a progressive two-stage training strategy, PolarVLM effectively prevents physical misinterpretations while preserving general visual abilities.<n> Experiments show that PolarVLM surpasses the RGB baseline by 25.4% overall across five evaluation tasks.
• Score: 79.07798242685293
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Mainstream vision-language models (VLMs) fundamentally struggle with severe optical ambiguities, such as reflections and transparent objects, due to the inherent limitations of standard RGB inputs. While polarization imaging captures polarimetric physical parameters that resolve these ambiguities, existing methods are constrained by fixed-format outputs and remain isolated from open-ended reasoning. To bridge this semantic-physical gap, we introduce PolarVLM, the first multimodal framework integrating polarimetric physical parameters into VLMs. By employing a dual-stream architecture and a progressive two-stage training strategy, PolarVLM effectively prevents physical misinterpretations while preserving general visual abilities. Complementing our architecture, we construct PolarVQA, the first benchmark for polarization-aware VQA, featuring 75K physics-grounded instruction-tuning pairs targeting reflective and transparent scenes. Experiments show that PolarVLM surpasses the RGB baseline by 25.4% overall across five evaluation tasks, with remarkable gains of 26.6% in reflection recognition and 34.0% in glass counting, successfully unlocking physics-aware semantic understanding.

Related papers

• Stokes-Informed Diffusion for Robust Linear Polarization Estimation [49.459667092743445]
  GenPolar is a Stokes-informed diffusion framework grounded in the Mueller formalism from an intensity observation.<n>It achieves state-of-the-art performance in both DoLP fidelity and AoP stability.<n>These improvements translate to significant and consistent gains in downstream applications, including material detection and de-reflection.
  arXiv  Detail & Related papers  (2026-07-23T11:56:15Z)
• Con-DSO: Learning Short-Horizon Consistency Priors for RGB-D Direct Sparse Odometry [8.079139316299182]
  Con-DSO is a consistency-aware RGB-D direct sparse odometry framework.<n>It predicts dense photometric and depth-geometric consistency uncertainty from temporally adjacent RGB-D frame pairs.<n>Experiments show substantial gains over direct RGB-D VO baselines.
  arXiv  Detail & Related papers  (2026-05-27T04:46:52Z)
• VLMaterial: Vision-Language Model-Based Camera-Radar Fusion for Physics-Grounded Material Identification [2.934626037282987]
  We introduce a training-free framework that fuses vision-language models (VLMs) with domain-specific radar knowledge for physics-grounded material identification.<n> Experimental results demonstrate that VLMaterial achieves a recognition accuracy of 96.08%, delivering performance on par with state-of-the-art closed-set benchmarks.
  arXiv  Detail & Related papers  (2026-04-13T16:16:28Z)
• CDPR: Cross-modal Diffusion with Polarization for Reliable Monocular Depth Estimation [12.658602122161989]
  CDPR is a novel diffusion-based framework that integrates physically grounded polarization priors to enhance estimation robustness.<n>We show that CDPR significantly outperforms RGB-only baselines in challenging regions while maintaining competitive performance in standard scenes.
  arXiv  Detail & Related papers  (2026-04-13T07:12:49Z)
• High-Resolution Single-Shot Polarimetric Imaging Made Easy [76.04914313843774]
  EasyPolar is a multi-view polarimetric imaging framework.<n>Three synchronized RGB cameras capture one unpolarized view and two polarized views with distinct orientations.<n>A confidence-guided polarization reconstruction network is proposed to address the potential misalignment in multi-view fusion.
  arXiv  Detail & Related papers  (2026-04-07T08:20:58Z)
• MMPhysVideo: Scaling Physical Plausibility in Video Generation via Joint Multimodal Modeling [55.72785604682579]
  MMPhysVideo is a framework to scale physical plausibility in video generation in joint multimodal modeling.<n>MMPhysVideo consistently improves physical plausibility and visual quality over advanced models across various benchmarks.
  arXiv  Detail & Related papers  (2026-04-03T07:32:24Z)
• Learning to See through Illumination Extremes with Event Streaming in Multimodal Large Language Models [26.703204607483404]
  Event-MLLM is an event-enhanced model that performs all-light visual reasoning by dynamically fusing event streams with RGB frames.<n>We curate the first multi-illumination event-instruction corpus for MLLMs, with 2,241 event-RGB samples.<n>Experiments show that Event-MLLM markedly outperforms general-purpose, illumination-adaptive, and event-only baselines.
  arXiv  Detail & Related papers  (2026-03-29T07:46:32Z)
• VLM-RobustBench: A Comprehensive Benchmark for Robustness of Vision-Language Models [64.56065206447788]
  Vision-language models (VLMs) achieve strong performance on standard, high-quality datasets.<n>We present VLM-RobustBench, a benchmark spanning 49 augmentation types across noise, blur, weather, digital, and geometrics.<n>We show that low-severity spatial perturbations often degrade performance more than visually severe photometric corruptions.
  arXiv  Detail & Related papers  (2026-03-06T10:58:02Z)
• Revisiting Shape from Polarization in the Era of Vision Foundation Models [11.779432473091754]
  We show that a lightweight model trained on a small dataset can outperform RGB-only vision foundation models (VFMs) in single-shot object-level surface normal estimation.<n>With only 40K training scenes, our method significantly outperforms both state-of-the-art SfP approaches and RGB-only VFMs.
  arXiv  Detail & Related papers  (2026-03-05T05:07:03Z)
• Fooling Polarization-based Vision using Locally Controllable Polarizing Projection [55.40484331029597]
  We warn the community of the vulnerability of polarization-based vision, which can be more serious than RGB-based vision. By adapting a commercial LCD projector, we achieve locally controllable polarizing projection, which is successfully utilized to fool state-of-the-art polarization-based vision algorithms.
  arXiv  Detail & Related papers  (2023-03-31T08:48:57Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.