Strategic Fusion of Vision Language Models: Shapley-Credited Context-Aware Dawid-Skene for Multi-Label Tasks in Autonomous Driving

• URL: http://arxiv.org/abs/2510.01126v1
• Date: Wed, 01 Oct 2025 17:14:11 GMT
• Title: Strategic Fusion of Vision Language Models: Shapley-Credited Context-Aware Dawid-Skene for Multi-Label Tasks in Autonomous Driving
• Abstract summary: Large vision-language models (VLMs) are increasingly used in autonomous-vehicle stacks, but hallucination limits their reliability in safety-critical pipelines.<n>We present Shapley-credited Context-Aware Dawid-Skene with Agreement, a game-theoretic fusion method for multi-label understanding of ego-view dashcam video.
• Score: 5.508731861264645
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large vision-language models (VLMs) are increasingly used in autonomous-vehicle (AV) stacks, but hallucination limits their reliability in safety-critical pipelines. We present Shapley-credited Context-Aware Dawid-Skene with Agreement, a game-theoretic fusion method for multi-label understanding of ego-view dashcam video. It learns per-model, per-label, context-conditioned reliabilities from labelled history and, at inference, converts each model's report into an agreement-guardrailed log-likelihood ratio that is combined with a contextual prior and a public reputation state updated via Shapley-based team credit. The result is calibrated, thresholdable posteriors that (i) amplify agreement among reliable models, (ii) preserve uniquely correct single-model signals, and (iii) adapt to drift. To specialise general VLMs, we curate 1,000 real-world dashcam clips with structured annotations (scene description, manoeuvre recommendation, rationale) via an automatic pipeline that fuses HDD ground truth, vehicle kinematics, and YOLOv11 + BoT-SORT tracking, guided by a three-step chain-of-thought prompt; three heterogeneous VLMs are then fine-tuned with LoRA. We evaluate with Hamming distance, Micro-Macro-F1, and average per-video latency. Empirically, the proposed method achieves a 23% reduction in Hamming distance, 55% improvement in Macro-F1, and 47% improvement in Micro-F1 when comparing with the best single model, supporting VLM fusion as a calibrated, interpretable, and robust decision-support component for AV pipelines.

Related papers

• OneDrive: Unified Multi-Paradigm Driving with Vision-Language-Action Models [69.2503510410147]
  We present a unified autonomous driving framework built upon a pretrained VLM.<n>We demonstrate that pretrained VLM attention exhibits strong transferability beyond pure language modeling.<n>Experiments on end-to-end autonomous driving benchmarks demonstrate state-of-the-art performance.
  arXiv  Detail & Related papers  (2026-04-20T07:50:00Z)
• Video-MME-v2: Towards the Next Stage in Benchmarks for Comprehensive Video Understanding [98.3098451637867]
  Video-MME-v2 is a comprehensive benchmark designed to rigorously evaluate the robustness and faithfulness of video understanding.<n>To guarantee data quality, Video-MME-v2 is constructed through a rigorously controlled human annotation pipeline.
  arXiv  Detail & Related papers  (2026-04-06T17:59:56Z)
• Bench2Drive-VL: Benchmarks for Closed-Loop Autonomous Driving with Vision-Language Models [50.22099309218635]
  In autonomous driving, closed-loop evaluation has become widely recognized as a more reliable validation method than open-loop evaluation.<n>We present Bench2Drive-VL, an extension of Bench2Drive that brings closed-loop evaluation to VLM-based driving.
  arXiv  Detail & Related papers  (2026-04-01T11:38:46Z)
• Collision-Aware Vision-Language Learning for End-to-End Driving with Multimodal Infraction Datasets [7.932761533792761]
  High infraction rates remain the primary bottleneck for end-to-end (E2E) autonomous driving.<n>We develop a Video-Language-Augmented Anomaly Detector (VLAAD)<n>VLAAD is a collision-aware plug-in module that can be seamlessly integrated into existing E2E driving models.
  arXiv  Detail & Related papers  (2026-03-26T22:32:52Z)
• VLM-AutoDrive: Post-Training Vision-Language Models for Safety-Critical Autonomous Driving Events [25.118024547769895]
  We introduce VLM-AutoDrive, a modular framework for adapting pretrained Vision-Language Models to high-fidelity anomaly detection.<n>Off-the-shelf VLMs such as NVIDIA's Cosmos-1 7B (CR1) exhibit near-zero Collision recall in zero-shot settings.<n>VLM-AutoDrive offers a scalable recipe for adapting general-purpose VLMs to safety-critical, temporally localized perception tasks.
  arXiv  Detail & Related papers  (2026-03-18T18:23:34Z)
• ReMoT: Reinforcement Learning with Motion Contrast Triplets [37.29312323908102]
  We present ReMoT, a unified training paradigm to address the fundamental shortcomings of VLMs in-temporal.<n>A rule-based automatic framework generates ReMoT-16K triplets, a large-scale (1K) motion-contrast triplets from video meta-annotations.<n>We also construct the first benchmark for fine-grained motion contrast triplets to measure a VLMs of subtle discrimination.
  arXiv  Detail & Related papers  (2026-02-28T04:42:34Z)
• FFP-300K: Scaling First-Frame Propagation for Generalizable Video Editing [97.35186681023025]
  We introduce FFP-300K, a new large-scale dataset of high-fidelity video pairs at 720p resolution and 81 frames in length.<n>We propose a novel framework designed for true guidance-free FFP that resolves the tension between maintaining first-frame appearance and preserving source video motion.
  arXiv  Detail & Related papers  (2026-01-05T01:46:22Z)
• OmniDrive-R1: Reinforcement-driven Interleaved Multi-modal Chain-of-Thought for Trustworthy Vision-Language Autonomous Driving [11.793091014248025]
  We introduce OmniDrive-R1, an end-to-end VLM framework designed for autonomous driving.<n>Our core innovation is an Reinforcement-driven visual grounding capability.<n>This capability is enabled by our pure two-stage reinforcement learning training pipeline and Clip-GRPO algorithm.
  arXiv  Detail & Related papers  (2025-12-16T03:19:28Z)
• Self-Improving VLM Judges Without Human Annotations [74.29324865147838]
  We present a framework to self-train a VLM judge model without any human preference annotations, using only self-synthesized data.<n>Our method improves a Llama-3.2-11B multimodal judge from 0.38 to 0.51 in overall accuracy on Multimodal RewardBench.<n>The overall strength of these human-annotation-free results suggest the potential for a future self-judge that evolves alongside rapidly improving VLM capabilities.
  arXiv  Detail & Related papers  (2025-12-02T20:52:19Z)
• A Tri-Modal Dataset and a Baseline System for Tracking Unmanned Aerial Vehicles [74.8162337823142]
  MM-UAV is the first large-scale benchmark for Multi-Modal UAV Tracking.<n>The dataset spans over 30 challenging scenarios, with 1,321 synchronised multi-modal sequences, and more than 2.8 million annotated frames.<n>Accompanying the dataset, we provide a novel multi-modal multi-UAV tracking framework.
  arXiv  Detail & Related papers  (2025-11-23T08:42:17Z)
• Vehicle-to-Infrastructure Collaborative Spatial Perception via Multimodal Large Language Models [41.00138090010061]
  A lightweight, plug-and-play bird's-eye view (BEV) injection connector is proposed to overcome the limitations of large language models (MLLMs)<n>Ray tracing is developed to generate RGB, LiDAR, GPS, and wireless signal data across varied sensing scenarios.<n> Simulation results show that the proposed BEV injection framework consistently improved performance across all tasks.
  arXiv  Detail & Related papers  (2025-09-04T02:57:47Z)
• KEPT: Knowledge-Enhanced Prediction of Trajectories from Consecutive Driving Frames with Vision-Language Models [19.625631486595505]
  This paper introduces KEPT, a knowledge-enhanced vision-language framework.<n>It predicts ego trajectories directly from consecutive front-view driving frames.<n>It achieves state-of-the-art performance across open-loop protocols.
  arXiv  Detail & Related papers  (2025-09-03T03:10:42Z)
• Discrete Diffusion VLA: Bringing Discrete Diffusion to Action Decoding in Vision-Language-Action Policies [64.20587321033149]
  We present a single-transformer policy that models discretized action chunks with discrete diffusion and is trained with the same cross-entropy objective as the VLM backbone.<n>Our method achieves an adaptive decoding order that resolves easy action elements before harder ones.<n>This unified decoder preserves pretrained vision language priors, supports parallel decoding, breaks the autoregressive bottleneck, and reduces the number of function evaluations.
  arXiv  Detail & Related papers  (2025-08-27T17:39:11Z)
• Test-Time Consistency in Vision Language Models [26.475993408532304]
  Vision-Language Models (VLMs) have achieved impressive performance across a wide range of multimodal tasks.<n>Recent benchmarks, such as MM-R3, highlight that even state-of-the-art VLMs can produce divergent predictions across semantically equivalent inputs.<n>We propose a simple and effective test-time consistency framework that enhances semantic consistency without supervised re-training.
  arXiv  Detail & Related papers  (2025-06-27T17:09:44Z)
• CronusVLA: Towards Efficient and Robust Manipulation via Multi-Frame Vision-Language-Action Modeling [84.51372201195132]
  CronusVLA is a unified framework that extends single-frame VLA models to the multi-frame paradigm.<n>CronusVLA achieves leading performance and superior robustness, with a 70.9% success rate.<n>These results highlight the potential of efficient multi-frame adaptation in VLA models for more powerful and robust real-world deployment.
  arXiv  Detail & Related papers  (2025-06-24T17:30:27Z)
• TS-VLM: Text-Guided SoftSort Pooling for Vision-Language Models in Multi-View Driving Reasoning [1.4087249809872509]
  Vision-Language Models (VLMs) have shown potential to enhance scene perception, reasoning, and decision-making.<n>Existing models suffer from computational overhead and inefficient integration of multi-view sensor data.<n>This paper introduces a lightweight VLM called TS-VLM, which incorporates a novel Text-Guided SoftSort Pooling (TGSSP) module.
  arXiv  Detail & Related papers  (2025-05-19T03:37:15Z)
• SafeAuto: Knowledge-Enhanced Safe Autonomous Driving with Multimodal Foundation Models [63.71984266104757]
  We propose SafeAuto, a framework that enhances MLLM-based autonomous driving by incorporating both unstructured and structured knowledge.<n>To explicitly integrate safety knowledge, we develop a reasoning component that translates traffic rules into first-order logic.<n>Our Multimodal Retrieval-Augmented Generation model leverages video, control signals, and environmental attributes to learn from past driving experiences.
  arXiv  Detail & Related papers  (2025-02-28T21:53:47Z)
• RAC3: Retrieval-Augmented Corner Case Comprehension for Autonomous Driving with Vision-Language Models [9.304973961799359]
  Vision-language models (VLMs) play a crucial role in enhancing scenario comprehension.<n>They face challenges, such as hallucination and insufficient real-world grounding.<n>In this work, RAC3 is proposed to enhance the performance of VLMs in corner case comprehension.
  arXiv  Detail & Related papers  (2024-12-15T04:51:30Z)
• Can SAM Boost Video Super-Resolution? [78.29033914169025]
  We propose a simple yet effective module -- SAM-guidEd refinEment Module (SEEM) This light-weight plug-in module is specifically designed to leverage the attention mechanism for the generation of semantic-aware feature. We apply our SEEM to two representative methods, EDVR and BasicVSR, resulting in consistently improved performance with minimal implementation effort.
  arXiv  Detail & Related papers  (2023-05-11T02:02:53Z)
• Sparse Conditional Hidden Markov Model for Weakly Supervised Named Entity Recognition [68.68300358332156]
  We propose the sparse conditional hidden Markov model (Sparse-CHMM) to evaluate noisy labeling functions. Sparse-CHMM is optimized through unsupervised learning with a three-stage training pipeline. It achieves a 3.01 average F1 score improvement on five comprehensive datasets.
  arXiv  Detail & Related papers  (2022-05-27T20:47:30Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.