RailVQA: A Benchmark and Framework for Efficient Interpretable Visual Cognition in Automatic Train Operation

• URL: http://arxiv.org/abs/2603.27112v1
• Date: Sat, 28 Mar 2026 03:41:40 GMT
• Title: RailVQA: A Benchmark and Framework for Efficient Interpretable Visual Cognition in Automatic Train Operation
• Abstract summary: We introduce RailVQA-bench, the first VQA benchmark for cab-view visual cognition in Automatic Train Operation (ATO)<n>We propose RailVQA-CoM, a collaborative large-small model framework that combines small-model efficiency with large-model cognition.<n> Experiments demonstrate that the proposed approach substantially improves performance, enhances interpretability, reduces inference latency, and strengthens cross-domain generalization.
• Score: 11.03879484190242
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Automatic Train Operation (ATO) relies on low-latency, reliable cab-view visual perception and decision-oriented inference to ensure safe operation in complex and dynamic railway environments. However, existing approaches focus primarily on basic perception and often generalize poorly to rare yet safety-critical corner cases. They also lack the high-level reasoning and planning capabilities required for operational decision-making. Although recent Large Multi-modal Models (LMMs) show strong generalization and cognitive capabilities, their use in safety-critical ATO is hindered by high computational cost and hallucination risk. Meanwhile, reliable domain-specific benchmarks for systematically evaluating cognitive capabilities are still lacking. To address these gaps, we introduce RailVQA-bench, the first VQA benchmark for cab-view visual cognition in ATO, comprising 20,000 single-frame and 1,168 video based QA pairs to evaluate cognitive generalization and interpretability in both static and dynamic scenarios. Furthermore, we propose RailVQA-CoM, a collaborative large-small model framework that combines small-model efficiency with large-model cognition via a transparent three-module architecture and adaptive temporal sampling, improving perceptual generalization and enabling efficient reasoning and planning. Experiments demonstrate that the proposed approach substantially improves performance, enhances interpretability, reduces inference latency, and strengthens cross-domain generalization, while enabling plug-and-play deployment in autonomous driving systems. Code and datasets will be available at https://github.com/Cybereye-bjtu/RailVQA.

Related papers

• From Skeletons to Semantics: Design and Deployment of a Hybrid Edge-Based Action Detection System for Public Safety [0.0]
  This paper presents the design and demonstrator-based deployment of a hybrid edge-based action detection system.<n>It combines skeleton-based motion analysis with vision-language models for semantic scene interpretation.<n>The presented system provides a practical foundation for privacy-aware, real-time video analysis in public safety applications.
  arXiv  Detail & Related papers  (2026-03-31T14:16:16Z)
• UAV traffic scene understanding: A regulation embedded multi-modal network and a unified benchmark [22.241798167957214]
  Traffic scene understanding from unmanned aerial vehicle (UAV) platforms is crucial for intelligent transportation systems.<n>Existing methods face significant challenges in real-world surveillance, as their heavy reliance on optical imagery leads to severe performance degradation.<n>We propose a novel Multi-modal Traffic Cognition Network (MTCNet) for robust UAV traffic scene understanding.
  arXiv  Detail & Related papers  (2026-03-11T12:53:38Z)
• CogRail: Benchmarking VLMs in Cognitive Intrusion Perception for Intelligent Railway Transportation Systems [29.385460126069386]
  We introduce a novel benchmark, CogRail, which integrates curated datasets with cognitively driven question-answer annotations.<n>Building upon this benchmark, we conduct a systematic evaluation of state-of-the-art visual-language models.<n>We propose a joint fine-tuning framework that integrates three core tasks, position perception, movement prediction, and threat analysis.
  arXiv  Detail & Related papers  (2026-01-14T16:36:26Z)
• COVLM-RL: Critical Object-Oriented Reasoning for Autonomous Driving Using VLM-Guided Reinforcement Learning [55.83415345423854]
  We propose COVLM-RL, a novel end-to-end driving framework that integrates Critical Object-oriented (CO) reasoning with reinforcement learning (RL)<n>Experiments conducted in the CARLA simulator demonstrate that COVLM-RL significantly improves the success rate by 30% in trained driving environments.
  arXiv  Detail & Related papers  (2025-12-10T06:18:16Z)
• dVLM-AD: Enhance Diffusion Vision-Language-Model for Driving via Controllable Reasoning [69.36145467833498]
  We introduce dVLM-AD, a diffusion-based vision-language model that unifies perception, structured reasoning, and low-level planning for end-to-end driving.<n> evaluated on nuScenes and WOD-E2E, dVLM-AD yields more consistent reasoning-action pairs and achieves planning performance comparable to existing driving VLM/VLA systems.
  arXiv  Detail & Related papers  (2025-12-04T05:05:41Z)
• AutoDrive-R$^2$: Incentivizing Reasoning and Self-Reflection Capacity for VLA Model in Autonomous Driving [37.260140808367716]
  We propose AutoDrive-R$2$, a novel VLA framework that enhances both reasoning and self-reflection capabilities of autonomous driving systems.<n>We first propose an innovative CoT dataset named nuScenesR$2$-6K for supervised fine-tuning.<n>We then employ the Group Relative Policy Optimization (GRPO) algorithm within a physics-grounded reward framework to ensure reliable smoothness and realistic trajectory planning.
  arXiv  Detail & Related papers  (2025-09-02T04:32:24Z)
• ReAgent-V: A Reward-Driven Multi-Agent Framework for Video Understanding [71.654781631463]
  ReAgent-V is a novel agentic video understanding framework.<n>It integrates efficient frame selection with real-time reward generation during inference.<n>Extensive experiments on 12 datasets demonstrate significant gains in generalization and reasoning.
  arXiv  Detail & Related papers  (2025-06-02T04:23:21Z)
• SOLVE: Synergy of Language-Vision and End-to-End Networks for Autonomous Driving [51.47621083057114]
  SOLVE is an innovative framework that synergizes Vision-Language Models with end-to-end (E2E) models to enhance autonomous vehicle planning.<n>Our approach emphasizes knowledge sharing at the feature level through a shared visual encoder, enabling comprehensive interaction between VLM and E2E components.
  arXiv  Detail & Related papers  (2025-05-22T15:44:30Z)
• DriveLMM-o1: A Step-by-Step Reasoning Dataset and Large Multimodal Model for Driving Scenario Understanding [76.3876070043663]
  We propose DriveLMM-o1, a dataset and benchmark designed to advance step-wise visual reasoning for autonomous driving.<n>Our benchmark features over 18k VQA examples in the training set and more than 4k in the test set, covering diverse questions on perception, prediction, and planning.<n>Our model achieves a +7.49% gain in final answer accuracy, along with a 3.62% improvement in reasoning score over the previous best open-source model.
  arXiv  Detail & Related papers  (2025-03-13T17:59:01Z)
• RAC3: Retrieval-Augmented Corner Case Comprehension for Autonomous Driving with Vision-Language Models [9.304973961799359]
  Vision-language models (VLMs) play a crucial role in enhancing scenario comprehension.<n>They face challenges, such as hallucination and insufficient real-world grounding.<n>In this work, RAC3 is proposed to enhance the performance of VLMs in corner case comprehension.
  arXiv  Detail & Related papers  (2024-12-15T04:51:30Z)
• Hints of Prompt: Enhancing Visual Representation for Multimodal LLMs in Autonomous Driving [55.609997552148826]
  We propose the Hints of Prompt (HoP) framework, which introduces three key enhancements.<n>These hints are fused through a Hint Fusion module, enriching visual representations by capturing driving-related representations with limited domain data.<n>Extensive experiments confirm the effectiveness of the HoP framework, showing that it significantly outperforms previous state-of-the-art methods in all key metrics.
  arXiv  Detail & Related papers  (2024-11-20T06:58:33Z)
• Scalable Vehicle Re-Identification via Self-Supervision [66.2562538902156]
  Vehicle Re-Identification is one of the key elements in city-scale vehicle analytics systems. Many state-of-the-art solutions for vehicle re-id mostly focus on improving the accuracy on existing re-id benchmarks and often ignore computational complexity. We propose a simple yet effective hybrid solution empowered by self-supervised training which only uses a single network during inference time.
  arXiv  Detail & Related papers  (2022-05-16T12:14:42Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.