UAV traffic scene understanding: A regulation embedded multi-modal network and a unified benchmark

• URL: http://arxiv.org/abs/2603.10722v2
• Date: Sat, 14 Mar 2026 08:04:56 GMT
• Title: UAV traffic scene understanding: A regulation embedded multi-modal network and a unified benchmark
• Abstract summary: Traffic scene understanding from unmanned aerial vehicle (UAV) platforms is crucial for intelligent transportation systems.<n>Existing methods face significant challenges in real-world surveillance, as their heavy reliance on optical imagery leads to severe performance degradation.<n>We propose a novel Multi-modal Traffic Cognition Network (MTCNet) for robust UAV traffic scene understanding.
• Score: 22.241798167957214
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Traffic scene understanding from unmanned aerial vehicle (UAV) platforms is crucial for intelligent transportation systems due to its flexible deployment and wide-area monitoring capabilities. However, existing methods face significant challenges in real-world surveillance, as their heavy reliance on optical imagery leads to severe performance degradation under adverse illumination conditions like nighttime and fog. Furthermore, current Visual Question Answering (VQA) models are restricted to elementary perception tasks, lacking the domain-specific regulatory knowledge required to assess complex traffic behaviors. To address these limitations, we propose a novel Multi-modal Traffic Cognition Network (MTCNet) for robust UAV traffic scene understanding. Specifically, we design a Prototype-Guided Knowledge Embedding (PGKE) module that leverages high-level semantic prototypes from an external Traffic Regulation Memory (TRM) to anchor domain-specific knowledge into visual representations, enabling the model to comprehend complex behaviors and distinguish fine-grained traffic violations. Moreover, we develop a Quality-Aware Spectral Compensation (QASC) module that exploits the complementary characteristics of optical and thermal modalities to perform bidirectional context exchange, effectively compensating for degraded features to ensure robust representation in complex environments. In addition, we construct Traffic-VQA, the first large-scale optical-thermal infrared benchmark for cognitive UAV traffic understanding, comprising 8,180 aligned image pairs and 1.3 million question-answer pairs across 31 diverse types. Extensive experiments demonstrate that MTCNet significantly outperforms state-of-the-art methods in both cognition and perception scenarios. The dataset is available at https://github.com/YuZhang-2004/UAV-traffic-scene-understanding.

Related papers

• A Review of Vision-Based Vehicle Detection for UAV-Based Traffic Monitoring: Experimental Insights and Future Directions [4.8342038441006805]
  Unmanned aerial vehicle (UAV)-based surveillance offers an innovative solution to traffic surveillance with wide coverage and real-time data collection capabilities.<n>In comparison to fixed ground-based infrastructure, UAVs are able to respond to dynamic traffic but present challenges such as vehicle detection at varying altitudes.<n>Deep learning has been largely beneficial on improving the detection accuracy; however, for practical deployment, a critical assessment of the accuracy, latency, and harmonization with current transportation systems needs to be carefully considered.
  arXiv  Detail & Related papers  (2026-08-04T08:15:38Z)
• OmniTraffic: A Controllable Generation Pipeline and Benchmark for Spatio-Temporal Traffic Reasoning [47.85568621735744]
  We introduce OmniTraffic, a controllable pipeline generation and benchmark for traffic reasoning.<n>It is built around 12 real-world intersections and reconstructed by surveillance footage from two countries.<n>It defines a three-level task hierarchy spanning scene perception, multi-view and temporal reasoning, and decision support.
  arXiv  Detail & Related papers  (2026-06-14T11:16:53Z)
• V2XCrafter: Learning to Generate Driving Scene Across Agents [52.22336670088219]
  Collaborative driving systems leverage vehicle-to-everything (V2X) communication for collaborative perception to enhance driving safety.<n>We propose V2XCrafter, the first framework for generating controllable and realistic collaborative driving scene across agents' camera views.<n>Experiments have shown that V2XCrafter can generate high-fidelity and controllable street views with consistency across agents, thereby effectively enhancing the downstream collaborative 3D object detection tasks.
  arXiv  Detail & Related papers  (2026-05-28T07:03:51Z)
• Surveillance Video-Based Traffic Accident Detection Using Transformer Architecture [2.621034368312571]
  Traffic accidents represent a leading cause of mortality globally, with incidence rates due to increasing population, urbanization and motorization.<n>Traditional computer methods for accident detection struggle with limited understanding and poor cross-domain generalization.<n>We propose an accident detection model based on a transformer architecture using pre-extracted spatial video features.
  arXiv  Detail & Related papers  (2025-12-12T07:57:36Z)
• RoadSceneVQA: Benchmarking Visual Question Answering in Roadside Perception Systems for Intelligent Transportation System [15.222742182076459]
  RoadSceneVQA is a large-scale visual question answering dataset specifically tailored for roadside scenarios.<n>The dataset comprises 34,736 diverse QA pairs collected under varying weather, illumination, and traffic conditions.<n>RoadSceneVQA challenges models to perform both explicit recognition and implicit commonsense reasoning.
  arXiv  Detail & Related papers  (2025-11-23T04:40:50Z)
• FlowXpert: Context-Aware Flow Embedding for Enhanced Traffic Detection in IoT Network [7.30584204219718]
  In the Internet of Things (IoT) environment, continuous interaction among a large number of devices generates complex and dynamic network traffic.<n>Machine learning (ML)-based traffic detection technology serves as a critical component in ensuring network security.
  arXiv  Detail & Related papers  (2025-09-25T07:52:58Z)
• Traffic-MLLM: A Spatio-Temporal MLLM with Retrieval-Augmented Generation for Causal Inference in Traffic [8.754321713184483]
  We propose Traffic-LM, a multimodal large language model tailored for fine-grained traffic analysis.<n>Our model leverages high-quality traffic-specific multimodal datasets and uses LowRanktemporal Adaptation (LoRA) for lightweight fine-tuning.<n>We also introduce an innovative knowledge module fusing Chain-of-the-art reasoning with Retrieval-Lomented Generation (LoRAG)
  arXiv  Detail & Related papers  (2025-09-14T08:53:06Z)
• Contrastive Learning-Driven Traffic Sign Perception: Multi-Modal Fusion of Text and Vision [2.0720154517628417]
  We propose a novel framework combining open-vocabulary detection and cross-modal learning.<n>For traffic sign detection, our NanoVerse YOLO model integrates a vision-language path aggregation network (RepVL-PAN) and an SPD-Conv module.<n>For traffic sign classification, we designed a Traffic Sign Recognition Multimodal Contrastive Learning model (TSR-MCL)<n>On the TT100K dataset, our method achieves a state-of-the-art 78.4% mAP in the long-tail detection task for all-class recognition.
  arXiv  Detail & Related papers  (2025-07-31T08:23:30Z)
• Towards Intelligent Transportation with Pedestrians and Vehicles In-the-Loop: A Surveillance Video-Assisted Federated Digital Twin Framework [62.47416496137193]
  We propose a surveillance video assisted federated digital twin (SV-FDT) framework to empower ITSs with pedestrians and vehicles in-the-loop.<n>The architecture consists of three layers: (i) the end layer, which collects traffic surveillance videos from multiple sources; (ii) the edge layer, responsible for semantic segmentation-based visual understanding, twin agent-based interaction modeling, and local digital twin system (LDTS) creation in local regions; and (iii) the cloud layer, which integrates LDTSs across different regions to construct a global DT model in realtime.
  arXiv  Detail & Related papers  (2025-03-06T07:36:06Z)
• Semantic Communication for Cooperative Perception using HARQ [51.148203799109304]
  We leverage an importance map to distill critical semantic information, introducing a cooperative perception semantic communication framework. To counter the challenges posed by time-varying multipath fading, our approach incorporates the use of frequency-division multiplexing (OFDM) along with channel estimation and equalization strategies. We introduce a novel semantic error detection method that is integrated with our semantic communication framework in the spirit of hybrid automatic repeated request (HARQ)
  arXiv  Detail & Related papers  (2024-08-29T08:53:26Z)
• A Holistic Framework Towards Vision-based Traffic Signal Control with Microscopic Simulation [53.39174966020085]
  Traffic signal control (TSC) is crucial for reducing traffic congestion that leads to smoother traffic flow, reduced idling time, and mitigated CO2 emissions. In this study, we explore the computer vision approach for TSC that modulates on-road traffic flows through visual observation. We introduce a holistic traffic simulation framework called TrafficDojo towards vision-based TSC and its benchmarking.
  arXiv  Detail & Related papers  (2024-03-11T16:42:29Z)
• OpenLane-V2: A Topology Reasoning Benchmark for Unified 3D HD Mapping [84.65114565766596]
  We present OpenLane-V2, the first dataset on topology reasoning for traffic scene structure. OpenLane-V2 consists of 2,000 annotated road scenes that describe traffic elements and their correlation to the lanes. We evaluate various state-of-the-art methods, and present their quantitative and qualitative results on OpenLane-V2 to indicate future avenues for investigating topology reasoning in traffic scenes.
  arXiv  Detail & Related papers  (2023-04-20T16:31:22Z)
• Road Network Guided Fine-Grained Urban Traffic Flow Inference [108.64631590347352]
  Accurate inference of fine-grained traffic flow from coarse-grained one is an emerging yet crucial problem. We propose a novel Road-Aware Traffic Flow Magnifier (RATFM) that exploits the prior knowledge of road networks. Our method can generate high-quality fine-grained traffic flow maps.
  arXiv  Detail & Related papers  (2021-09-29T07:51:49Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.