SGTA: Scene-Graph Based Multi-Modal Traffic Agent for Video Understanding

• URL: http://arxiv.org/abs/2604.03697v1
• Date: Sat, 04 Apr 2026 12:01:49 GMT
• Title: SGTA: Scene-Graph Based Multi-Modal Traffic Agent for Video Understanding
• Abstract summary: Scene-Graph Based Multi-Modal Traffic Agent (SGTA) is a modular framework for traffic video understanding.<n>It constructs a traffic scene graph from roadside videos using detection, tracking, and lane extraction, followed by tool-based reasoning over both symbolic graph queries and visual inputs.
• Score: 48.47968852253859
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We present Scene-Graph Based Multi-Modal Traffic Agent (SGTA), a modular framework for traffic video understanding that combines structured scene graphs with multi-modal reasoning. It constructs a traffic scene graph from roadside videos using detection, tracking, and lane extraction, followed by tool-based reasoning over both symbolic graph queries and visual inputs. SGTA adopts ReAct to process interleaved reasoning traces from large language models with tool invocations, enabling interpretable decision-making for complex video questions. Experiments on selected TUMTraffic VideoQA dataset sample demonstrate that SGTA achieves competitive accuracy across multiple question types while providing transparent reasoning steps. These results highlight the potential of integrating structured scene representations with multi-modal agents for traffic video understanding.

Related papers

• Scene Graph Thinking: Reinforcing Structured Visual Reasoning for Multimodal Large Language Models [46.09809083890632]
  Scene Graph Thinking (SaGe) is a novel paradigm that enables fine-grained and structured visual reasoning through explicit scene-graph representations.<n>We construct 120K high-quality training data by sampling reasoning traces from scene graphs.<n>With curated data and graph-aligned training, our approach achieves significant improvements across eight multimodal benchmarks.
  arXiv  Detail & Related papers  (2026-07-07T01:00:51Z)
• TrafficLens: Multi-Camera Traffic Video Analysis Using LLMs [8.205106134817763]
  efficiently managing and analyzing multi-camera feeds poses challenges due to the vast amount of data.<n>To address these challenges, we propose TrafficLens, a tailored algorithm for multi-camera traffic intersections.<n> Experimental results with real-world datasets demonstrate that TrafficLens reduces video-to-text conversion time by up to $4times$ while maintaining information accuracy.
  arXiv  Detail & Related papers  (2025-11-26T01:34:08Z)
• Language-Guided Graph Representation Learning for Video Summarization [96.2763459348758]
  We propose a novel Language-guided Graph Representation Learning Network (LGRLN) for video summarization.<n>Specifically, we introduce a video graph generator that converts video frames into a structured graph to preserve temporal order and contextual dependencies.<n>Our method outperforms existing approaches across multiple benchmarks.
  arXiv  Detail & Related papers  (2025-11-14T04:35:48Z)
• FrameMind: Frame-Interleaved Video Reasoning via Reinforcement Learning [65.42201665046505]
  Current video understanding models rely on fixed frame sampling strategies, processing predetermined visual inputs regardless of the specific reasoning requirements of each question.<n>This static approach limits their ability to adaptively gather visual evidence, leading to suboptimal performance on tasks that require broad temporal coverage or fine-grained spatial detail.<n>We introduce FrameMind, an end-to-end framework trained with reinforcement learning that enables models to dynamically request visual information during reasoning through Frame-Interleaved Chain-of-Thought (FiCOT)<n>Unlike traditional approaches, FrameMind operates in multiple turns where the model alternates between textual reasoning and active visual perception, using tools to extract
  arXiv  Detail & Related papers  (2025-09-28T17:59:43Z)
• Multimodal Referring Segmentation: A Survey [93.24051010753817]
  Multimodal referring segmentation aims to segment target objects in visual scenes, such as images, videos, and 3D scenes, based on referring expressions in text or audio format.<n>Over the past decade, it has gained significant attention in the multimodal community, driven by advances in convolutional neural networks, transformers, and large language models.
  arXiv  Detail & Related papers  (2025-08-01T02:14:00Z)
• When language and vision meet road safety: leveraging multimodal large language models for video-based traffic accident analysis [6.213279061986497]
  SeeUnsafe is a framework that transforms video-based traffic accident analysis into a more interactive, conversational approach.<n>Our framework employs a multimodal-based aggregation strategy to handle videos of various lengths and generate structured responses for review and evaluation.<n>We conduct extensive experiments on the Toyota Woven Traffic Safety dataset, demonstrating that SeeUnsafe effectively performs accident-aware video classification and visual grounding.
  arXiv  Detail & Related papers  (2025-01-17T23:35:34Z)
• TrafficVLM: A Controllable Visual Language Model for Traffic Video Captioning [0.0]
  We present TrafficVLM, a novel multi-modal dense video captioning model for vehicle ego camera view. Our solution achieved outstanding results in Track 2 of the AI City Challenge 2024, ranking us third in the challenge standings.
  arXiv  Detail & Related papers  (2024-04-14T14:51:44Z)
• Traffic Scene Parsing through the TSP6K Dataset [109.69836680564616]
  We introduce a specialized traffic monitoring dataset, termed TSP6K, with high-quality pixel-level and instance-level annotations. The dataset captures more crowded traffic scenes with several times more traffic participants than the existing driving scenes. We propose a detail refining decoder for scene parsing, which recovers the details of different semantic regions in traffic scenes.
  arXiv  Detail & Related papers  (2023-03-06T02:05:14Z)
• Dynamic Graph Representation Learning for Video Dialog via Multi-Modal Shuffled Transformers [89.00926092864368]
  We present a semantics-controlled multi-modal shuffled Transformer reasoning framework for the audio-visual scene aware dialog task. We also present a novel dynamic scene graph representation learning pipeline that consists of an intra-frame reasoning layer producing-semantic graph representations for every frame. Our results demonstrate state-of-the-art performances on all evaluation metrics.
  arXiv  Detail & Related papers  (2020-07-08T02:00:22Z)
• Understanding Dynamic Scenes using Graph Convolution Networks [22.022759283770377]
  We present a novel framework to model on-road vehicle behaviors from a sequence of temporally ordered frames as grabbed by a moving camera. We show a seamless transfer of learning to multiple datasets without resorting to fine-tuning. Such behavior prediction methods find immediate relevance in a variety of navigation tasks.
  arXiv  Detail & Related papers  (2020-05-09T13:05:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.