SceneGraphVLM: Dynamic Scene Graph Generation from Video with Vision-Language Models

• URL: http://arxiv.org/abs/2605.13667v1
• Date: Wed, 13 May 2026 15:27:41 GMT
• Title: SceneGraphVLM: Dynamic Scene Graph Generation from Video with Vision-Language Models
• Abstract summary: SceneGraphVLM is a compact method for image and video scene graph generation with small visual language models.<n>SceneGraphVLM serializes graphs in a token-efficient TOON format and trains the model in two stages.<n>We evaluate SceneGraphVLM on PSG, PVSG, and Action Genome.
• Score: 0.25489046505746704
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Scene graph generation provides a compact structured representation for visual perception, but accurate and fast graph prediction from images and videos remains challenging. Recent VLM-based methods can generate scene graphs end-to-end as structured text, yet often produce long outputs with irrelevant objects and relations. We present SceneGraphVLM, a compact method for image and video scene graph generation with small visual language models. SceneGraphVLM serializes graphs in a token-efficient TOON format and trains the model in two stages: supervised fine-tuning followed by reinforcement learning with hallucination-aware rewards that balance relation coverage and precision while penalizing unsupported objects and relations. For videos, the model can optionally condition each frame on the previously generated graph, providing lightweight short-term context without tracking or post-processing. We evaluate SceneGraphVLM on PSG, PVSG, and Action Genome. With compact VLMs and vLLM-accelerated decoding, SceneGraphVLM achieves a strong quality-speed trade-off, improves precision-oriented SGG metrics while preserving reasonable recall, and generates complete scene graphs with approximately one-second latency. Code and implementation details are available at: https://github.com/markus0440/SceneGraphVLM.git.

Related papers

• Dependency-Aware Discrete Diffusion for Scene Graph Generation [8.87774679281067]
  We introduce a dependency-aware, hierarchically constrained discrete diffusion model for scene graph generation.<n>Our approach decouples structure and semantics across the forward and reverse processes, enabling the model to capture conditional dependencies.<n>When fed to downstream image generation, our approach yields improved compositional alignment compared to text-to-image models.
  arXiv  Detail & Related papers  (2026-05-09T17:16:20Z)
• FDSG: Forecasting Dynamic Scene Graphs [41.18167591493808]
  We propose a novel framework that predicts future entity labels, bounding boxes, and relationships for unobserved frames.<n>A temporal aggregation module further refines predictions by integrating forecasted and observed information via crossattention.<n>Experiments on Action Genome show that FDSG outperforms state-of-the-art methods on dynamic scene graph generation, scene graph anticipation, and scene graph forecasting.
  arXiv  Detail & Related papers  (2025-06-02T09:46:22Z)
• Generate Any Scene: Scene Graph Driven Data Synthesis for Visual Generation Training [61.75337990107149]
  We introduce Generate Any Scene, a data engine that enumerates scene graphs representing an array of possible visual scenes.<n>Given a sampled scene graph, Generate Any Scene translates it into a caption for text-to-image or text-to-video generation.<n>It also translates it into a set of visual question answers that allow automatic evaluation and reward modeling of semantic alignment.
  arXiv  Detail & Related papers  (2024-12-11T09:17:39Z)
• TESGNN: Temporal Equivariant Scene Graph Neural Networks for Efficient and Robust Multi-View 3D Scene Understanding [9.702862490344485]
  We propose Temporal Equivariant Scene Graph Neural Network (TESGNN), consisting of two key components.<n>ESGNN extracts information from 3D point clouds to generate scene graph while preserving crucial symmetry properties.<n>We show that leveraging the symmetry-preserving property produces a more stable and accurate global scene representation.
  arXiv  Detail & Related papers  (2024-11-15T15:39:04Z)
• From Pixels to Graphs: Open-Vocabulary Scene Graph Generation with Vision-Language Models [81.92098140232638]
  Scene graph generation (SGG) aims to parse a visual scene into an intermediate graph representation for downstream reasoning tasks. Existing methods struggle to generate scene graphs with novel visual relation concepts. We introduce a new open-vocabulary SGG framework based on sequence generation.
  arXiv  Detail & Related papers  (2024-04-01T04:21:01Z)
• FACTUAL: A Benchmark for Faithful and Consistent Textual Scene Graph Parsing [66.70054075041487]
  Existing scene graphs that convert image captions into scene graphs often suffer from two types of errors. First, the generated scene graphs fail to capture the true semantics of the captions or the corresponding images, resulting in a lack of faithfulness. Second, the generated scene graphs have high inconsistency, with the same semantics represented by different annotations.
  arXiv  Detail & Related papers  (2023-05-27T15:38:31Z)
• Incorporating Structured Representations into Pretrained Vision & Language Models Using Scene Graphs [79.64891686479213]
  We show that it is possible to improve vision and language models (VLMs) when learning from scene graphs (SGs) For the visual side, we incorporate a special "SG Component" in the image transformer trained to predict SG information, while for the textual side, we utilize SGs to generate fine-grained captions. Our method improves the performance of several popular VLMs on multiple datasets with only a mild degradation in ZS capabilities.
  arXiv  Detail & Related papers  (2023-05-10T17:52:26Z)
• Diffusion-Based Scene Graph to Image Generation with Masked Contrastive Pre-Training [112.94542676251133]
  We propose to learn scene graph embeddings by directly optimizing their alignment with images. Specifically, we pre-train an encoder to extract both global and local information from scene graphs. The resulting method, called SGDiff, allows for the semantic manipulation of generated images by modifying scene graph nodes and connections.
  arXiv  Detail & Related papers  (2022-11-21T01:11:19Z)
• Graph-to-3D: End-to-End Generation and Manipulation of 3D Scenes Using Scene Graphs [85.54212143154986]
  Controllable scene synthesis consists of generating 3D information that satisfy underlying specifications. Scene graphs are representations of a scene composed of objects (nodes) and inter-object relationships (edges) We propose the first work that directly generates shapes from a scene graph in an end-to-end manner.
  arXiv  Detail & Related papers  (2021-08-19T17:59:07Z)
• Unconditional Scene Graph Generation [72.53624470737712]
  We develop a deep auto-regressive model called SceneGraphGen which can learn the probability distribution over labelled and directed graphs. We show that the scene graphs generated by SceneGraphGen are diverse and follow the semantic patterns of real-world scenes.
  arXiv  Detail & Related papers  (2021-08-12T17:57:16Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.