The GEST-Engine: From Event Graphs to Synthetic Video. A Full Technical Report

• URL: http://arxiv.org/abs/2607.12231v1
• Date: Tue, 14 Jul 2026 00:25:00 GMT
• Title: The GEST-Engine: From Event Graphs to Synthetic Video. A Full Technical Report
• Abstract summary: We present the GEST-Engine, a complete system that goes from natural-language text to fully-annotated multi-actor video.<n>At its core is an explicit world model: rather than encoding state as a learned latent, the engine maintains a complete, inspectable representation of the world.<n>We describe an in-game world editor, runtime capability extraction, a text-generation pipeline, and a production system that renders corpora at scale across parallel virtual machines.
• Score: 4.964902130083661
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We present the GEST-Engine, a complete system that goes from natural-language text to fully-annotated multi-actor video. At its core is an explicit world model: rather than encoding state as a learned latent, the engine maintains a complete, inspectable representation of the world (which actors exist, where they are, what they are doing, which objects they hold, and how events relate in time and space), expressed as a formal Graph of Events in Space and Time (GEST) and realized deterministically inside the open world of a commercial game engine driven through an open-source multiplayer scripting framework. GESTs are produced either procedurally or by an agentic text-to-GEST system in which an LLM Director plans a story through tool calls validated by a programmatic state backend, so every generated specification is executable by construction. A GEST then enters a four-stage execution pipeline: graph parsing and validation, entity and action grounding, temporal orchestration (Allen-style constraints resolved by Floyd-Warshall transitive closure), and execution and capture. In a single simulation pass the engine emits frame-aligned RGB video, dense per-pixel depth, instance segmentation, per-actor skeletal pose, per-frame pairwise spatial-relation graphs, 2D bounding boxes, event-to-frame temporal mappings, and natural-language descriptions, all at zero marginal annotation cost. We further describe an in-game world editor, runtime capability extraction, a text-generation pipeline, and a production system that renders corpora at scale across parallel virtual machines. Because every frame traces back to a semantic specification, the engine guarantees object permanence, multi-actor coordination, and temporal consistency by construction, making its output valuable as training data, evaluation benchmarks, and diagnostic tools for video understanding.

Related papers

• ESG: Generating Physically Consistent Dynamic 3D Scenes from Text Descriptions [19.29011683959245]
  We present a unified framework for generating physically consistent 3D scenes from text, with directly executable in Unreal Engine.<n>We use the emphEvolutive Scene Graph (ESG), which specifies entities with physical attributes, spatial relations, and event-driven timelines in a machine-checkable form.
  arXiv  Detail & Related papers  (2026-09-14T11:19:25Z)
• Building Pretraining Data for World Models: An Unreal Engine-Based Pipeline for Action-Conditioned Video Generation [53.851483224691634]
  We present a large-scale synthetic data production pipeline built on Unreal Engine for generating action-conditioned, multi-view video.<n>The pipeline has produced 2,691 hours of 1080p video and 6,076 hours of 720p video.
  arXiv  Detail & Related papers  (2026-09-03T08:58:04Z)
• SimWorlds: A Multi-Agent System for Dynamic 3D Scene Creation [50.175987640001644]
  We present SimWorlds: a multi-agent framework that produces dynamic, editable 4D scenes from text.<n>We also introduce 4DBuildBench, a benchmark for assessing both visual fidelity and physical consistency of the procedural dynamic 3D scenes.
  arXiv  Detail & Related papers  (2026-07-02T06:29:18Z)
• SceneGraphGrounder: Zero-Shot 3D Visual Grounding via Structured Scene Graph Matching [2.4737219202679808]
  We propose SceneGrapher, a framework that reformulates 3D grounding as structured graph matching over a reconstructed 3D scene graph.<n>We demonstrate our framework through real-world deployment on a mobile robot, demonstrating robust spatial reasoning in long-horizon physical environments.
  arXiv  Detail & Related papers  (2026-05-20T22:30:51Z)
• SceneCode: Executable World Programs for Editable Indoor Scenes with Articulated Objects [69.20984454755512]
  Indoor scene synthesis underpins embodied AI, robotic manipulation, and simulation-based policy evaluation.<n>Existing pipelines represent generated content as static meshes and inherit articulation only from curated asset libraries.<n>We present SceneCode, a framework that compiles a natural language prompt into an executable, code-driven indoor world.
  arXiv  Detail & Related papers  (2026-05-19T09:31:04Z)
• GTASA: Ground Truth Annotations for Spatiotemporal Analysis, Evaluation and Training of Video Models [4.964902130083661]
  We introduce GTASA, a corpus of multiactor videos with per-frame spatial- graphs and event-level temporal mappings.<n>We produce it based on Graphs of Events in Space and Time (EST): GEST-Engine.<n>Probing four frozen video encoders across 11temporal reasoning tasks enabled GTASA's exact 3D ground truth.
  arXiv  Detail & Related papers  (2026-04-12T00:01:51Z)
• Agentic Video Generation: From Text to Executable Event Graphs via Tool-Constrained LLM Planning [4.964902130083661]
  Existing multi-agent video generation systems use LLM agents to orchestrate neural video generators.<n>We present an agentic system that inverts this paradigm, constructing a formal Graph of Events in Space and Time.<n>A programmatic state backend enforces all simulator constraints through validated tool calls, guaranteeing that every generated specification is executable.
  arXiv  Detail & Related papers  (2026-04-11T23:51:13Z)
• LangDriveCTRL: Natural Language Controllable Driving Scene Editing with Multi-modal Agents [61.91651123290512]
  LangDriveCTRL is a framework for editing real-world driving videos to synthesize diverse traffic scenarios.<n>It supports both object node editing (removal, insertion and replacement) and multi-object behavior editing from a single natural-language instruction.
  arXiv  Detail & Related papers  (2025-12-19T10:57:03Z)
• SVAG-Bench: A Large-Scale Benchmark for Multi-Instance Spatio-temporal Video Action Grounding [48.64661382961745]
  We introduce Spatio-temporal Video Action Grounding (SVAG), a novel task that requires models to simultaneously detect, track, and temporally localize all referent objects in videos.<n>To support this task, we construct SVAG-Bench, a large-scale benchmark comprising 688 videos, 19,590 annotated records, and 903 unique verbs.<n> Empirical results show that existing models perform poorly on SVAG, particularly in dense or complex scenes.
  arXiv  Detail & Related papers  (2025-10-14T22:10:49Z)
• Agentic 3D Scene Generation with Spatially Contextualized VLMs [67.31920821192323]
  We introduce a new paradigm that enables vision-language models to generate, understand, and edit complex 3D environments.<n>We develop an agentic 3D scene generation pipeline in which the VLM iteratively reads from and updates the spatial context.<n>Results show that our framework can handle diverse and challenging inputs, achieving a level of generalization not observed in prior work.
  arXiv  Detail & Related papers  (2025-05-26T15:28:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.