Agentic Video Generation: From Text to Executable Event Graphs via Tool-Constrained LLM Planning

• URL: http://arxiv.org/abs/2604.10383v1
• Date: Sat, 11 Apr 2026 23:51:13 GMT
• Title: Agentic Video Generation: From Text to Executable Event Graphs via Tool-Constrained LLM Planning
• Abstract summary: Existing multi-agent video generation systems use LLM agents to orchestrate neural video generators.<n>We present an agentic system that inverts this paradigm, constructing a formal Graph of Events in Space and Time.<n>A programmatic state backend enforces all simulator constraints through validated tool calls, guaranteeing that every generated specification is executable.
• Score: 4.964902130083661
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Existing multi-agent video generation systems use LLM agents to orchestrate neural video generators, producing visually impressive but semantically unreliable outputs with no ground truth annotations. We present an agentic system that inverts this paradigm: instead of generating pixels, the LLM constructs a formal Graph of Events in Space and Time (GEST) -- a structured specification of actors, actions, objects, and temporal constraints -- which is then executed deterministically in a 3D game engine. A staged LLM refinement pipeline fails entirely at this task (0 of 50 attempts produce an executable specification), motivating a fundamentally different architecture based on a separation of concerns: the LLM handles narrative planning through natural language reasoning, while a programmatic state backend enforces all simulator constraints through validated tool calls, guaranteeing that every generated specification is executable by construction. The system uses a hierarchical two-agent architecture -- a Director that plans the story and a Scene Builder that constructs individual scenes through a round-based state machine -- with dedicated Relation Subagents that populate the logical and semantic edge types of the GEST formalism that procedural generation leaves empty, making this the first approach to exercise the full expressive capacity of the representation. We evaluate in two stages: autonomous generation against procedural baselines via a 3-model LLM jury, where agentic narratives win 79% of text and 74% of video comparisons; and seeded generation where the same text is given to our system, VEO 3.1, and WAN 2.2, with human annotations showing engine-generated videos substantially outperform neural generators on physical validity (58% vs 25% and 20%) and semantic alignment (3.75/5 vs 2.33 and 1.50).

Related papers

• Beyond Text Conditioning: A Systematic Study of MLLM-DiT Fusion for Video Generation [50.92925099642139]
  Diffusion Transformers (DiTs) have become the dominant paradigm for high-fidelity video generation, yet their ability to perform high-level semantic planning remains limited.<n>We propose BiVidGen, a hybrid framework where an MLLM first generates semantic visual tokens and a DiT renders videos conditioned on both text and these tokens via multi-layer cross-attention.
  arXiv  Detail & Related papers  (2026-08-14T07:40:41Z)
• VideoCoCo: Code-as-CoT for Physically-Consistent Video Generation via an Agentic Dual-Engine System [77.32035315337522]
  Text-video models struggle to generate physically consistent dynamics the temporal evolution of a scene must be inferred implicitly from a compressed text prompt.<n>Existing chain-of-thought approaches introduce intermediate plans or visual states, but these representations are typically non-cutable or temporally sparse.<n>We introduce VideoCo, an agentic dual-engine framework in which executable Blender code serves as a process-level chain of thought.
  arXiv  Detail & Related papers  (2026-07-29T18:38:23Z)
• The GEST-Engine: From Event Graphs to Synthetic Video. A Full Technical Report [4.964902130083661]
  We present the GEST-Engine, a complete system that goes from natural-language text to fully-annotated multi-actor video.<n>At its core is an explicit world model: rather than encoding state as a learned latent, the engine maintains a complete, inspectable representation of the world.<n>We describe an in-game world editor, runtime capability extraction, a text-generation pipeline, and a production system that renders corpora at scale across parallel virtual machines.
  arXiv  Detail & Related papers  (2026-07-14T00:25:00Z)
• SR-Platform: An Agentic Pipeline for Natural Language-Driven Robot Simulation Environment Synthesis [0.3499870393443268]
  SR-Platform is a production-deployed agentic system that converts free-form natural language descriptions into executable MuJoCo environments.<n>It decomposes scene synthesis into four stages: an orchestrator that converts user intent into a structured scene plan; an asset forge that retrieves cached assets or generates new 3D geometry.<n>The system is deployed as a nine-service Docker stack with WebSocket progress streaming, MinIO-backed mesh storage, Qdrant-based semantic asset retrieval, Redis job state, and InfluxDB telemetry.
  arXiv  Detail & Related papers  (2026-05-14T11:14:20Z)
• OneDrive: Unified Multi-Paradigm Driving with Vision-Language-Action Models [69.2503510410147]
  We present a unified autonomous driving framework built upon a pretrained VLM.<n>We demonstrate that pretrained VLM attention exhibits strong transferability beyond pure language modeling.<n>Experiments on end-to-end autonomous driving benchmarks demonstrate state-of-the-art performance.
  arXiv  Detail & Related papers  (2026-04-20T07:50:00Z)
• From Agent Loops to Structured Graphs:A Scheduler-Theoretic Framework for LLM Agent Execution [1.8222732878503212]
  The dominant paradigm for building LLM based agents is the Agent Loop, an iterative cycle where a single language model decides what to do next by reading an ever growing context window.<n>This perspective places Agent Loops and graph based execution engines on a single semantic continuum.<n>We propose SGH, which lifts control flow from implicit context into an explicit static DAG.
  arXiv  Detail & Related papers  (2026-04-13T12:16:45Z)
• OmniScript: Towards Audio-Visual Script Generation for Long-Form Cinematic Video [59.391671069156274]
  This paper introduces the novel video-to-script (V2S) task, aiming to generate hierarchical, scene-by-scene scripts.<n>We present OmniScript, an 8B- parameter omni-modal (audio-visual) language model tailored for long-form narrative comprehension.
  arXiv  Detail & Related papers  (2026-04-13T07:19:27Z)
• Learning Trajectory-Aware Multimodal Large Language Models for Video Reasoning Segmentation [55.743701532984126]
  We propose TrajSeg, a simple and unified framework built upon Multimodal Large Language Models.<n>In this work, we introduce bidirectional text-trajectory alignment, where MLLMs accept grounding-intended (text-to-trajectory) and captioning-intended (trajectory-to-text) instructions.<n>This way, MLLMs can benefit from enhanced correspondence and better perceive object trajectories in videos.
  arXiv  Detail & Related papers  (2026-03-23T02:25:51Z)
• RAVEL: Reasoning Agents for Validating and Evaluating LLM Text Synthesis [78.32151470154422]
  We introduce RAVEL, an agentic framework that enables the testers to autonomously plan and execute typical synthesis operations.<n>We present C3EBench, a benchmark comprising 1,258 samples derived from professional human writings.<n>By augmenting RAVEL with SOTA LLMs as operators, we find that such agentic text synthesis is dominated by the LLM's reasoning capability.
  arXiv  Detail & Related papers  (2026-02-28T14:47:34Z)
• Beyond End-to-End Video Models: An LLM-Based Multi-Agent System for Educational Video Generation [15.004606775581356]
  LAVES is a hierarchical multi-agent system for generating high-quality instructional videos from educational problems.<n>In large-scale deployments, LAVES achieves a throughput exceeding one million videos per day, delivering over a 95% reduction in cost.
  arXiv  Detail & Related papers  (2026-02-12T10:14:36Z)
• Towards AI-Assisted Generation of Military Training Scenarios [3.491673646235722]
  This paper introduces a multi-agent, multi-modal reasoning framework that leverages Large Language Models (LLMs) to generate critical training artifacts.<n>We structure our framework by decomposing scenario generation into a hierarchy of subproblems, and for each one, defining the role of the AI tool.<n>Our results demonstrate the potential of LLM-driven multi-agent systems to generate coherent, nuanced documents.
  arXiv  Detail & Related papers  (2025-11-10T23:24:35Z)
• URDF-Anything: Constructing Articulated Objects with 3D Multimodal Language Model [76.08429266631823]
  We propose an end-to-end automatic reconstruction framework based on a 3D multimodal large language model (MLLM)<n>URDF-Anything utilizes an autoregressive prediction framework based on point-cloud and text multimodal input to jointly optimize geometric segmentation and kinematic parameter prediction.<n> Experiments on both simulated and real-world datasets demonstrate that our method significantly outperforms existing approaches.
  arXiv  Detail & Related papers  (2025-11-02T13:45:51Z)
• A Multi-Agent Framework for Automated Qinqiang Opera Script Generation Using Large Language Models [2.919625687404969]
  This paper introduces a novel multi-Agent framework that automates the end to end production of Qinqiang opera by integrating Large Language Models, visual generation, and Text to Speech synthesis.<n>In a case study on Dou E Yuan, the system achieved expert ratings of 3.8 for script fidelity, 3.5 for visual coherence, and 3.8 for speech accuracy-culminating in an overall score of 3.6, a 0.3 point improvement over a Single Agent baseline.
  arXiv  Detail & Related papers  (2025-04-22T03:14:29Z)
• GenMAC: Compositional Text-to-Video Generation with Multi-Agent Collaboration [20.988801611785522]
  We propose GenMAC, an iterative, multi-agent framework that enables compositional text-to-video generation.<n>The collaborative workflow includes three stages: Design, Generation, and Redesign.<n>To tackle diverse scenarios of compositional text-to-video generation, we design a self-routing mechanism to adaptively select the proper correction agent from a collection of correction agents each specialized for one scenario.
  arXiv  Detail & Related papers  (2024-12-05T18:56:05Z)
• InTraGen: Trajectory-controlled Video Generation for Object Interactions [100.79494904451246]
  InTraGen is a pipeline for improved trajectory-based generation of object interaction scenarios. Our results demonstrate improvements in both visual fidelity and quantitative performance.
  arXiv  Detail & Related papers  (2024-11-25T14:27:50Z)
• 3D-GPT: Procedural 3D Modeling with Large Language Models [47.72968643115063]
  We introduce 3D-GPT, a framework utilizing large language models(LLMs) for instruction-driven 3D modeling. 3D-GPT positions LLMs as proficient problem solvers, dissecting the procedural 3D modeling tasks into accessible segments and appointing the apt agent for each task. Our empirical investigations confirm that 3D-GPT not only interprets and executes instructions, delivering reliable results but also collaborates effectively with human designers.
  arXiv  Detail & Related papers  (2023-10-19T17:41:48Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.