DIRECT: Video Mashup Creation via Hierarchical Multi-Agent Planning and Intent-Guided Editing

• URL: http://arxiv.org/abs/2604.04875v1
• Date: Mon, 06 Apr 2026 17:26:04 GMT
• Title: DIRECT: Video Mashup Creation via Hierarchical Multi-Agent Planning and Intent-Guided Editing
• Abstract summary: Video mashup creation represents a complex video editing paradigm that recomposes existing footage to craft engaging audio-visual experiences.<n>We formulate video mashup creation as a Multimodal Coherency Satisfaction Problem (MMCSP) and propose the DIRECT framework.<n>Our hierarchical multi-agent framework decomposes the challenge into three cascade levels: the Screenwriter for source-aware global structural anchoring, the Director for instantiating adaptive editing intent and guidance, and the Editor for intent-guided shot sequence editing with fine-grained optimization.
• Score: 17.53577850169204
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Video mashup creation represents a complex video editing paradigm that recomposes existing footage to craft engaging audio-visual experiences, demanding intricate orchestration across semantic, visual, and auditory dimensions and multiple levels. However, existing automated editing frameworks often overlook the cross-level multimodal orchestration to achieve professional-grade fluidity, resulting in disjointed sequences with abrupt visual transitions and musical misalignment. To address this, we formulate video mashup creation as a Multimodal Coherency Satisfaction Problem (MMCSP) and propose the DIRECT framework. Simulating a professional production pipeline, our hierarchical multi-agent framework decomposes the challenge into three cascade levels: the Screenwriter for source-aware global structural anchoring, the Director for instantiating adaptive editing intent and guidance, and the Editor for intent-guided shot sequence editing with fine-grained optimization. We further introduce Mashup-Bench, a comprehensive benchmark with tailored metrics for visual continuity and auditory alignment. Extensive experiments demonstrate that DIRECT significantly outperforms state-of-the-art baselines in both objective metrics and human subjective evaluation. Project page and code: https://github.com/AK-DREAM/DIRECT

Related papers

• Thinking on Shots: Consistent Multi-Shot Video Editing with Agentic Reasoning [56.17582599802973]
  We introduce the Multi-Instruction Multi-Shot Long-Video Editing (MMLVE) task, which is structured around three core objectives.<n>To tackle these three unique challenges, we introduce an agentic editing framework that leverages the Large Language Models (LLMs) and Vision-Language Models (VLMs)<n>Extensive experiments demonstrate that our MMLVE-Agent outperforms existing closed-source SOTA approaches.
  arXiv  Detail & Related papers  (2026-08-27T08:48:13Z)
• MAVIN: Multi-Shot Audio-Visual Generation with Narrative Control [66.04301887685004]
  Existing methods suffer from temporal misalignment, limited controllability, and incomplete scripting.<n>We propose MAVIN, the first framework for multi-shot audio-visual generation with customized narrative control.<n> MAVIN achieves state-of-the-art performance, opening up a new avenue for integrating generative models into professional filmmaking.
  arXiv  Detail & Related papers  (2026-06-28T16:01:04Z)
• TIDE: Task-Isolated Diffusion for Unified Video Editing and Generation [30.09687573502619]
  Diffusion Transformers have driven rapid progress in video generation and editing, yet these capabilities are still handled by task-specific models.<n>We propose TIDE, a unified framework that integrates instruction-based editing, reference-guided editing, and multi-reference generation.
  arXiv  Detail & Related papers  (2026-06-06T17:11:13Z)
• A Benchmark and Multi-Agent System for Instruction-driven Cinematic Video Compilation [75.66313073019326]
  We introduce CineBench, the first benchmark for instruction-driven cinematic video compilation.<n>We present CineAgents, a multi-agent system that reformulates cinematic video compilation into design-and-compose'' paradigm.
  arXiv  Detail & Related papers  (2026-04-12T04:39:09Z)
• CutClaw: Agentic Hours-Long Video Editing via Music Synchronization [96.62825277039117]
  CutClaw is an autonomous multi-agent framework designed to edit hours-long raw footage into meaningful short videos.<n>It produces videos with synchronized music, followed by instructions, and a visually appealing appearance.<n>We conduct detailed experiments to demonstrate that CutClaw significantly outperforms state-of-the-art baselines in generating high-quality, rhythm-aligned videos.
  arXiv  Detail & Related papers  (2026-03-31T12:25:53Z)
• Beyond End-to-End Video Models: An LLM-Based Multi-Agent System for Educational Video Generation [15.004606775581356]
  LAVES is a hierarchical multi-agent system for generating high-quality instructional videos from educational problems.<n>In large-scale deployments, LAVES achieves a throughput exceeding one million videos per day, delivering over a 95% reduction in cost.
  arXiv  Detail & Related papers  (2026-02-12T10:14:36Z)
• IMAGAgent: Orchestrating Multi-Turn Image Editing via Constraint-Aware Planning and Reflection [40.21337735524356]
  IMAGAgent is a multi-turn image editing agent framework based on a "plan-execute-reflect" closed-loop mechanism.<n>It achieves deep synergy among instruction parsing, tool scheduling, and adaptive correction within a unified pipeline.<n>Experiments on our constructed textbfMTEditBench and the MagicBrush dataset demonstrate that IMAGAgent achieves performance significantly superior to existing methods.
  arXiv  Detail & Related papers  (2026-02-12T02:37:38Z)
• Tele-Omni: a Unified Multimodal Framework for Video Generation and Editing [93.8111348452324]
  Tele- Omni is a unified framework for video generation and editing that follows multimodal instructions.<n>It supports text-to-video generation, image-to-video generation, first-last-frame video generation, in-context video generation, and in-context video editing.
  arXiv  Detail & Related papers  (2026-02-10T10:01:16Z)
• MAViS: A Multi-Agent Framework for Long-Sequence Video Storytelling [24.22367257991941]
  MAViS is a multi-agent collaborative framework designed to assist in long-sequence video storytelling.<n>It orchestrates specialized agents across multiple stages, including script writing, shot designing, character modeling, generation, video animation, and audio generation.<n>With just a brief idea description, MAViS enables users to rapidly explore diverse visual storytelling and creative directions for sequential video generation by efficiently producing high-quality, complete long-sequence videos.
  arXiv  Detail & Related papers  (2025-08-11T21:42:41Z)
• FOCUS: Unified Vision-Language Modeling for Interactive Editing Driven by Referential Segmentation [55.01077993490845]
  Recent Large Vision Language Models (LVLMs) demonstrate promising capabilities in unifying visual understanding and generative modeling.<n>We introduce FOCUS, a unified LVLM that integrates segmentation-aware perception and controllable object-centric generation within an end-to-end framework.
  arXiv  Detail & Related papers  (2025-06-20T07:46:40Z)
• Image Editing As Programs with Diffusion Models [69.05164729625052]
  We introduce Image Editing As Programs (IEAP), a unified image editing framework built upon the Diffusion Transformer (DiT) architecture.<n>IEAP approaches instructional editing through a reductionist lens, decomposing complex editing instructions into sequences of atomic operations.<n>Our framework delivers superior accuracy and semantic fidelity, particularly for complex, multi-step instructions.
  arXiv  Detail & Related papers  (2025-06-04T16:57:24Z)
• MAKIMA: Tuning-free Multi-Attribute Open-domain Video Editing via Mask-Guided Attention Modulation [55.101611012677616]
  Diffusion-based text-to-image (T2I) models have demonstrated remarkable results in global video editing tasks.<n>We present MAKIMA, a tuning-free MAE framework built upon pretrained T2I models for open-domain video editing.
  arXiv  Detail & Related papers  (2024-12-28T02:36:51Z)
• GenMAC: Compositional Text-to-Video Generation with Multi-Agent Collaboration [20.988801611785522]
  We propose GenMAC, an iterative, multi-agent framework that enables compositional text-to-video generation.<n>The collaborative workflow includes three stages: Design, Generation, and Redesign.<n>To tackle diverse scenarios of compositional text-to-video generation, we design a self-routing mechanism to adaptively select the proper correction agent from a collection of correction agents each specialized for one scenario.
  arXiv  Detail & Related papers  (2024-12-05T18:56:05Z)
• Dynamic Graph Representation Learning for Video Dialog via Multi-Modal Shuffled Transformers [89.00926092864368]
  We present a semantics-controlled multi-modal shuffled Transformer reasoning framework for the audio-visual scene aware dialog task. We also present a novel dynamic scene graph representation learning pipeline that consists of an intra-frame reasoning layer producing-semantic graph representations for every frame. Our results demonstrate state-of-the-art performances on all evaluation metrics.
  arXiv  Detail & Related papers  (2020-07-08T02:00:22Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.