VideoAgent: All-in-One Framework for Video Understanding and Editing

• URL: http://arxiv.org/abs/2606.23327v1
• Date: Mon, 22 Jun 2026 13:37:08 GMT
• Title: VideoAgent: All-in-One Framework for Video Understanding and Editing
• Abstract summary: VideoAgent is an all-in-one agentic framework for video editing.<n>We develop automated video shot creation with shot planning agents for coherent narratives.<n>Intent parsing filters relevant tools while textual-gradient graph optimizations assemble editing pipelines.
• Score: 26.019384130449733
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Video editing has become essential in digital media creation, yet existing automated systems are restricted to short segment processing and domain-specific tasks. They face two critical limitations: i) inability to handle diverse video comprehension and editing operations, and ii) lack of long-video understanding for coherent narrative creation. We propose VideoAgent, an all-in-one agentic framework addressing these challenges through two key innovations. First, we develop automated video shot creation with shot planning agents for coherent narratives and cross-modal retrieval for aligned visual content. Second, we design a multi-agent orchestration framework integrating over thirty specialized editing agents. Intent parsing filters relevant tools while textual-gradient graph optimization assembles complex editing pipelines. Extensive experiments on our newly-proposed VideoEdit benchmark and public datasets demonstrate VideoAgent's superiority over existing multimodal LLMs and agentic systems. VideoAgent achieves 87-95% orchestration success rates while reducing API costs by 60%. Human evaluation across six video categories shows VideoAgent produces professional-quality content approaching human-level performance, with ratings only 4% below human-created videos. We release our code at https://github.com/HKUDS/VideoAgent.

Related papers

• VideoGAIA: A Benchmark for General AI Assistants on Agentic Video Understanding [73.10050069618302]
  VideoGAIA is an agentic video understanding benchmark for general artificial intelligence assistants.<n>VideoGAIA contains 271 model-human co-designed tasks covering diverse and complex real-world scenarios.<n>All evaluated MLLMs, including frontier models such as GPT-5.5 and Kimi-K3, achieve less than 60% accuracy on VideoGAIA.
  arXiv  Detail & Related papers  (2026-08-12T03:24:44Z)
• Beyond End-to-End Video Models: An LLM-Based Multi-Agent System for Educational Video Generation [15.004606775581356]
  LAVES is a hierarchical multi-agent system for generating high-quality instructional videos from educational problems.<n>In large-scale deployments, LAVES achieves a throughput exceeding one million videos per day, delivering over a 95% reduction in cost.
  arXiv  Detail & Related papers  (2026-02-12T10:14:36Z)
• A Versatile Multimodal Agent for Multimedia Content Generation [66.86040734610073]
  We propose a MultiMedia-Agent designed to automate complex content creation tasks.<n>Our agent system includes a data generation pipeline, a tool library for content creation, and a set of metrics for evaluating preference alignment.
  arXiv  Detail & Related papers  (2026-01-06T18:49:47Z)
• LongVideoAgent: Multi-Agent Reasoning with Long Videos [69.28914905197426]
  We propose a multi-agent framework in which a master LLM coordinates a grounding agent to localize question-relevant segments and a vision agent to extract targeted textual observations.<n>The master agent plans with a step limit, and is trained with reinforcement learning to encourage concise, correct, and efficient multi-agent cooperation.<n>On our proposed LongTVQA and LongTVQA+ which are episode-level datasets aggregated from TVQA/TVQA+, our multi-agent system significantly outperforms strong non-agent baselines.
  arXiv  Detail & Related papers  (2025-12-23T18:59:49Z)
• UniVA: Universal Video Agent towards Open-Source Next-Generation Video Generalist [107.04196084992907]
  We introduce UniVA, an omni-capable multi-agent framework for next-generation video generalists.<n>UniVA employs a Plan-and-Act dual-agent architecture that drives a highly automated and proactive workflow.<n>We also introduce UniVA-Bench, a benchmark suite of multi-step video tasks spanning understanding, editing, segmentation, and generation.
  arXiv  Detail & Related papers  (2025-11-11T17:58:13Z)
• Deep Video Discovery: Agentic Search with Tool Use for Long-form Video Understanding [60.88843818016968]
  Long-form video understanding presents significant challenges due to temporal-spatial complexity and difficulty of question answering.<n>We propose the Deep Video Discovery (DVD) agent to leverage an agentic search strategy over segmented video clips.<n>Our DVD agent achieves state-of-the-art performance on the challenging LVBench dataset, reaching an accuracy of 74.2%.
  arXiv  Detail & Related papers  (2025-05-23T16:37:36Z)
• VideoMultiAgents: A Multi-Agent Framework for Video Question Answering [11.514596823413736]
  Video Question Answering (VQA) inherently relies on multimodal reasoning.<n>We introduce VideoMultiAgents, a framework that integrates specialized agents for vision, scene graph analysis, and text processing.<n>Our approach is also supplemented with a question-guided caption generation, which produces captions that highlight objects, actions, and temporal transitions.
  arXiv  Detail & Related papers  (2025-04-25T22:08:09Z)
• StoryAgent: Customized Storytelling Video Generation via Multi-Agent Collaboration [88.94832383850533]
  We propose a multi-agent framework designed for Customized Storytelling Video Generation (CSVG) StoryAgent decomposes CSVG into distinct subtasks assigned to specialized agents, mirroring the professional production process. Specifically, we introduce a customized Image-to-Video (I2V) method, LoRA-BE, to enhance intra-shot temporal consistency. Our contributions include the introduction of StoryAgent, a versatile framework for video generation tasks, and novel techniques for preserving protagonist consistency.
  arXiv  Detail & Related papers  (2024-11-07T18:00:33Z)
• OmAgent: A Multi-modal Agent Framework for Complex Video Understanding with Task Divide-and-Conquer [14.503628667535425]
  processing extensive videos presents significant challenges due to the vast data and processing demands. We develop OmAgent, efficiently stores and retrieves relevant video frames for specific queries. It features an Divide-and-Conquer Loop capable of autonomous reasoning. We have endowed it with greater autonomy and a robust tool-calling system, enabling it to accomplish even more intricate tasks.
  arXiv  Detail & Related papers  (2024-06-24T13:05:39Z)
• Reframe Anything: LLM Agent for Open World Video Reframing [0.8424099022563256]
  We introduce Reframe Any Video Agent (RAVA), an AI-based agent that restructures visual content for video reframing. RAVA operates in three stages: perception, where it interprets user instructions and video content; planning, where it determines aspect ratios and reframing strategies; and execution, where it invokes the editing tools to produce the final video. Our experiments validate the effectiveness of RAVA in video salient object detection and real-world reframing tasks, demonstrating its potential as a tool for AI-powered video editing.
  arXiv  Detail & Related papers  (2024-03-10T03:29:56Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.