Scaling Video Understanding via Compact Latent Multi-Agent Collaboration

• URL: http://arxiv.org/abs/2605.00444v1
• Date: Fri, 01 May 2026 06:24:40 GMT
• Title: Scaling Video Understanding via Compact Latent Multi-Agent Collaboration
• Abstract summary: We propose an end-to-end Multi-Agent Collaboration Framework that decouples per-agent perception budgets from global video complexity.<n>We show that MACF consistently outperforms state-of-the-art MLLMs and multi-agent systems under identical budget constraints.
• Score: 42.74404806675165
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Multi-modal large language models (MLLMs) advance vision language understanding but face inherent limitations in long-video tasks due to bounded perception context budgets. Existing agentic methods mitigate this via rule-based preprocessing, yet often suffer from information loss, high cost, and reliance on textual intermediates. We propose MACF, an end-to-end Multi-Agent Collaboration Framework that decouples per-agent perception budgets from global video complexity, enabling scalable video understanding while preserving visual fidelity. MACF partitions videos into segments for locally budgeted agents and enables holistic reasoning via an agent-native latent communication protocol. Each agent encodes partial observations into compact, task-sufficient tokens in a shared embedding space, allowing efficient and information-preserving collaboration by a central coordinator. We introduce a curriculum training strategy that progressively enforces semantic alignment, evidence summarization, and cross-agent coordination. Extensive experiments on diverse video understanding benchmarks show that MACF consistently outperforms state-of-the-art MLLMs and multi-agent systems under identical budget constraints, demonstrating the effectiveness of our latent collaboration for scalable video understanding.

Related papers

• InternVideo3: Agentify Foundation Models with Multimodal Contextual Reasoning [61.87294123934038]
  We present InternVideo3, a framework enhancing multimodal understanding in video tasks.<n>MCR treats understanding as a closed-loop process over a shared, evolving context.<n>We show InternVideo3 achieves strong performance on benchmarks like Video-MME, MLVU, and Ego.
  arXiv  Detail & Related papers  (2026-06-10T15:17:08Z)
• Beyond End-to-End Video Models: An LLM-Based Multi-Agent System for Educational Video Generation [15.004606775581356]
  LAVES is a hierarchical multi-agent system for generating high-quality instructional videos from educational problems.<n>In large-scale deployments, LAVES achieves a throughput exceeding one million videos per day, delivering over a 95% reduction in cost.
  arXiv  Detail & Related papers  (2026-02-12T10:14:36Z)
• Refer-Agent: A Collaborative Multi-Agent System with Reasoning and Reflection for Referring Video Object Segmentation [50.22481337087162]
  Referring Video Object (RVOS) aims to segment objects in videos based on textual queries.<n>Refer-Agent is a collaborative multi-agent system with alternating reasoning-reflection mechanisms.
  arXiv  Detail & Related papers  (2026-02-03T14:48:12Z)
• GCAgent: Long-Video Understanding via Schematic and Narrative Episodic Memory [59.869552603264076]
  We introduce GCAgent, a novel Global-Context-Aware Agent framework that achieves comprehensive long-video understanding.<n>Our core innovation is the Schematic and Narrative Episodic Memory, which structurally models events and their causal and temporal relations into a concise, organized context.<n>Experiments confirm that GCAgent significantly enhances long-video understanding, achieving up to 23.5% accuracy improvement on the Video-MME Long split over a strong MLLM baseline.
  arXiv  Detail & Related papers  (2025-11-15T04:29:00Z)
• FOCUS: Unified Vision-Language Modeling for Interactive Editing Driven by Referential Segmentation [55.01077993490845]
  Recent Large Vision Language Models (LVLMs) demonstrate promising capabilities in unifying visual understanding and generative modeling.<n>We introduce FOCUS, a unified LVLM that integrates segmentation-aware perception and controllable object-centric generation within an end-to-end framework.
  arXiv  Detail & Related papers  (2025-06-20T07:46:40Z)
• Layered Chain-of-Thought Prompting for Multi-Agent LLM Systems: A Comprehensive Approach to Explainable Large Language Models [0.0]
  We propose Layered Chain-of-Thought (Layered-CoT) Prompting, a novel framework that systematically segments the reasoning process into multiple layers.<n>We present three scenarios -- medical triage, financial risk assessment, and agile engineering -- and demonstrate how Layered-CoT surpasses vanilla CoT in terms of transparency, correctness, and user engagement.
  arXiv  Detail & Related papers  (2025-01-29T13:21:09Z)
• Collaborative Multi-Agent Video Fast-Forwarding [30.843484383185473]
  We develop two collaborative multi-agent video fast-forwarding frameworks in distributed and centralized settings. In these frameworks, each individual agent can selectively process or skip video frames at adjustable paces based on multiple strategies. We show that compared with other approaches in the literature, our frameworks achieve better coverage of important frames, while significantly reducing the number of frames processed at each agent.
  arXiv  Detail & Related papers  (2023-05-27T20:12:19Z)
• SOC: Semantic-Assisted Object Cluster for Referring Video Object Segmentation [35.063881868130075]
  This paper studies referring video object segmentation (RVOS) by boosting video-level visual-linguistic alignment. We propose Semantic-assisted Object Cluster (SOC), which aggregates video content and textual guidance for unified temporal modeling and cross-modal alignment. We conduct extensive experiments on popular RVOS benchmarks, and our method outperforms state-of-the-art competitors on all benchmarks by a remarkable margin.
  arXiv  Detail & Related papers  (2023-05-26T15:13:44Z)
• See More, Know More: Unsupervised Video Object Segmentation with Co-Attention Siamese Networks [184.4379622593225]
  We introduce a novel network, called CO-attention Siamese Network (COSNet), to address the unsupervised video object segmentation task. We emphasize the importance of inherent correlation among video frames and incorporate a global co-attention mechanism. We propose a unified and end-to-end trainable framework where different co-attention variants can be derived for mining the rich context within videos.
  arXiv  Detail & Related papers  (2020-01-19T11:10:39Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.