MR. Video: "MapReduce" is the Principle for Long Video Understanding

• URL: http://arxiv.org/abs/2504.16082v1
• Date: Tue, 22 Apr 2025 17:59:41 GMT
• Title: MR. Video: "MapReduce" is the Principle for Long Video Understanding
• Authors: Ziqi Pang, Yu-Xiong Wang,
• Abstract summary: MR. Video is an agentic long video understanding framework.<n>It performs detailed short video perception without being limited by context length.<n>It achieves over 10% accuracy improvement on the challenging LVBench.
• Score: 27.9561679446938
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We propose MR. Video, an agentic long video understanding framework that demonstrates the simple yet effective MapReduce principle for processing long videos: (1) Map: independently and densely perceiving short video clips, and (2) Reduce: jointly aggregating information from all clips. Compared with sequence-to-sequence vision-language models (VLMs), MR. Video performs detailed short video perception without being limited by context length. Compared with existing video agents that typically rely on sequential key segment selection, the Map operation enables simpler and more scalable sequence parallel perception of short video segments. Its Reduce step allows for more comprehensive context aggregation and reasoning, surpassing explicit key segment retrieval. This MapReduce principle is applicable to both VLMs and video agents, and we use LLM agents to validate its effectiveness. In practice, MR. Video employs two MapReduce stages: (A) Captioning: generating captions for short video clips (map), then standardizing repeated characters and objects into shared names (reduce); (B) Analysis: for each user question, analyzing relevant information from individual short videos (map), and integrating them into a final answer (reduce). MR. Video achieves over 10% accuracy improvement on the challenging LVBench compared to state-of-the-art VLMs and video agents. Code is available at: https://github.com/ziqipang/MR-Video

Related papers

• ViSMaP: Unsupervised Hour-long Video Summarisation by Meta-Prompting [29.049727807251084]
  ViSMap is a system to summarise hour long videos with no-supervision.<n>We bridge the gap between short videos (where annotated data is plentiful) and long ones (where it's not)
  arXiv  Detail & Related papers  (2025-04-22T14:06:01Z)
• CaReBench: A Fine-Grained Benchmark for Video Captioning and Retrieval [24.203328970223527]
  We present CaReBench, a testing benchmark for fine-grained video captioning and retrieval. Uniquely, it provides manually separated spatial annotations and temporal annotations for each video. Based on this design, we introduce two evaluation metrics, ReBias and CapST, specifically tailored for video retrieval and video captioning tasks.
  arXiv  Detail & Related papers  (2024-12-31T15:53:50Z)
• ReTaKe: Reducing Temporal and Knowledge Redundancy for Long Video Understanding [55.320254859515714]
  ReTaKe enables VideoLLMs to process 8 times longer frames (up to 2048), similar-sized models by 3-5% and even rivaling much larger ones on VideoMME, MLVU, LongVideoBench, and LVBench.<n>Our code is available at https://github.com/SCZwangxiao/video-ReTaKe.
  arXiv  Detail & Related papers  (2024-12-29T15:42:24Z)
• AuroraCap: Efficient, Performant Video Detailed Captioning and a New Benchmark [89.73538448786405]
  We propose AuroraCap, a video captioner based on a large multimodal model. We implement the token merging strategy, reducing the number of input visual tokens. AuroraCap shows superior performance on various video and image captioning benchmarks.
  arXiv  Detail & Related papers  (2024-10-04T00:13:54Z)
• VideoGPT+: Integrating Image and Video Encoders for Enhanced Video Understanding [15.959757105308238]
  Video LMMs rely on either image or video encoders to process visual inputs, each of which has its own limitations. We introduce VideoGPT+, which combines the complementary benefits of the image encoder (for detailed spatial understanding) and the video encoder (for global temporal context modeling) Our architecture showcases improved performance across multiple video benchmarks, including VCGBench, MVBench and Zero-shot question-answering.
  arXiv  Detail & Related papers  (2024-06-13T17:59:59Z)
• LongVLM: Efficient Long Video Understanding via Large Language Models [55.813206751150716]
  LongVLM is a simple yet powerful VideoLLM for long video understanding. We encode video representations that incorporate both local and global information. Our model produces more precise responses for long video understanding.
  arXiv  Detail & Related papers  (2024-04-04T11:33:29Z)
• HierVL: Learning Hierarchical Video-Language Embeddings [108.77600799637172]
  HierVL is a novel hierarchical video-language embedding that simultaneously accounts for both long-term and short-term associations. We introduce a hierarchical contrastive training objective that encourages text-visual alignment at both the clip level and video level. Our hierarchical scheme yields a clip representation that outperforms its single-level counterpart as well as a long-term video representation that achieves SotA.
  arXiv  Detail & Related papers  (2023-01-05T21:53:19Z)
• TL;DW? Summarizing Instructional Videos with Task Relevance & Cross-Modal Saliency [133.75876535332003]
  We focus on summarizing instructional videos, an under-explored area of video summarization. Existing video summarization datasets rely on manual frame-level annotations. We propose an instructional video summarization network that combines a context-aware temporal video encoder and a segment scoring transformer.
  arXiv  Detail & Related papers  (2022-08-14T04:07:40Z)
• Less is More: ClipBERT for Video-and-Language Learning via Sparse Sampling [98.41300980759577]
  A canonical approach to video-and-language learning dictates a neural model to learn from offline-extracted dense video features. We propose a generic framework ClipBERT that enables affordable end-to-end learning for video-and-language tasks. Experiments on text-to-video retrieval and video question answering on six datasets demonstrate that ClipBERT outperforms existing methods.
  arXiv  Detail & Related papers  (2021-02-11T18:50:16Z)
• Localizing the Common Action Among a Few Videos [51.09824165433561]
  This paper strives to localize the temporal extent of an action in a long untrimmed video. We introduce a new 3D convolutional network architecture able to align representations from the support videos with the relevant query video segments.
  arXiv  Detail & Related papers  (2020-08-13T11:31:23Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.