CoFiE: Coarse-to-Fine Evidence Selection for Efficient Streaming Video Understanding

• URL: http://arxiv.org/abs/2609.03675v1
• Date: Thu, 03 Sep 2026 11:14:21 GMT
• Title: CoFiE: Coarse-to-Fine Evidence Selection for Efficient Streaming Video Understanding
• Abstract summary: Streaming video understanding requires Vision Language Models (VLLMs) to process growing video streams and answer user questions under tight latency constraints.<n>We propose CoFiE, a Coarse-to-Fine Evidence Selection framework that decouples evidence selection into a coarse, query-agnostic filtering stage before the vision encoder and a fine, query-specific refinement stage during LLM prefill.<n>CoFiE establishes a new state-of-the-art accuracy-efficiency trade-off across multiple video understanding benchmarks, reaching 78.86% accuracy on StreamingBench and 68.72% on OvO-Bench, with improvements of up to 3
• Score: 9.22946015931432
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Streaming video understanding requires Vision Language Models (VLLMs) to process growing video streams and answer user questions under tight latency constraints. Existing methods improve efficiency through token pruning and memory-bank schemes, but mainly reduce visual tokens after visual encoding. Consequently, downstream token pruning alone cannot substantially reduce end-to-end latency because the expensive frame encoding cost has already been incurred. We propose CoFiE, a Coarse-to-Fine Evidence Selection framework that decouples evidence selection into a coarse, query-agnostic filtering stage before the vision encoder and a fine, query-specific refinement stage during LLM prefill. CoFiE introduces Novelty-Guided Frame Filtering to retain visually distinctive candidate frames and Query-Specific Evidence Refinement to select the frames most relevant to the user query. This design removes substantial redundancy before frame encoding while preserving query-specific refinement once semantic information becomes available. Experiments show that CoFiE establishes a new state-of-the-art accuracy-efficiency trade-off across multiple video understanding benchmarks, reaching 78.86% accuracy on StreamingBench and 68.72% on OvO-Bench, with improvements of up to 3.15% over prior methods. Even with up to 80% evidence-frame filtering, CoFiE outperforms strong open-source multimodal models while improving end-to-end inference latency by up to 2.54 times.

Related papers

• When and Where to Look: Adaptive Visual Evidence Scheduling for Efficient Long Video Understanding [17.062781672214104]
  We propose EcoFrame, a framework for low-overhead query-adaptive visual evidence scheduling.<n>Experiments on Video-MME, LongVideoBench, and MLVU demonstrate that EcoFrame achieves a better accuracy--efficiency trade-off across multiple VLM backbones.
  arXiv  Detail & Related papers  (2026-08-04T16:49:53Z)
• LiteFrame: Efficient Vision Encoders Unlock Frame Scaling in Video LLMs [90.77662862634509]
  LiteFrame is a strong, yet highly efficient backbone for Video Large Language Models.<n>LiteFrame provides a 35% reduction in end-to-end latency while processing 8$times$ more frames.<n>Our results demonstrate a potential path to unlocking longer-form video understanding under fixed compute budgets.
  arXiv  Detail & Related papers  (2026-05-17T05:02:52Z)
• AdaFocus: Adaptive Relevance-Diversity Sampling with Zero-Cache Look-back for Efficient Long Video Understanding [4.238932162084854]
  Long video understanding is heavily bottlenecked by a rigid one-shot paradigm.<n>We propose AdaFocus, an efficient framework that rethinks long-video understanding as progressive evidence acquisition rather than one-pass encoding.
  arXiv  Detail & Related papers  (2026-05-13T03:40:21Z)
• Progressive Online Video Understanding with Evidence-Aligned Timing and Transparent Decisions [75.23170605943457]
  textbfmodel is a framework that decouples reasoning control from memory integration.<n>emphActive Thinking Decision Maker (ATDM) is a transparent reasoning controller that externalizes its decision process.<n>emphHierarchical Progressive Semantic Integration (HPSI) module acts as an efficient memory system.
  arXiv  Detail & Related papers  (2026-04-20T16:15:33Z)
• Query-Conditioned Evidential Keyframe Sampling for MLLM-Based Long-Form Video Understanding [21.306038832082553]
  Multimodal Large Language Models (MLLMs) have shown strong performance on video question answering.<n>Their application to long-form videos is constrained by limited context length and computational cost.<n>We propose an evidence-driven sampling framework grounded in information theory.
  arXiv  Detail & Related papers  (2026-04-01T15:02:50Z)
• GIFT: Global Irreplaceability Frame Targeting for Efficient Video Understanding [63.62445065977448]
  GIFT: Global Irreplaceability Frame Targeting is a training-free framework that selects frames by assessing their intrinsic irreplaceability.<n>We show that GIFT achieves a maximum average improvement of 12.5% across long-form video benchmarks on LLaVA-Video-7B compared to uniform sampling.
  arXiv  Detail & Related papers  (2026-03-26T06:21:41Z)
• Event-Anchored Frame Selection for Effective Long-Video Understanding [67.56884568828508]
  Event-Anchored Frame Selection (EFS) is a hierarchical, event-aware pipeline.<n>As a training-free, plug-and-play module, EFS can be seamlessly integrated into off-the-shelf LVLMs.
  arXiv  Detail & Related papers  (2026-03-01T08:25:37Z)
• Going Down Memory Lane: Scaling Tokens for Video Stream Understanding with Dynamic KV-Cache Memory [50.30283773196725]
  Existing approaches rely on key-value caching to accumulate frame-level details over time, but use a limited number of tokens per frame.<n>We propose scaling the token budget to enable more granular-temporal understanding and reasoning.
  arXiv  Detail & Related papers  (2026-02-20T18:59:50Z)
• Less is More: Token-Efficient Video-QA via Adaptive Frame-Pruning and Semantic Graph Integration [24.337139909108117]
  "Less is more" phenomenon where excessive frames can paradoxically degrade performance due to context dilution.<n>"Visual echoes" yield significant temporal redundancy, which we term 'visual echoes'<n>"AFP" employs an adaptive hierarchical clustering algorithm on a fused ResNet-50 and CLIP feature space to identify and merge these echoes into single representatives.<n>Our full approach demonstrates a drastic reduction in required frames by up to 86.9% and total input tokens by up to 83.2%.
  arXiv  Detail & Related papers  (2025-08-05T11:31:55Z)
• QuoTA: Query-oriented Token Assignment via CoT Query Decouple for Long Video Comprehension [86.0749609778104]
  We propose QuoTA, an ante-hoc training-free modular that extends existing large video-language models.<n>QuoTA strategically allocates frame-level importance scores based on query relevance.<n>We decouple the query through Chain-of-Thoughts reasoning to facilitate more precise LVLM-based frame importance scoring.
  arXiv  Detail & Related papers  (2025-03-11T17:59:57Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.