Beyond Frame Selection: Generative Latent Evidence Aggregation for Long-Video Understanding

• URL: http://arxiv.org/abs/2607.28516v1
• Date: Thu, 30 Jul 2026 16:55:00 GMT
• Title: Beyond Frame Selection: Generative Latent Evidence Aggregation for Long-Video Understanding
• Abstract summary: Long-video understanding commonly compresses videos into a small set of frames or visual tokens for answer generation.<n>Our key idea is to organize selected frames into query-relevant cross-frame evidence before generation.<n>Across four benchmarks and two Video-MLLM backbones, GenEvA consistently improves matched-frame baselines.
• Score: 17.716106078756066
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Long-video understanding commonly compresses videos into a small set of frames or visual tokens for answer generation. Existing compact pipelines focus on retaining relevant visual content as explicit evidence. Yet making evidence available does not ensure that complementary cues across moments are integrated for answering. Our key idea is to organize selected frames into query-relevant cross-frame evidence before generation. We formulate this post-selection stage as a latent evidence interface and instantiate it with GenEvA ($\textbf{Gen}erative$ $Latent$ $\textbf{Ev}idence$ $\textbf{A}ggregation$), a distribution-guided latent evidence aggregation framework. Specifically, GenEvA uses a query-conditioned evidence distribution to focus aggregation on relevant frames, forming compact cross-frame latent evidence from their frame-specific information. Since cross-frame integration is not always needed, the same distribution determines whether to insert this latent complement. Across four benchmarks and two Video-MLLM backbones, GenEvA consistently improves matched-frame baselines. At 8 frames, it raises the four-benchmark LLaVA-Video average by $+5.2$ points and Qwen2.5-VL accuracy on LVBench by $+10.1$ points. These gains require only $0.11\%$--$0.40\%$ average video-token overhead; analyses further show task-aware allocation and benefits from Adaptive Evidence Invocation.

Related papers

• RIDGE: Region-Informed Derivative-Guided Evidence Selection for Long Video Understanding [55.75126327155788]
  RIDGE is a frame selection framework that reads the frame-query similarity curve as a temporal signal.<n>It partitions the timeline into structural regions and applies region-specific selection to preserve event cores, transitions, buildup, aftermath, and contextual frames under a fixed budget.<n>It achieves the best performance in most settings and remains competitive in the others.
  arXiv  Detail & Related papers  (2026-08-30T18:40:51Z)
• Stable Curves, Unstable Items: Item-Level Scaling Heterogeneity in Video LLMs [16.31176289941024]
  We show that Video LLMs improve smoothly or saturate as visual budgets grow.<n>We represent each frozen model--item pair by its response trajectory under controlled visual budgets.<n>We derive matched-grid measures of configuration complementarity, harmful transitions, and text overwrite.
  arXiv  Detail & Related papers  (2026-08-07T09:25:20Z)
• Ground, Cover, and Refine: Evidence-Centric Frame Selection for Long-Video Question Answering [9.937860041147625]
  Long-video question answering requires identifying sparse yet critical evidence from videos under a constrained visual-token budget.<n>Existing methods either select query-aware frames in a single pass or rely on timestamped text solely as retrieval guidance.<n>We propose GCR, a training-free framework that casts fixed-budget frame selection as a joint evidence curation problem.
  arXiv  Detail & Related papers  (2026-08-03T03:50:22Z)
• Efficient Frame Selection for Long Videos at Test Time with Attention-Based MLLM Selectors [69.75895255395702]
  Cross-modal attention at validation-selected extraction layers in MLLMs already provides query-relevant frame evidence without requiring autoregressive generation.<n>A lightweight MLLM selector, even with only 2B parameters, can extract frame-level evidence by converting selected-layer attention into relevance scores through query-conditioned aggregation.<n>Under a 32-frame budget, our selector improves over uniform sampling by up to 6.4 points on Video-MME and outperforms prior training-based selectors under matched frame budgets.
  arXiv  Detail & Related papers  (2026-07-17T07:04:31Z)
• QCA: Query- and Content-Aware Keyframe Selection for Long Video Understanding [78.36929439152566]
  We propose a Query- and Content-Aware (QCA) selection framework that can select a compact yet information-rich set of frames from long videos.<n>QCA anchors on the most query-relevant frame and iteratively incorporates additional frames to maximize diversity while maintaining high semantic relevance to the query.<n>Our method requires no additional training and can be seamlessly integrated into existing Video-LLMs.
  arXiv  Detail & Related papers  (2026-07-01T14:19:24Z)
• GridProbe: Posterior-Probing for Adaptive Test-Time Compute in Long-Video VLMs [3.9266376632068485]
  GridProbe is an efficient training-free posterior-probing inference paradigm.<n>It scores evidence in answer space using a frozen VLM's own reasoning.<n>It selects question-relevant frames adaptively, resulting in sub-quadratic attention cost with little to no accuracy loss.
  arXiv  Detail & Related papers  (2026-05-11T15:57:46Z)
• Progressive Online Video Understanding with Evidence-Aligned Timing and Transparent Decisions [75.23170605943457]
  textbfmodel is a framework that decouples reasoning control from memory integration.<n>emphActive Thinking Decision Maker (ATDM) is a transparent reasoning controller that externalizes its decision process.<n>emphHierarchical Progressive Semantic Integration (HPSI) module acts as an efficient memory system.
  arXiv  Detail & Related papers  (2026-04-20T16:15:33Z)
• Adaptive Greedy Frame Selection for Long Video Understanding [11.923839324117674]
  Large vision--language models (VLMs) are increasingly applied to long-video question answering.<n>Inference is often bottlenecked by the number of input frames and resulting visual tokens.<n>We propose a question-adaptive greedy frame selection method that jointly optimize query relevance and semantic representativeness under a fixed frame budget.
  arXiv  Detail & Related papers  (2026-03-20T17:55:32Z)
• Keeping the Evidence Chain: Semantic Evidence Allocation for Training-Free Token Pruning in Video Temporal Grounding [16.537148896706036]
  Video Temporal Grounding (VTG) localizes the temporal boundaries of a query-relevant moment in long, untrimmed videos.<n>We propose SemVID, a training-free pruning framework that constructs a compact yet coherent token subset with semantic roles.
  arXiv  Detail & Related papers  (2026-03-05T20:25:32Z)
• Event-Anchored Frame Selection for Effective Long-Video Understanding [67.56884568828508]
  Event-Anchored Frame Selection (EFS) is a hierarchical, event-aware pipeline.<n>As a training-free, plug-and-play module, EFS can be seamlessly integrated into off-the-shelf LVLMs.
  arXiv  Detail & Related papers  (2026-03-01T08:25:37Z)
• HERBench: A Benchmark for Multi-Evidence Integration in Video Question Answering [13.370338205427911]
  We present HERBench, a VideoQA benchmark to assess multi-evidence integration across time.<n>HERBench comprises 26K five-way multiple-choice questions organized into twelve compositional tasks.<n>We show that HERBench imposes substantially higher demand than prior datasets.
  arXiv  Detail & Related papers  (2025-12-16T19:34:47Z)
• Single-Stage Visual Query Localization in Egocentric Videos [79.71065005161566]
  We propose a single-stage VQL framework that is end-to-end trainable. We establish the query-video relationship by considering query-to-frame correspondences between the query and each video frame. Our experiments demonstrate that our approach outperforms prior VQL methods by 20% accuracy while obtaining a 10x improvement in inference speed.
  arXiv  Detail & Related papers  (2023-06-15T17:57:28Z)
• Correspondence Matters for Video Referring Expression Comprehension [64.60046797561455]
  Video Referring Expression (REC) aims to localize the referent objects described in the sentence to visual regions in the video frames. Existing methods suffer from two problems: 1) inconsistent localization results across video frames; 2) confusion between the referent and contextual objects. We propose a novel Dual Correspondence Network (dubbed as DCNet) which explicitly enhances the dense associations in both the inter-frame and cross-modal manners.
  arXiv  Detail & Related papers  (2022-07-21T10:31:39Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.