QCA: Query- and Content-Aware Keyframe Selection for Long Video Understanding
Abstract Overview
This paper introduces QCA, a training-free keyframe selection framework for long video understanding under limited frame budgets. The method first partitions a video into temporal segments, scores each segment by combining query relevance with content deviation, and allocates the frame budget adaptively across segments. Within each segment, it starts from the most query-relevant frame and then greedily adds frames that preserve semantic relevance while increasing diversity. The framework is designed to plug into existing Video-LLMs without architectural changes, aiming to reduce temporal redundancy while retaining information needed for downstream reasoning.
Novelty
The distinctive aspect of QCA is its joint treatment of query relevance, segment-level content deviation, adaptive inter-segment budget allocation, and diversity-aware intra-segment frame selection within a single training-free framework. Unlike approaches that rely only on uniform sampling or top relevance scores, it explicitly balances semantic alignment with visual diversity and temporal coverage.
Results
Across LongVideoBench, Video-MME, MLVU, and LVBench, QCA consistently improves over uniform sampling and several prior keyframe selection baselines on multiple backbones including LLaVA-Video-7B, InternVL-3.5-8B, and Qwen3-VL-8B. With Qwen3-VL-8B and 64 frames, it reports 66.9 on LongVideoBench, 69.5 on Video-MME, 75.7 on MLVU, and 51.8 on LVBench, and the paper also reports 67.8 on LongVideoBench using 128 frames. Ablations further indicate that semantic matching, content deviation, anchor-based selection, candidate filtering, and diversity modeling each contribute to performance.
Key Points
- QCA adaptively allocates a limited keyframe budget across temporal segments using both query-frame semantic matching and segment-level content deviation.
- Within each segment, the method anchors on the most query-relevant frame and adds additional frames from a relevance-filtered candidate set to maximize diversity.
- The approach is training-free, integrates with existing Video-LLMs, and shows consistent gains over uniform sampling and other keyframe selection baselines across several long-video benchmarks.