Rethinking RAG in Long Videos: What to Retrieve and How to Use It?

• URL: http://arxiv.org/abs/2606.13141v1
• Date: Thu, 11 Jun 2026 10:05:49 GMT
• Title: Rethinking RAG in Long Videos: What to Retrieve and How to Use It?
• Abstract summary: We introduce V-RAGBench, a benchmark of $langle$query, evidence chunk, answer$rangle$ triplets that enables faithful, decoupled evaluation of retrieval and generation.<n>We also present CARVE, a method that runs parallel retrievers across configurations and employs chunk-adaptive reranking to identify the winning configuration for each chunk.
• Score: 56.38819694781005
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Retrieval-augmented generation is moving beyond text into long, egocentric video, where systems must select query-relevant chunks across multiple modalities and temporal granularities. Yet progress in VideoRAG is limited by two gaps: existing benchmarks allow queries to be answered without the video, obscuring retrieval errors, and prior methods apply a single modality-granularity configuration per query, ignoring chunk-level variability. We address both by introducing V-RAGBench, a benchmark of $\langle$query, evidence chunk, answer$\rangle$ triplets that enables faithful, decoupled evaluation of retrieval and generation, and CARVE, a simple method that runs parallel retrievers across configurations and employs chunk-adaptive reranking to identify the winning configuration for each chunk. Each chunk then enters the generator under its winning configuration selected during retrieval, yielding an interleaved evidence form where the chunk-level decision propagates across both stages. CARVE outperforms eight recent VideoRAG baselines, with the chunks supplied to the generator interleaving multiple configurations rather than sharing a single one, a behavior unattainable by query-level methods.

Related papers

• ORDER: Task-Conditioned Routing for Retrieval-Augmented Generation [3.6028767235872965]
  In this paper, we introduce a query-conditioned RAG framework that jointly adapts indexing and retrieval to the incoming query.<n>We evaluate our framework on large-scale, heterogeneous historical archives and show that conditioning both indexing and retrieval on the query consistently outperforms both naive baselines and strong state-of-the-art RAG systems.
  arXiv  Detail & Related papers  (2026-09-15T11:23:16Z)
• Not All Inputs Are Valid: Towards Open-Set Video Moment Retrieval Using Language [82.18047702848021]
  Video Moment Retrieval (VMR) targets to retrieve the specific moment corresponding to a sentence query from an untrimmed video.<n>To this end, we creatively explore a brand-new VMR setting termed Open-Set Video Moment Retrieval (OS-VMR)<n>In this paper, we make the first attempt to step toward OS-VMR and propose a novel model textOpenVMR, which first distinguishes ID and OOD queries based on the normalizing flow technology.
  arXiv  Detail & Related papers  (2026-05-28T11:57:14Z)
• Fewer Steps, Better Performance: Efficient Cross-Modal Clip Trimming for Video Moment Retrieval Using Language [60.91064560080974]
  We propose an efficient approach, SpotVMR, to trim the query-relevant clip.<n>Our proposed SpotVMR can serve as plug-and-play module, which achieves efficiency for state-of-the-art VMR methods.
  arXiv  Detail & Related papers  (2026-05-28T11:42:28Z)
• DynFrame: Adaptive Reasoning-Driven Multimodal Framework with Dynamic Frame Augmentation for Complex Video Understanding [19.492762364136393]
  We present DynFrame, a framework that emits the temporal window and the sampling density as native tokens within a single autoregressive pass.<n>This learnable span-density retrieval enables acquiring multi-granularity evidence with a single retrieval step.
  arXiv  Detail & Related papers  (2026-05-26T08:16:16Z)
• R$^3$AG: Retriever Routing for Retrieval-Augmented Generation [59.47703698994575]
  R$3$AG decomposes retriever capability into two learnable dimensions: retrieval quality and generation utility.<n>R$3$AG consistently outperforms both the best individual retrievers and state-of-the-art static routing methods.
  arXiv  Detail & Related papers  (2026-04-22T06:51:20Z)
• Divide, then Ground: Adapting Frame Selection to Query Types for Long-Form Video Understanding [21.18266593437182]
  We propose a training-free frame selection framework that adapts its strategy based on the query type.<n> Experiments on three long-form video understanding benchmarks demonstrate that DIG consistently outperforms existing baselines.
  arXiv  Detail & Related papers  (2025-12-03T17:36:06Z)
• Towards Hyper-Efficient RAG Systems in VecDBs: Distributed Parallel Multi-Resolution Vector Search [5.216774377033164]
  We propose textbfSemantic Pyramid Indexing (SPI), a novel multi-resolution vector indexing framework for RAG in VecDBs.<n>Unlike existing hierarchical methods that require offline tuning or separate model training, SPI constructs a semantic pyramid over document embeddings and dynamically selects the optimal resolution level per query.<n>We implement SPI as a plugin for both FAISS and Qdrant backends and evaluate it across multiple RAG tasks including MS MARCO, Natural Questions, and multimodal retrieval benchmarks.
  arXiv  Detail & Related papers  (2025-11-12T09:31:08Z)
• LOVO: Efficient Complex Object Query in Large-Scale Video Datasets [11.821229903544404]
  LOVO is a novel system designed to efficiently handle comp$underlineL$ex $underlineO$bject queries in large-scale $underlineV$ide$underlineO$ datasets.<n>Agnostic to user queries, LOVO performs one-time feature extraction using pre-trained visual encoders, generating compact visual embeddings for key frames.<n>During the query phase, LOVO transforms object queries to query embeddings and conducts fast approximate nearest-neighbor searches on the visual embeddings.
  arXiv  Detail & Related papers  (2025-07-18T18:21:43Z)
• ImpRAG: Retrieval-Augmented Generation with Implicit Queries [34.72864597562907]
  ImpRAG is a query-free RAG system that integrates retrieval and generation into a unified model.<n>We show that ImpRAG achieves 3.6-11.5 improvements in exact match scores on unseen tasks with diverse formats.
  arXiv  Detail & Related papers  (2025-06-02T21:38:21Z)
• From Single to Multi-Granularity: Toward Long-Term Memory Association and Selection of Conversational Agents [79.87304940020256]
  Large Language Models (LLMs) have been widely adopted in conversational agents.<n>MemGAS is a framework that enhances memory consolidation by constructing multi-granularity association, adaptive selection, and retrieval.<n> Experiments on four long-term memory benchmarks demonstrate that MemGAS outperforms state-of-the-art methods on both question answer and retrieval tasks.
  arXiv  Detail & Related papers  (2025-05-26T06:13:07Z)
• Single-Stage Visual Query Localization in Egocentric Videos [79.71065005161566]
  We propose a single-stage VQL framework that is end-to-end trainable. We establish the query-video relationship by considering query-to-frame correspondences between the query and each video frame. Our experiments demonstrate that our approach outperforms prior VQL methods by 20% accuracy while obtaining a 10x improvement in inference speed.
  arXiv  Detail & Related papers  (2023-06-15T17:57:28Z)
• Enhancing Multi-modal and Multi-hop Question Answering via Structured Knowledge and Unified Retrieval-Generation [33.56304858796142]
  Multi-modal multi-hop question answering involves answering a question by reasoning over multiple input sources from different modalities. Existing methods often retrieve evidences separately and then use a language model to generate an answer based on the retrieved evidences. We propose a Structured Knowledge and Unified Retrieval-Generation (RG) approach to address these issues.
  arXiv  Detail & Related papers  (2022-12-16T18:12:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.