SVFSearch: A Multimodal Knowledge-Intensive Benchmark for Short-Video Frame Search in the Gaming Vertical Domain

• URL: http://arxiv.org/abs/2605.17946v2
• Date: Wed, 20 May 2026 03:33:36 GMT
• Title: SVFSearch: A Multimodal Knowledge-Intensive Benchmark for Short-Video Frame Search in the Gaming Vertical Domain
• Abstract summary: We introduce SVFSearch, the first open benchmark for short-video frame search in the Chinese gaming domain.<n>We evaluate representative paradigms ranging from direct QA and RAG workflow to Plan-Act-Replan agents and learned search models.<n>Results reveal a large gap between model-only answering, practical agentic search, and oracle knowledge.
• Score: 11.397893639012272
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Multimodal large language models are increasingly used as agent backbones that understand multimodal inputs, plan retrieval actions, invoke external tools, and reason over retrieved information. Yet existing benchmarks rarely evaluate this ability in short-video applications, where a paused frame is often visually ambiguous and answering requires vertical, long-tail, and fast-evolving domain knowledge. We introduce SVFSearch, the first open benchmark for short-video frame search in the Chinese gaming domain. SVFSearch contains 5,000 four-choice test examples and 4,198 auxiliary training examples, each centered on a paused game scene from a real short-video clip. To support fair and reproducible evaluation, SVFSearch provides a frozen offline retrieval environment with a game-domain text corpus, a topic-linked image gallery, and text, image, and multimodal retrieval interfaces, avoiding reliance on uncontrolled web search APIs. We evaluate representative paradigms ranging from direct QA and RAG workflow to Plan-Act-Replan agents and learned search models. Results reveal a large gap between model-only answering, practical agentic search, and oracle knowledge: the best open-source direct-QA model reaches 66.4%, the best practical agent achieves 79.1%, and oracle knowledge reaches 95.4%. Further analysis exposes bottlenecks in visual grounding, retrieval quality, evidence-grounded reasoning, and tool-use behavior, including over-search, answer-only shortcuts, and retrieval-induced misleading.

Related papers

• CrossModalQA: A Cross-modal and Multi-hop Benchmark for Multimodal Retrieval-augmented Generation [15.857678905969273]
  CrossModalQA is an open-domain benchmark for evaluating multimodal retrieval and reasoning over heterogeneous corpora.<n>It covers five complementary reasoning paths: vision-to-text, text-to-vision, vision-to-text-to-vision, multi-image intersection, and image-set reasoning.
  arXiv  Detail & Related papers  (2026-08-31T16:00:59Z)
• VideoSearcher: Empowering Video Deep Research with Multi-Tool Agentic Reasoning via Reinforcement Learning [55.2367119060441]
  Video understanding is moving beyond closed-context perception toward open-world evidence exploration.<n>Existing multimodal search agents primarily target static images, and the current VDR benchmark relies on text-centric retrieval.<n>We propose VideoSearcher, a closed-loop agentic framework that empowers Vision-Language Models with multi-tool reasoning for VDR.
  arXiv  Detail & Related papers  (2026-07-03T03:50:09Z)
• VisBrowse-Bench: Benchmarking Visual-Native Search for Multimodal Browsing Agents [27.574370658412864]
  We introduce a new benchmark for visual-native search, VisBrowse-Bench.<n>It contains 169 VQA instances covering multiple domains.<n>It evaluates the models' visual reasoning capabilities during the search process.
  arXiv  Detail & Related papers  (2026-03-17T09:24:13Z)
• MultiHaystack: Benchmarking Multimodal Retrieval and Reasoning over 40K Images, Videos, and Documents [57.32877731797049]
  MultiHaystack is the first benchmark designed to evaluate both retrieval and reasoning under large-scale, cross-modal conditions.<n>We find that models perform competitively when provided with the corresponding evidence, but their performance drops sharply when required to retrieve that evidence from the full corpus.
  arXiv  Detail & Related papers  (2026-03-05T21:43:02Z)
• Pixel-Grounded Retrieval for Knowledgeable Large Multimodal Models [58.46663983451155]
  PixSearch is an end-to-end Segmenting Large Multimodal Model (LMM) that unifies region-level perception and retrieval-augmented reasoning.<n>During encoding, PixSearch emits search> tokens to trigger retrieval, selects query modalities (text, image, or region), and generates pixel-level masks that directly serve as visual queries.<n>On egocentric and entity-centric VQA benchmarks, PixSearch substantially improves factual consistency and generalization.
  arXiv  Detail & Related papers  (2026-01-27T00:46:08Z)
• Video-BrowseComp: Benchmarking Agentic Video Research on Open Web [64.53060049124961]
  Video-BrowseComp is a benchmark comprising 210 questions tailored for open-web agentic video reasoning.<n>It enforces a mandatory dependency on temporal visual evidence, ensuring answers cannot be derived solely through text search.<n>As the first open-web video research benchmark, Video-BrowseComp advances the field beyond passive perception toward proactive video reasoning.
  arXiv  Detail & Related papers  (2025-12-28T19:08:27Z)
• MMSearch-Plus: Benchmarking Provenance-Aware Search for Multimodal Browsing Agents [44.63565009665076]
  We introduce MMSearch-Plus, a 311-task benchmark that enforces multimodal understanding.<n>We provide a model-agnostic agent framework with standard browsing tools and a set-of-mark (SoM) module.<n>SoM enables provenance-aware zoom-and-retrieve and improves robustness in multi-step reasoning.
  arXiv  Detail & Related papers  (2025-08-29T09:58:27Z)
• MM-BrowseComp: A Comprehensive Benchmark for Multimodal Browsing Agents [78.3863007028688]
  MM-BrowseComp is a novel benchmark comprising 224 challenging, hand-crafted questions.<n>These questions often incorporate images in prompts, and crucial information encountered during the search and reasoning process may also be embedded within images or videos on webpages.<n>Our comprehensive evaluation of state-of-the-art models on MM-BrowseComp reveals that even top models like OpenAI o3 with tools achieve only 29.02% accuracy.
  arXiv  Detail & Related papers  (2025-08-14T13:46:47Z)
• VSI: Visual Subtitle Integration for Keyframe Selection to enhance Long Video Understanding [22.400847202448478]
  Long video understanding presents a significant challenge to large language models (MLs)<n>VisualSubtitleation(VSI) integrates subtitles, semantic timestamps, and scene boundaries into a unified multimodal search process.<n>The proposed method captures the visual information of video frames as well as the complementary textual information through a dual-stream search mechanism.
  arXiv  Detail & Related papers  (2025-08-09T07:38:48Z)
• MMSearch-R1: Incentivizing LMMs to Search [49.889749277236376]
  We present MMSearch-R1, the first end-to-end reinforcement learning framework that enables on-demand, multi-turn search in real-world Internet environments.<n>Our framework integrates both image and text search tools, allowing the model to reason about when and how to invoke them guided by an outcome-based reward with a search penalty.
  arXiv  Detail & Related papers  (2025-06-25T17:59:42Z)
• T*: Re-thinking Temporal Search for Long-Form Video Understanding [66.72243342954823]
  Current temporal search methods only achieve 2.1% temporal F1 score on the Longvideobench subset.<n>Inspired by visual search in images, we propose a lightweight temporal search framework, T* that reframes costly temporal search as spatial search.<n>Extensive experiments show that integrating T* with existing methods significantly improves SOTA long-form video understanding.
  arXiv  Detail & Related papers  (2025-04-03T04:03:10Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.