PRVQL: Progressive Knowledge-guided Refinement for Robust Egocentric Visual Query Localization

• URL: http://arxiv.org/abs/2502.07707v1
• Date: Tue, 11 Feb 2025 17:04:31 GMT
• Title: PRVQL: Progressive Knowledge-guided Refinement for Robust Egocentric Visual Query Localization
• Authors: Bing Fan, Yunhe Feng, Yapeng Tian, Yuewei Lin, Yan Huang, Heng Fan,
• Abstract summary: Egocentric visual query localization (EgoVQL) focuses on localizing the target of interest in space and time from first-person videos.<n>We introduce PRVQL, a novel Progressive knowledge-guided Refinement framework for EgoVQL.
• Score: 32.75411084716383
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Egocentric visual query localization (EgoVQL) focuses on localizing the target of interest in space and time from first-person videos, given a visual query. Despite recent progressive, existing methods often struggle to handle severe object appearance changes and cluttering background in the video due to lacking sufficient target cues, leading to degradation. Addressing this, we introduce PRVQL, a novel Progressive knowledge-guided Refinement framework for EgoVQL. The core is to continuously exploit target-relevant knowledge directly from videos and utilize it as guidance to refine both query and video features for improving target localization. Our PRVQL contains multiple processing stages. The target knowledge from one stage, comprising appearance and spatial knowledge extracted via two specially designed knowledge learning modules, are utilized as guidance to refine the query and videos features for the next stage, which are used to generate more accurate knowledge for further feature refinement. With such a progressive process, target knowledge in PRVQL can be gradually improved, which, in turn, leads to better refined query and video features for localization in the final stage. Compared to previous methods, our PRVQL, besides the given object cues, enjoys additional crucial target information from a video as guidance to refine features, and hence enhances EgoVQL in complicated scenes. In our experiments on challenging Ego4D, PRVQL achieves state-of-the-art result and largely surpasses other methods, showing its efficacy. Our code, model and results will be released at https://github.com/fb-reps/PRVQL.

Related papers

• PVChat: Personalized Video Chat with One-Shot Learning [15.328085576102106]
  PVChat is a one-shot learning framework that enables subject-aware question answering from a single video for each subject. Our approach optimize a Mixture-of-Heads (MoH) enhanced ViLLM on a synthetically augmented video-QA dataset. We evaluate PVChat on diverse datasets covering medical scenarios, TV series, anime, and real-world footage.
  arXiv  Detail & Related papers  (2025-03-21T11:50:06Z)
• LAVID: An Agentic LVLM Framework for Diffusion-Generated Video Detection [14.687867348598035]
  Large Vision Language Model (LVLM) has become an emerging tool for AI-generated content detection. We propose LAVID, a novel LVLMs-based ai-generated video detection with explicit knowledge enhancement. Our proposed pipeline automatically selects a set of explicit knowledge tools for detection, and then adaptively adjusts the structure prompt by self-rewriting.
  arXiv  Detail & Related papers  (2025-02-20T19:34:58Z)
• RELOCATE: A Simple Training-Free Baseline for Visual Query Localization Using Region-Based Representations [55.74675012171316]
  RELOCATE is a training-free baseline designed to perform the challenging task of visual query localization in long videos.<n>To eliminate the need for task-specific training, RELOCATE leverages a region-based representation derived from pretrained vision models.
  arXiv  Detail & Related papers  (2024-12-02T18:59:53Z)
• Teaching VLMs to Localize Specific Objects from In-context Examples [56.797110842152]
  Vision-Language Models (VLMs) have shown remarkable capabilities across diverse visual tasks. Current VLMs lack a fundamental cognitive ability: learning to localize objects in a scene by taking into account the context. This work is the first to explore and benchmark personalized few-shot localization for VLMs.
  arXiv  Detail & Related papers  (2024-11-20T13:34:22Z)
• Zero-shot Action Localization via the Confidence of Large Vision-Language Models [19.683461002518147]
  We introduce a true ZEro-shot Action localization method (ZEAL) Specifically, we leverage the built-in action knowledge of a large language model (LLM) to inflate actions into highly-detailed descriptions. We demonstrate remarkable results in zero-shot action localization on a challenging benchmark, without any training.
  arXiv  Detail & Related papers  (2024-10-18T09:51:14Z)
• VaQuitA: Enhancing Alignment in LLM-Assisted Video Understanding [63.075626670943116]
  We introduce a cutting-edge framework, VaQuitA, designed to refine the synergy between video and textual information. At the data level, instead of sampling frames uniformly, we implement a sampling method guided by CLIP-score rankings. At the feature level, we integrate a trainable Video Perceiver alongside a Visual-Query Transformer.
  arXiv  Detail & Related papers  (2023-12-04T19:48:02Z)
• Single-Stage Visual Query Localization in Egocentric Videos [79.71065005161566]
  We propose a single-stage VQL framework that is end-to-end trainable. We establish the query-video relationship by considering query-to-frame correspondences between the query and each video frame. Our experiments demonstrate that our approach outperforms prior VQL methods by 20% accuracy while obtaining a 10x improvement in inference speed.
  arXiv  Detail & Related papers  (2023-06-15T17:57:28Z)
• VRAG: Region Attention Graphs for Content-Based Video Retrieval [85.54923500208041]
  Region Attention Graph Networks (VRAG) improves the state-of-the-art video-level methods. VRAG represents videos at a finer granularity via region-level features and encodes video-temporal dynamics through region-level relations. We show that the performance gap between video-level and frame-level methods can be reduced by segmenting videos into shots and using shot embeddings for video retrieval.
  arXiv  Detail & Related papers  (2022-05-18T16:50:45Z)
• Unsupervised Domain Adaptation for Video Semantic Segmentation [91.30558794056054]
  Unsupervised Domain Adaptation for semantic segmentation has gained immense popularity since it can transfer knowledge from simulation to real. In this work, we present a new video extension of this task, namely Unsupervised Domain Adaptation for Video Semantic approaches. We show that our proposals significantly outperform previous image-based UDA methods both on image-level (mIoU) and video-level (VPQ) evaluation metrics.
  arXiv  Detail & Related papers  (2021-07-23T07:18:20Z)
• When Video Classification Meets Incremental Classes [12.322018693269952]
  We propose a framework to address the challenge of textitcatastrophic forgetting forgetting. To better it, we utilize some characteristics of videos. First, we alleviate the granularity-temporal knowledge before distillation. Second, we propose a dual exemplar selection method to select and store representative video instances of old classes and key-frames inside videos under tight storage budget.
  arXiv  Detail & Related papers  (2021-06-30T06:12:33Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.