Prompt-aware of Frame Sampling for Efficient Text-Video Retrieval

• URL: http://arxiv.org/abs/2507.15491v1
• Date: Mon, 21 Jul 2025 10:46:49 GMT
• Title: Prompt-aware of Frame Sampling for Efficient Text-Video Retrieval
• Abstract summary: ProCLIP is a user-centric framework that achieves state-of-the-art accuracy with significantly improved efficiency.<n>We design a prompt-aware frame sampling strategy that dynamically guides lightweight feature extractors using textual prompts to select semantically relevant frames.<n>We adopt a two-stage candidate pruning strategy that combines rapid coarse filtering via a lightweight module with CLIP-powered fine-grained re-ranking, enhancing retrieval efficiency while preserving accuracy.
• Score: 21.65508603851204
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Enabling efficient text-video retrieval on edge-end devices is critical for real-world applications. Yet, existing methods face a critical challenge in balancing accuracy and computational efficiency: uniform frame sampling methods ensure content coverage but incur prohibitive computational costs, while salient-frame sampling methods reduce overhead but suffer from query-agnostic frame selection that biases retrieval results. To address this, we propose ProCLIP, a user-centric framework that achieves state-of-the-art accuracy with significantly improved efficiency. We design a prompt-aware frame sampling strategy that dynamically guides lightweight feature extractors using textual prompts to select semantically relevant frames, overcoming the limitations of existing salient-frame sampling methods which rely on static, query-agnostic selection criteria. Moreover, we adopt a two-stage candidate pruning strategy that combines rapid coarse filtering via a lightweight module with CLIP-powered fine-grained re-ranking, enhancing retrieval efficiency while preserving accuracy. Experiments across benchmarks show ProCLIP achieves 75.3% latency reduction versus baselines while maintaining competitive accuracy, i.e., R@1=49.0 in MSR-VTT dataset. Code is available at https://github.com/tiffylong/ProCLIP.

Related papers

• Evidence-Driven Dynamic Visual Selector for Efficient Long Video Understanding [54.840531495584095]
  We propose a dynamic visual selection framework named EviSelect grounded in the target MLLM's internal attention evidence.<n>Our method efficiently approximates maps of the target MLLM using highly compressed visual inputs and sparse attention.<n>EviSelect achieves superior performance compared to existing methods while reducing selected visual tokens by about 50% and achieving a 3.9x end-to-end speedup.
  arXiv  Detail & Related papers  (2026-08-06T09:15:43Z)
• Query-Conditioned Evidential Keyframe Sampling for MLLM-Based Long-Form Video Understanding [21.306038832082553]
  Multimodal Large Language Models (MLLMs) have shown strong performance on video question answering.<n>Their application to long-form videos is constrained by limited context length and computational cost.<n>We propose an evidence-driven sampling framework grounded in information theory.
  arXiv  Detail & Related papers  (2026-04-01T15:02:50Z)
• LooComp: Leverage Leave-One-Out Strategy to Encoder-only Transformer for Efficient Query-aware Context Compression [5.171670380356107]
  We propose a margin-based framework for query-driven context pruning.<n>It identifies sentences that are critical for answering a query by measuring changes in clue richness when they are omitted.<n>Our method yields effective compression ratios without degrading answering performance.
  arXiv  Detail & Related papers  (2026-03-10T05:44:20Z)
• AdaSpot: Spend Resolution Where It Matters for Precise Event Spotting [59.31340724915079]
  Event Spotting is a key task for applications in sports analytics, robotics, and autonomous systems.<n>bfAdaSpot achieves state-of-the-art performance under strict evaluation metrics.
  arXiv  Detail & Related papers  (2026-02-25T16:24:48Z)
• Ada-RS: Adaptive Rejection Sampling for Selective Thinking [8.760287445955045]
  Large language models (LLMs) are increasingly being deployed in cost and latency-sensitive settings.<n>We study selective thinking for tool-using LLMs and introduce Adaptive Rejection Sampling (Ada-RS)<n>Ada-RS scores multiple sampled completions with an adaptive length-penalized reward then applies rejection sampling to retain only high-reward candidates.
  arXiv  Detail & Related papers  (2026-02-23T05:20:15Z)
• SimpleMem: Efficient Lifelong Memory for LLM Agents [73.74399447715052]
  We introduce SimpleMem, an efficient memory framework based on semantic lossless compression.<n>We propose a three-stage pipeline designed to maximize information density and token utilization.<n> Experiments on benchmark datasets show that our method consistently outperforms baseline approaches in accuracy, retrieval efficiency, and inference cost.
  arXiv  Detail & Related papers  (2026-01-05T21:02:49Z)
• Adaptive Prefiltering for High-Dimensional Similarity Search: A Frequency-Aware Approach [0.0]
  We present an adaptive prefiltering framework that partitions the query space into frequency tiers following Zipfian distributions.<n>Experiments on ImageNet-1k using CLIP embeddings demonstrate that frequency-aware budget allocation achieves equivalent recall with 20.4% fewer distance computations.<n>The framework introduces minimal overhead through lightweight frequency tracking and provides graceful degradation for unseen queries through coherence-based fallback policies.
  arXiv  Detail & Related papers  (2025-12-03T10:11:35Z)
• LRANet++: Low-Rank Approximation Network for Accurate and Efficient Text Spotting [118.93173826110815]
  We propose a novel parameterized text shape method based on low-rank approximation for precise detection.<n>By exploiting the inherent shape correlation among different text contours, our method achieves consistency and compactness in shape representation.<n>We integrate the enhanced detection module with a lightweight recognition branch to form an end-to-end text spotting framework, termed LRANet++.
  arXiv  Detail & Related papers  (2025-11-08T03:08:03Z)
• A.I.R.: Enabling Adaptive, Iterative, and Reasoning-based Frame Selection For Video Question Answering [15.220013605396396]
  A.I.R. is a training-free approach for Adaptive, Iterative, and Reasoning-based frame selection.<n>We leverage a powerful Vision-Language Models (VLMs) to perform deep, semantic analysis on complex queries.<n>Our approach significantly boosts the performance of the foundation VLM, and achieves substantial gains in computational efficiency.
  arXiv  Detail & Related papers  (2025-10-06T01:51:13Z)
• Less is More: Token-Efficient Video-QA via Adaptive Frame-Pruning and Semantic Graph Integration [21.69452489173625]
  "Less is more" phenomenon where excessive frames can paradoxically degrade performance due to context dilution.<n>"Visual echoes" yield significant temporal redundancy, which we term 'visual echoes'<n>"AFP" employs an adaptive hierarchical clustering algorithm on a fused ResNet-50 and CLIP feature space to identify and merge these echoes into single representatives.<n>Our full approach demonstrates a drastic reduction in required frames by up to 86.9% and total input tokens by up to 83.2%.
  arXiv  Detail & Related papers  (2025-08-05T11:31:55Z)
• E-VRAG: Enhancing Long Video Understanding with Resource-Efficient Retrieval Augmented Generation [8.441615871480858]
  We propose E-VRAG, a novel and efficient video RAG framework for video understanding.<n>We first apply a frame pre-filtering method based on hierarchical query decomposition to eliminate irrelevant frames.<n>We then employ a lightweight VLM for frame scoring, further reducing computational costs at the model level.
  arXiv  Detail & Related papers  (2025-08-03T02:09:54Z)
• An Memory-Efficient Framework for Deformable Transformer with Neural Architecture Search [4.533437219543447]
  Deformable Attention Transformers ( DAT) have shown remarkable performance in computer vision tasks.<n>Existing acceleration methods either incur high hardware overhead or compromise model accuracy.<n>This paper proposes a hardware-friendly optimization framework for DAT.
  arXiv  Detail & Related papers  (2025-07-13T06:42:00Z)
• ReFoCUS: Reinforcement-guided Frame Optimization for Contextual Understanding [52.050036778325094]
  We introduce ReFoCUS (Reinforcement-guided Frame Optimization for Contextual UnderStanding), a novel frame-level policy optimization framework.<n>ReFoCUS learns a frame selection policy via reinforcement learning, using reward signals derived from a reference LMM to reflect the model's intrinsic preferences for frames.<n>Our approach consistently improves reasoning performance across multiple video QA benchmarks.
  arXiv  Detail & Related papers  (2025-06-02T03:08:07Z)
• Seeing What Matters: Empowering CLIP with Patch Generation-to-Selection [54.21851618853518]
  We present a concise yet effective approach called Patch Generation-to-Selection to enhance CLIP's training efficiency.<n>Our approach, CLIP-PGS, sets new state-of-the-art results in zero-shot classification and retrieval tasks.
  arXiv  Detail & Related papers  (2025-03-21T12:10:38Z)
• Balancing Accuracy and Efficiency for Large-Scale SLAM: A Minimal Subset Approach for Scalable Loop Closures [7.066382982173528]
  This article proposes an online sampling approach that constructs the pose graph using the most impactful superiors for loop closure.<n> evaluations on diverse public datasets show that the proposed approach outperforms naive methods in reducing false positive rates in place recognition.
  arXiv  Detail & Related papers  (2025-01-03T12:48:01Z)
• TAPT: Test-Time Adversarial Prompt Tuning for Robust Inference in Vision-Language Models [53.91006249339802]
  We propose a novel defense method called Test-Time Adversarial Prompt Tuning (TAPT) to enhance the inference robustness of CLIP against visual adversarial attacks. TAPT is a test-time defense method that learns defensive bimodal (textual and visual) prompts to robustify the inference process of CLIP. We evaluate the effectiveness of TAPT on 11 benchmark datasets, including ImageNet and 10 other zero-shot datasets.
  arXiv  Detail & Related papers  (2024-11-20T08:58:59Z)
• Bridging SFT and DPO for Diffusion Model Alignment with Self-Sampling Preference Optimization [67.8738082040299]
  Self-Sampling Preference Optimization (SSPO) is a new alignment method for post-training reinforcement learning.<n>SSPO eliminates the need for paired data and reward models while retaining the training stability of SFT.<n>SSPO surpasses all previous approaches on the text-to-image benchmarks and demonstrates outstanding performance on the text-to-video benchmarks.
  arXiv  Detail & Related papers  (2024-10-07T17:56:53Z)
• Transforming Image Super-Resolution: A ConvFormer-based Efficient Approach [58.57026686186709]
  We introduce the Convolutional Transformer layer (ConvFormer) and propose a ConvFormer-based Super-Resolution network (CFSR) CFSR inherits the advantages of both convolution-based and transformer-based approaches. Experiments demonstrate that CFSR strikes an optimal balance between computational cost and performance.
  arXiv  Detail & Related papers  (2024-01-11T03:08:00Z)
• TMP: Temporal Motion Propagation for Online Video Super-Resolution [20.56797955203068]
  Online video super-resolution (online-VSR) highly relies on an effective alignment module to aggregate temporal information. Most of the existing online-VSR methods estimate the motion fields of each frame separately to perform alignment. We propose an efficient Temporal Motion propagation (TMP) method, which leverages the continuity of motion field to achieve fast pixel-level alignment.
  arXiv  Detail & Related papers  (2023-12-15T16:17:34Z)
• Efficient Adversarial Contrastive Learning via Robustness-Aware Coreset Selection [59.77647907277523]
  Adversarial contrast learning (ACL) does not require expensive data annotations but outputs a robust representation that withstands adversarial attacks. ACL needs tremendous running time to generate the adversarial variants of all training data. This paper proposes a robustness-aware coreset selection (RCS) method to speed up ACL.
  arXiv  Detail & Related papers  (2023-02-08T03:20:14Z)
• OCSampler: Compressing Videos to One Clip with Single-step Sampling [82.0417131211353]
  We propose a framework named OCSampler to explore a compact yet effective video representation with one short clip. Our basic motivation is that the efficient video recognition task lies in processing a whole sequence at once rather than picking up frames sequentially.
  arXiv  Detail & Related papers  (2022-01-12T09:50:38Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.