Test-time Adaptation for Cross-modal Retrieval with Query Shift

• URL: http://arxiv.org/abs/2410.15624v1
• Date: Mon, 21 Oct 2024 04:08:19 GMT
• Title: Test-time Adaptation for Cross-modal Retrieval with Query Shift
• Authors: Haobin Li, Peng Hu, Qianjun Zhang, Xi Peng, Xiting Liu, Mouxing Yang,
• Abstract summary: We propose a novel method dubbed Test-time adaptation for Cross-modal Retrieval (TCR) In this paper, we observe that query shift would not only diminish the uniformity (namely, within-modality scatter) of the query modality but also amplify the gap between query and gallery modalities.
• Score: 14.219337695007207
• License:
• Abstract: The success of most existing cross-modal retrieval methods heavily relies on the assumption that the given queries follow the same distribution of the source domain. However, such an assumption is easily violated in real-world scenarios due to the complexity and diversity of queries, thus leading to the query shift problem. Specifically, query shift refers to the online query stream originating from the domain that follows a different distribution with the source one. In this paper, we observe that query shift would not only diminish the uniformity (namely, within-modality scatter) of the query modality but also amplify the gap between query and gallery modalities. Based on the observations, we propose a novel method dubbed Test-time adaptation for Cross-modal Retrieval (TCR). In brief, TCR employs a novel module to refine the query predictions (namely, retrieval results of the query) and a joint objective to prevent query shift from disturbing the common space, thus achieving online adaptation for the cross-modal retrieval models with query shift. Expensive experiments demonstrate the effectiveness of the proposed TCR against query shift. The code will be released upon acceptance.

Related papers

• pEBR: A Probabilistic Approach to Embedding Based Retrieval [4.8338111302871525]
  Embedding retrieval aims to learn a shared semantic representation space for both queries and items. In current industrial practice, retrieval systems typically retrieve a fixed number of items for different queries.
  arXiv  Detail & Related papers  (2024-10-25T07:14:12Z)
• Query matching for spatio-temporal action detection with query-based object detector [0.0]
  We propose a method that extends the query-based object detection model, DETR, to maintain temporal consistency in videos. Our method applies DETR to each frame and uses feature shift to incorporate temporal information. To overcome this issue, we propose query matching across different frames, ensuring that queries for the same object are matched and used for the feature shift.
  arXiv  Detail & Related papers  (2024-09-27T02:54:24Z)
• Self-Supervised Query Reformulation for Code Search [6.415583252034772]
  We propose SSQR, a self-supervised query reformulation method that does not rely on any parallel query corpus. Inspired by pre-trained models, SSQR treats query reformulation as a masked language modeling task.
  arXiv  Detail & Related papers  (2023-07-01T08:17:23Z)
• Direct Diffusion Bridge using Data Consistency for Inverse Problems [65.04689839117692]
  Diffusion model-based inverse problem solvers have shown impressive performance, but are limited in speed. Several recent works have tried to alleviate this problem by building a diffusion process, directly bridging the clean and the corrupted. We propose a modified inference procedure that imposes data consistency without the need for fine-tuning.
  arXiv  Detail & Related papers  (2023-05-31T12:51:10Z)
• CAPSTONE: Curriculum Sampling for Dense Retrieval with Document Expansion [68.19934563919192]
  We propose a curriculum sampling strategy that utilizes pseudo queries during training and progressively enhances the relevance between the generated query and the real query. Experimental results on both in-domain and out-of-domain datasets demonstrate that our approach outperforms previous dense retrieval models.
  arXiv  Detail & Related papers  (2022-12-18T15:57:46Z)
• LoL: A Comparative Regularization Loss over Query Reformulation Losses for Pseudo-Relevance Feedback [70.44530794897861]
  Pseudo-relevance feedback (PRF) has proven to be an effective query reformulation technique to improve retrieval accuracy. Existing PRF methods independently treat revised queries originating from the same query but using different numbers of feedback documents. We propose the Loss-over-Loss (LoL) framework to compare the reformulation losses between different revisions of the same query during training.
  arXiv  Detail & Related papers  (2022-04-25T10:42:50Z)
• Session-Aware Query Auto-completion using Extreme Multi-label Ranking [61.753713147852125]
  We take the novel approach of modeling session-aware query auto-completion as an e Multi-Xtreme Ranking (XMR) problem. We adapt a popular XMR algorithm for this purpose by proposing several modifications to the key steps in the algorithm. Our approach meets the stringent latency requirements for auto-complete systems while leveraging session information in making suggestions.
  arXiv  Detail & Related papers  (2020-12-09T17:56:22Z)
• Incorporating Behavioral Hypotheses for Query Generation [3.5873361386290643]
  We propose a generic encoder-decoder Transformer framework to aggregate arbitrary hypotheses of choice for query generation. Our experimental results show that the proposed approach leads to significant improvements on top-$k$ word error rate and Bert F1 Score.
  arXiv  Detail & Related papers  (2020-10-06T12:38:02Z)
• Query Resolution for Conversational Search with Limited Supervision [63.131221660019776]
  We propose QuReTeC (Query Resolution by Term Classification), a neural query resolution model based on bidirectional transformers. We show that QuReTeC outperforms state-of-the-art models, and furthermore, that our distant supervision method can be used to substantially reduce the amount of human-curated data required to train QuReTeC.
  arXiv  Detail & Related papers  (2020-05-24T11:37:22Z)
• Query Focused Multi-Document Summarization with Distant Supervision [88.39032981994535]
  Existing work relies heavily on retrieval-style methods for estimating the relevance between queries and text segments. We propose a coarse-to-fine modeling framework which introduces separate modules for estimating whether segments are relevant to the query. We demonstrate that our framework outperforms strong comparison systems on standard QFS benchmarks.
  arXiv  Detail & Related papers  (2020-04-06T22:35:19Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.