Regularized Contrastive Learning of Semantic Search

• URL: http://arxiv.org/abs/2209.13241v1
• Date: Tue, 27 Sep 2022 08:25:19 GMT
• Title: Regularized Contrastive Learning of Semantic Search
• Authors: Mingxi Tan, Alexis Rolland, Andong Tian
• Abstract summary: Transformer-based models are widely used as retrieval models due to their excellent ability to learn semantic representations. We propose a new regularization method: Regularized Contrastive Learning. It augments several different semantic representations for every sentence, then take them into the contrastive objective as regulators.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Semantic search is an important task which objective is to find the relevant index from a database for query. It requires a retrieval model that can properly learn the semantics of sentences. Transformer-based models are widely used as retrieval models due to their excellent ability to learn semantic representations. in the meantime, many regularization methods suitable for them have also been proposed. In this paper, we propose a new regularization method: Regularized Contrastive Learning, which can help transformer-based models to learn a better representation of sentences. It firstly augments several different semantic representations for every sentence, then take them into the contrastive objective as regulators. These contrastive regulators can overcome overfitting issues and alleviate the anisotropic problem. We firstly evaluate our approach on 7 semantic search benchmarks with the outperforming pre-trained model SRoBERTA. The results show that our method is more effective for learning a superior sentence representation. Then we evaluate our approach on 2 challenging FAQ datasets, Cough and Faqir, which have long query and index. The results of our experiments demonstrate that our method outperforms baseline methods.

Related papers

• Likelihood as a Performance Gauge for Retrieval-Augmented Generation [78.28197013467157]
  We show that likelihoods serve as an effective gauge for language model performance. We propose two methods that use question likelihood as a gauge for selecting and constructing prompts that lead to better performance.
  arXiv  Detail & Related papers  (2024-11-12T13:14:09Z)
• SHINE: Saliency-aware HIerarchical NEgative Ranking for Compositional Temporal Grounding [52.98133831401225]
  Temporal grounding, also known as video moment retrieval, aims at locating video segments corresponding to a given query sentence. We propose a large language model-driven method for negative query construction, utilizing GPT-3.5-Turbo. We introduce a coarse-to-fine saliency ranking strategy, which encourages the model to learn the multi-granularity semantic relationships between videos and hierarchical negative queries.
  arXiv  Detail & Related papers  (2024-07-06T16:08:17Z)
• Adapting Dual-encoder Vision-language Models for Paraphrased Retrieval [55.90407811819347]
  We consider the task of paraphrased text-to-image retrieval where a model aims to return similar results given a pair of paraphrased queries. We train a dual-encoder model starting from a language model pretrained on a large text corpus. Compared to public dual-encoder models such as CLIP and OpenCLIP, the model trained with our best adaptation strategy achieves a significantly higher ranking similarity for paraphrased queries.
  arXiv  Detail & Related papers  (2024-05-06T06:30:17Z)
• Semantic-aware Contrastive Learning for More Accurate Semantic Parsing [32.74456368167872]
  We propose a semantic-aware contrastive learning algorithm, which can learn to distinguish fine-grained meaning representations. Experiments on two standard datasets show that our approach achieves significant improvements over MLE baselines.
  arXiv  Detail & Related papers  (2023-01-19T07:04:32Z)
• Beyond Contrastive Learning: A Variational Generative Model for Multilingual Retrieval [109.62363167257664]
  We propose a generative model for learning multilingual text embeddings. Our model operates on parallel data in $N$ languages. We evaluate this method on a suite of tasks including semantic similarity, bitext mining, and cross-lingual question retrieval.
  arXiv  Detail & Related papers  (2022-12-21T02:41:40Z)
• Query Expansion Using Contextual Clue Sampling with Language Models [69.51976926838232]
  We propose a combination of an effective filtering strategy and fusion of the retrieved documents based on the generation probability of each context. Our lexical matching based approach achieves a similar top-5/top-20 retrieval accuracy and higher top-100 accuracy compared with the well-established dense retrieval model DPR. For end-to-end QA, the reader model also benefits from our method and achieves the highest Exact-Match score against several competitive baselines.
  arXiv  Detail & Related papers  (2022-10-13T15:18:04Z)
• HyperImpute: Generalized Iterative Imputation with Automatic Model Selection [77.86861638371926]
  We propose a generalized iterative imputation framework for adaptively and automatically configuring column-wise models. We provide a concrete implementation with out-of-the-box learners, simulators, and interfaces.
  arXiv  Detail & Related papers  (2022-06-15T19:10:35Z)
• End-to-End Open Vocabulary Keyword Search [13.90172596423425]
  We propose a model directly optimized for keyword search. The proposed model outperforms similar end-to-end models on a task where the ratio of positive and negative trials is artificially balanced. Using our system to rescore the outputs an LVCSR-based keyword search system leads to significant improvements.
  arXiv  Detail & Related papers  (2021-08-23T18:34:53Z)
• Sequential Sentence Matching Network for Multi-turn Response Selection in Retrieval-based Chatbots [45.920841134523286]
  We propose a matching network, called sequential sentence matching network (S2M), to use the sentence-level semantic information to address the problem. Firstly, we find that by using the sentence-level semantic information, the network successfully addresses the problem and gets a significant improvement on matching, resulting in a state-of-the-art performance.
  arXiv  Detail & Related papers  (2020-05-16T09:47:19Z)
• Leveraging Cognitive Search Patterns to Enhance Automated Natural Language Retrieval Performance [0.0]
  We show that cognitive reformulation patterns that mimic user search behaviour are highlighted. We formalize the application of these patterns by considering a query conceptual representation. A genetic algorithm-based weighting process allows placing emphasis on terms according to their conceptual role-type.
  arXiv  Detail & Related papers  (2020-04-21T14:13:33Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.