SamToNe: Improving Contrastive Loss for Dual Encoder Retrieval Models
with Same Tower Negatives
- URL: http://arxiv.org/abs/2306.02516v1
- Date: Mon, 5 Jun 2023 00:43:37 GMT
- Title: SamToNe: Improving Contrastive Loss for Dual Encoder Retrieval Models
with Same Tower Negatives
- Authors: Fedor Moiseev, Gustavo Hernandez Abrego, Peter Dornbach, Imed Zitouni,
Enrique Alfonseca, Zhe Dong
- Abstract summary: A standard way to train dual encoders is using a contrastive loss with in-batch negatives.
In this work, we propose an improved contrastive learning objective by adding queries or documents from the same encoder towers to the negatives.
We demonstrate that SamToNe can effectively improve the retrieval quality for both symmetric and asymmetric dual encoders.
- Score: 4.864332428224798
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: Dual encoders have been used for retrieval tasks and representation learning
with good results. A standard way to train dual encoders is using a contrastive
loss with in-batch negatives. In this work, we propose an improved contrastive
learning objective by adding queries or documents from the same encoder towers
to the negatives, for which we name it as "contrastive loss with SAMe TOwer
NEgatives" (SamToNe). By evaluating on question answering retrieval benchmarks
from MS MARCO and MultiReQA, and heterogenous zero-shot information retrieval
benchmarks (BEIR), we demonstrate that SamToNe can effectively improve the
retrieval quality for both symmetric and asymmetric dual encoders. By directly
probing the embedding spaces of the two encoding towers via the t-SNE algorithm
(van der Maaten and Hinton, 2008), we observe that SamToNe ensures the
alignment between the embedding spaces from the two encoder towers. Based on
the analysis of the embedding distance distributions of the top-$1$ retrieved
results, we further explain the efficacy of the method from the perspective of
regularisation.
Related papers
- How to Make Cross Encoder a Good Teacher for Efficient Image-Text Retrieval? [99.87554379608224]
Cross-modal similarity score distribution of cross-encoder is more concentrated while the result of dual-encoder is nearly normal.
Only the relative order between hard negatives conveys valid knowledge while the order information between easy negatives has little significance.
We propose a novel Contrastive Partial Ranking Distillation (DCPR) method which implements the objective of mimicking relative order between hard negative samples with contrastive learning.
arXiv Detail & Related papers (2024-07-10T09:10:01Z) - SparseCL: Sparse Contrastive Learning for Contradiction Retrieval [87.02936971689817]
Contradiction retrieval refers to identifying and extracting documents that explicitly disagree with or refute the content of a query.
Existing methods such as similarity search and crossencoder models exhibit significant limitations.
We introduce SparseCL that leverages specially trained sentence embeddings designed to preserve subtle, contradictory nuances between sentences.
arXiv Detail & Related papers (2024-06-15T21:57:03Z) - REBAR: Retrieval-Based Reconstruction for Time-series Contrastive Learning [64.08293076551601]
We propose a novel method of using a learned measure for identifying positive pairs.
Our Retrieval-Based Reconstruction measure measures the similarity between two sequences.
We show that the REBAR error is a predictor of mutual class membership.
arXiv Detail & Related papers (2023-11-01T13:44:45Z) - Quick Dense Retrievers Consume KALE: Post Training Kullback Leibler
Alignment of Embeddings for Asymmetrical dual encoders [89.29256833403169]
We introduce Kullback Leibler Alignment of Embeddings (KALE), an efficient and accurate method for increasing the inference efficiency of dense retrieval methods.
KALE extends traditional Knowledge Distillation after bi-encoder training, allowing for effective query encoder compression without full retraining or index generation.
Using KALE and asymmetric training, we can generate models which exceed the performance of DistilBERT despite having 3x faster inference.
arXiv Detail & Related papers (2023-03-31T15:44:13Z) - UniASM: Binary Code Similarity Detection without Fine-tuning [0.8271859911016718]
We propose a novel transformer-based binary code embedding model named UniASM to learn representations of the binary functions.
In the real-world task of known vulnerability search, UniASM outperforms all the current baselines.
arXiv Detail & Related papers (2022-10-28T14:04:57Z) - LoopITR: Combining Dual and Cross Encoder Architectures for Image-Text
Retrieval [117.15862403330121]
We propose LoopITR, which combines dual encoders and cross encoders in the same network for joint learning.
Specifically, we let the dual encoder provide hard negatives to the cross encoder, and use the more discriminative cross encoder to distill its predictions back to the dual encoder.
arXiv Detail & Related papers (2022-03-10T16:41:12Z) - Adversarial Retriever-Ranker for dense text retrieval [51.87158529880056]
We present Adversarial Retriever-Ranker (AR2), which consists of a dual-encoder retriever plus a cross-encoder ranker.
AR2 consistently and significantly outperforms existing dense retriever methods.
This includes the improvements on Natural Questions R@5 to 77.9%(+2.1%), TriviaQA R@5 to 78.2%(+1.4), and MS-MARCO MRR@10 to 39.5%(+1.3%)
arXiv Detail & Related papers (2021-10-07T16:41:15Z) - Self-Supervised Bernoulli Autoencoders for Semi-Supervised Hashing [1.8899300124593648]
This paper investigates the robustness of hashing methods based on variational autoencoders to the lack of supervision.
We propose a novel supervision method in which the model uses its label distribution predictions to implement the pairwise objective.
Our experiments show that both methods can significantly increase the hash codes' quality.
arXiv Detail & Related papers (2020-07-17T07:47:10Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.