Generalized Contrastive Learning for Multi-Modal Retrieval and Ranking

• URL: http://arxiv.org/abs/2404.08535v1
• Date: Fri, 12 Apr 2024 15:30:03 GMT
• Title: Generalized Contrastive Learning for Multi-Modal Retrieval and Ranking
• Authors: Tianyu Zhu, Myong Chol Jung, Jesse Clark,
• Abstract summary: We propose Generalized Contrastive Learning for Multi-Modal Retrieval and Ranking (GCL) GCL is designed to learn from fine-grained rankings beyond binary relevance scores. Our results show that GCL achieves a 94.5% increase in NDCG@10 for in-domain and 26.3 to 48.8% increases for cold-start evaluations.
• Score: 2.5238707656136694
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Contrastive learning has gained widespread adoption for retrieval tasks due to its minimal requirement for manual annotations. However, popular contrastive frameworks typically learn from binary relevance, making them ineffective at incorporating direct fine-grained rankings. In this paper, we curate a large-scale dataset featuring detailed relevance scores for each query-document pair to facilitate future research and evaluation. Subsequently, we propose Generalized Contrastive Learning for Multi-Modal Retrieval and Ranking (GCL), which is designed to learn from fine-grained rankings beyond binary relevance scores. Our results show that GCL achieves a 94.5% increase in NDCG@10 for in-domain and 26.3 to 48.8% increases for cold-start evaluations, all relative to the CLIP baseline and involving ground truth rankings.

Related papers

• A Large-Scale Study of Relevance Assessments with Large Language Models: An Initial Look [52.114284476700874]
  This paper reports on the results of a large-scale evaluation (the TREC 2024 RAG Track) where four different relevance assessment approaches were deployed. We find that automatically generated UMBRELA judgments can replace fully manual judgments to accurately capture run-level effectiveness. Surprisingly, we find that LLM assistance does not appear to increase correlation with fully manual assessments, suggesting that costs associated with human-in-the-loop processes do not bring obvious tangible benefits.
  arXiv  Detail & Related papers  (2024-11-13T01:12:35Z)
• Self-Calibrated Listwise Reranking with Large Language Models [137.6557607279876]
  Large language models (LLMs) have been employed in reranking tasks through a sequence-to-sequence approach. This reranking paradigm requires a sliding window strategy to iteratively handle larger candidate sets. We propose a novel self-calibrated listwise reranking method, which aims to leverage LLMs to produce global relevance scores for ranking.
  arXiv  Detail & Related papers  (2024-11-07T10:31:31Z)
• Efficient course recommendations with T5-based ranking and summarization [2.6968321526169503]
  We develop a two-step retrieval pipeline with RankT5 finetuned on MSMARCO as re-ranker. We evaluate our rankers on two newly labelled datasets, with an A/B test, and with a user questionnaire. We conclude that T5-based re-ranking and summarization for online course recommendation can obtain much better effectiveness than single-step lexical retrieval.
  arXiv  Detail & Related papers  (2024-06-27T09:07:32Z)
• Empowering Large Language Models to Set up a Knowledge Retrieval Indexer via Self-Learning [17.83428132220955]
  We propose a pre-retrieval framework named Pseudo-Graph Retrieval-Augmented Generation (PG-RAG) PG-RAG conceptualizes LLMs as students by providing them with abundant raw reading materials. During the retrieval phase, PG-RAG mimics the human behavior in flipping through notes.
  arXiv  Detail & Related papers  (2024-05-27T08:26:45Z)
• Unbiased Learning to Rank Meets Reality: Lessons from Baidu's Large-Scale Search Dataset [48.708591046906896]
  Unbiased learning-to-rank (ULTR) is a well-established framework for learning from user clicks. We revisit and extend the available experiments on the Baidu-ULTR dataset. We find that standard unbiased learning-to-rank techniques robustly improve click predictions but struggle to consistently improve ranking performance.
  arXiv  Detail & Related papers  (2024-04-03T08:00:46Z)
• Bipartite Ranking Fairness through a Model Agnostic Ordering Adjustment [54.179859639868646]
  We propose a model agnostic post-processing framework xOrder for achieving fairness in bipartite ranking. xOrder is compatible with various classification models and ranking fairness metrics, including supervised and unsupervised fairness metrics. We evaluate our proposed algorithm on four benchmark data sets and two real-world patient electronic health record repositories.
  arXiv  Detail & Related papers  (2023-07-27T07:42:44Z)
• Large Language Models are Effective Text Rankers with Pairwise Ranking Prompting [65.00288634420812]
  Pairwise Ranking Prompting (PRP) is a technique to significantly reduce the burden on Large Language Models (LLMs) Our results are the first in the literature to achieve state-of-the-art ranking performance on standard benchmarks using moderate-sized open-sourced LLMs.
  arXiv  Detail & Related papers  (2023-06-30T11:32:25Z)
• Integrating Rankings into Quantized Scores in Peer Review [61.27794774537103]
  In peer review, reviewers are usually asked to provide scores for the papers. To mitigate this issue, conferences have started to ask reviewers to additionally provide a ranking of the papers they have reviewed. There are no standard procedure for using this ranking information and Area Chairs may use it in different ways. We take a principled approach to integrate the ranking information into the scores.
  arXiv  Detail & Related papers  (2022-04-05T19:39:13Z)
• Query-level Early Exit for Additive Learning-to-Rank Ensembles [14.240566571342924]
  Search engine ranking pipelines are commonly based on large ensembles of machine-learned decision trees. In this paper, we investigate the novel problem of textitquery-level early exiting We show that query-level early exiting achieves an overall gain of up to 7.5% in terms of NDCG@10 with a speedup of the scoring process of up to 2.2x.
  arXiv  Detail & Related papers  (2020-04-30T08:59:45Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.