Modeling Relevance Ranking under the Pre-training and Fine-tuning Paradigm

• URL: http://arxiv.org/abs/2108.05652v1
• Date: Thu, 12 Aug 2021 10:37:12 GMT
• Title: Modeling Relevance Ranking under the Pre-training and Fine-tuning Paradigm
• Authors: Lin Bo, Liang Pang, Gang Wang, Jun Xu, XiuQiang He, Ji-Rong Wen
• Abstract summary: We propose a novel ranking framework called Pre-Rank that takes both user's view and system's view into consideration. To model the user's view of relevance, Pre-Rank pre-trains the initial query-document representations based on large-scale user activities data. To model the system's view of relevance, Pre-Rank further fine-tunes the model on expert-labeled relevance data.
• Score: 44.96049217770624
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Recently, pre-trained language models such as BERT have been applied to document ranking for information retrieval, which first pre-train a general language model on an unlabeled large corpus and then conduct ranking-specific fine-tuning on expert-labeled relevance datasets. Ideally, an IR system would model relevance from a user-system dualism: the user's view and the system's view. User's view judges the relevance based on the activities of "real users" while the system's view focuses on the relevance signals from the system side, e.g., from the experts or algorithms, etc. Inspired by the user-system relevance views and the success of pre-trained language models, in this paper we propose a novel ranking framework called Pre-Rank that takes both user's view and system's view into consideration, under the pre-training and fine-tuning paradigm. Specifically, to model the user's view of relevance, Pre-Rank pre-trains the initial query-document representations based on large-scale user activities data such as the click log. To model the system's view of relevance, Pre-Rank further fine-tunes the model on expert-labeled relevance data. More importantly, the pre-trained representations, are fine-tuned together with handcrafted learning-to-rank features under a wide and deep network architecture. In this way, Pre-Rank can model the relevance by incorporating the relevant knowledge and signals from both real search users and the IR experts. To verify the effectiveness of Pre-Rank, we showed two implementations by using BERT and SetRank as the underlying ranking model, respectively. Experimental results base on three publicly available benchmarks showed that in both of the implementations, Pre-Rank can respectively outperform the underlying ranking models and achieved state-of-the-art performances.

Related papers

• Optimizing Compound Retrieval Systems [30.21938278062652]
  We propose the concept of compound retrieval systems as a broader class of retrieval systems that apply multiple prediction models. We show how our compound approach can combine the classic BM25 retrieval model with state-of-the-art (pairwise) LLM relevance predictions. Our experimental results show optimized compound retrieval systems provide better trade-offs between effectiveness and efficiency than cascading approaches.
  arXiv  Detail & Related papers  (2025-04-16T13:18:16Z)
• RankingSHAP -- Listwise Feature Attribution Explanations for Ranking Models [48.895510739010355]
  We present three key contributions to address this gap. First, we rigorously define listwise feature attribution for ranking models. Second, we introduce RankingSHAP, extending the popular SHAP framework to accommodate listwise ranking attribution. Third, we propose two novel evaluation paradigms for assessing the faithfulness of attributions in learning-to-rank models.
  arXiv  Detail & Related papers  (2024-03-24T10:45:55Z)
• A Visual Interpretation-Based Self-Improved Classification System Using Virtual Adversarial Training [4.722922834127293]
  This paper proposes a visual interpretation-based self-improving classification model with a combination of virtual adversarial training (VAT) and BERT models to address the problems. Specifically, a fine-tuned BERT model is used as a classifier to classify the sentiment of the text. The predicted sentiment classification labels are used as part of the input of another BERT for spam classification via a semi-supervised training manner.
  arXiv  Detail & Related papers  (2023-09-03T15:07:24Z)
• Recommender Systems with Generative Retrieval [58.454606442670034]
  We propose a novel generative retrieval approach, where the retrieval model autoregressively decodes the identifiers of the target candidates. To that end, we create semantically meaningful of codewords to serve as a Semantic ID for each item. We show that recommender systems trained with the proposed paradigm significantly outperform the current SOTA models on various datasets.
  arXiv  Detail & Related papers  (2023-05-08T21:48:17Z)
• Incorporating Relevance Feedback for Information-Seeking Retrieval using Few-Shot Document Re-Ranking [56.80065604034095]
  We introduce a kNN approach that re-ranks documents based on their similarity with the query and the documents the user considers relevant. To evaluate our different integration strategies, we transform four existing information retrieval datasets into the relevance feedback scenario.
  arXiv  Detail & Related papers  (2022-10-19T16:19:37Z)
• Ordinal Graph Gamma Belief Network for Social Recommender Systems [54.9487910312535]
  We develop a hierarchical Bayesian model termed ordinal graph factor analysis (OGFA), which jointly models user-item and user-user interactions. OGFA not only achieves good recommendation performance, but also extracts interpretable latent factors corresponding to representative user preferences. We extend OGFA to ordinal graph gamma belief network, which is a multi-stochastic-layer deep probabilistic model.
  arXiv  Detail & Related papers  (2022-09-12T09:19:22Z)
• TPRM: A Topic-based Personalized Ranking Model for Web Search [9.032465976745305]
  We propose a topic-based personalized ranking model (TPRM) that integrates user topical profile with pretrained contextualized term representations to tailor the general document ranking list. Experiments on the real-world dataset demonstrate that TPRM outperforms state-of-the-art ad-hoc ranking models and personalized ranking models significantly.
  arXiv  Detail & Related papers  (2021-08-13T01:16:55Z)
• Few-Shot Named Entity Recognition: A Comprehensive Study [92.40991050806544]
  We investigate three schemes to improve the model generalization ability for few-shot settings. We perform empirical comparisons on 10 public NER datasets with various proportions of labeled data. We create new state-of-the-art results on both few-shot and training-free settings.
  arXiv  Detail & Related papers  (2020-12-29T23:43:16Z)
• Document Ranking with a Pretrained Sequence-to-Sequence Model [56.44269917346376]
  We show how a sequence-to-sequence model can be trained to generate relevance labels as "target words" Our approach significantly outperforms an encoder-only model in a data-poor regime.
  arXiv  Detail & Related papers  (2020-03-14T22:29:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.