Efficient course recommendations with T5-based ranking and summarization

• URL: http://arxiv.org/abs/2406.19018v1
• Date: Thu, 27 Jun 2024 09:07:32 GMT
• Title: Efficient course recommendations with T5-based ranking and summarization
• Authors: Thijmen Bijl, Niels van Weeren, Suzan Verberne,
• Abstract summary: We develop a two-step retrieval pipeline with RankT5 finetuned on MSMARCO as re-ranker. We evaluate our rankers on two newly labelled datasets, with an A/B test, and with a user questionnaire. We conclude that T5-based re-ranking and summarization for online course recommendation can obtain much better effectiveness than single-step lexical retrieval.
• Score: 2.6968321526169503
• License:
• Abstract: In this paper, we implement and evaluate a two-stage retrieval pipeline for a course recommender system that ranks courses for skill-occupation pairs. The in-production recommender system BrightFit provides course recommendations from multiple sources. Some of the course descriptions are long and noisy, while retrieval and ranking in an online system have to be highly efficient. We developed a two-step retrieval pipeline with RankT5 finetuned on MSMARCO as re-ranker. We compare two summarizers for course descriptions: a LongT5 model that we finetuned for the task, and a generative LLM (Vicuna) with in-context learning. We experiment with quantization to reduce the size of the ranking model and increase inference speed. We evaluate our rankers on two newly labelled datasets, with an A/B test, and with a user questionnaire. On the two labelled datasets, our proposed two-stage ranking with automatic summarization achieves a substantial improvement over the in-production (BM25) ranker: nDCG@10 scores improve from 0.482 to 0.684 and from 0.447 to 0.844 on the two datasets. We also achieve a 40% speed-up by using a quantized version of RankT5. The improved quality of the ranking was confirmed by the questionnaire completed by 29 respondents, but not by the A/B test. In the A/B test, a higher clickthrough rate was observed for the BM25-ranking than for the proposed two-stage retrieval. We conclude that T5-based re-ranking and summarization for online course recommendation can obtain much better effectiveness than single-step lexical retrieval, and that quantization has a large effect on RankT5. In the online evaluation, however, other factors than relevance play a role (such as speed and interpretability of the retrieval results), as well as individual preferences.

Related papers

• Generalized Contrastive Learning for Multi-Modal Retrieval and Ranking [2.5238707656136694]
  We propose Generalized Contrastive Learning for Multi-Modal Retrieval and Ranking (GCL) GCL is designed to learn from fine-grained rankings beyond binary relevance scores. Our results show that GCL achieves a 94.5% increase in NDCG@10 for in-domain and 26.3 to 48.8% increases for cold-start evaluations.
  arXiv  Detail & Related papers  (2024-04-12T15:30:03Z)
• Large Language Models are Effective Text Rankers with Pairwise Ranking Prompting [65.00288634420812]
  Pairwise Ranking Prompting (PRP) is a technique to significantly reduce the burden on Large Language Models (LLMs) Our results are the first in the literature to achieve state-of-the-art ranking performance on standard benchmarks using moderate-sized open-sourced LLMs.
  arXiv  Detail & Related papers  (2023-06-30T11:32:25Z)
• T5Score: Discriminative Fine-tuning of Generative Evaluation Metrics [94.69907794006826]
  We present a framework that combines the best of both worlds, using both supervised and unsupervised signals from whatever data we have available. We operationalize this idea by training T5Score, a metric that uses these training signals with mT5 as the backbone. T5Score achieves the best performance on all datasets against existing top-scoring metrics at the segment level.
  arXiv  Detail & Related papers  (2022-12-12T06:29:04Z)
• Incorporating Relevance Feedback for Information-Seeking Retrieval using Few-Shot Document Re-Ranking [56.80065604034095]
  We introduce a kNN approach that re-ranks documents based on their similarity with the query and the documents the user considers relevant. To evaluate our different integration strategies, we transform four existing information retrieval datasets into the relevance feedback scenario.
  arXiv  Detail & Related papers  (2022-10-19T16:19:37Z)
• RocketQAv2: A Joint Training Method for Dense Passage Retrieval and Passage Re-ranking [89.82301733609279]
  We propose a novel joint training approach for dense passage retrieval and passage re-ranking. A major contribution is that we introduce the dynamic listwise distillation, where we design a unified listwise training approach for both the retriever and the re-ranker. During the dynamic distillation, the retriever and the re-ranker can be adaptively improved according to each other's relevance information.
  arXiv  Detail & Related papers  (2021-10-14T13:52:55Z)
• Efficiently Teaching an Effective Dense Retriever with Balanced Topic Aware Sampling [37.01593605084575]
  TAS-Balanced is an efficient topic-aware query and balanced margin sampling technique. We show that our TAS-Balanced training method achieves state-of-the-art low-latency (64ms per query) results on two TREC Deep Learning Track query sets.
  arXiv  Detail & Related papers  (2021-04-14T16:49:18Z)
• A Systematic Evaluation of Transfer Learning and Pseudo-labeling with BERT-based Ranking Models [2.0498977512661267]
  We evaluate transferability of BERT-based neural ranking models across five English datasets. Each of our collections has a substantial number of queries, which enables a full-shot evaluation mode. We find that training on pseudo-labels can produce a competitive or better model compared to transfer learning.
  arXiv  Detail & Related papers  (2021-03-04T21:08:06Z)
• PiRank: Learning To Rank via Differentiable Sorting [85.28916333414145]
  We propose PiRank, a new class of differentiable surrogates for ranking. We show that PiRank exactly recovers the desired metrics in the limit of zero temperature.
  arXiv  Detail & Related papers  (2020-12-12T05:07:36Z)
• Towards Model-Agnostic Post-Hoc Adjustment for Balancing Ranking Fairness and Algorithm Utility [54.179859639868646]
  Bipartite ranking aims to learn a scoring function that ranks positive individuals higher than negative ones from labeled data. There have been rising concerns on whether the learned scoring function can cause systematic disparity across different protected groups. We propose a model post-processing framework for balancing them in the bipartite ranking scenario.
  arXiv  Detail & Related papers  (2020-06-15T10:08:39Z)
• Listwise Learning to Rank with Deep Q-Networks [3.9726605190181976]
  We show that DeepQRank, our deep q-learning to rank agent, demonstrates performance that can be considered state-of-the-art. We run our algorithm against Microsoft's LETOR listwise dataset and achieve an NDCG@1 of 0.5075, narrowly beating out the leading supervised learning model, SVMRank (0.4958)
  arXiv  Detail & Related papers  (2020-02-13T22:45:56Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.