Dense Retrieval with Continuous Explicit Feedback for Systematic Review Screening Prioritisation

• URL: http://arxiv.org/abs/2407.00635v2
• Date: Wed, 17 Jul 2024 10:26:36 GMT
• Title: Dense Retrieval with Continuous Explicit Feedback for Systematic Review Screening Prioritisation
• Authors: Xinyu Mao, Shengyao Zhuang, Bevan Koopman, Guido Zuccon,
• Abstract summary: The goal of screening prioritisation in systematic reviews is to identify relevant documents with high recall and rank them in early positions for review. Recent studies have shown that neural models have good potential on this task, but their time-consuming fine-tuning and inference discourage their widespread use for screening prioritisation. We propose an alternative approach that still relies on neural models, but leverages dense representations and relevance feedback to enhance screening prioritisation.
• Score: 28.80089773616623
• License:
• Abstract: The goal of screening prioritisation in systematic reviews is to identify relevant documents with high recall and rank them in early positions for review. This saves reviewing effort if paired with a stopping criterion, and speeds up review completion if performed alongside downstream tasks. Recent studies have shown that neural models have good potential on this task, but their time-consuming fine-tuning and inference discourage their widespread use for screening prioritisation. In this paper, we propose an alternative approach that still relies on neural models, but leverages dense representations and relevance feedback to enhance screening prioritisation, without the need for costly model fine-tuning and inference. This method exploits continuous relevance feedback from reviewers during document screening to efficiently update the dense query representation, which is then applied to rank the remaining documents to be screened. We evaluate this approach across the CLEF TAR datasets for this task. Results suggest that the investigated dense query-driven approach is more efficient than directly using neural models and shows promising effectiveness compared to previous methods developed on the considered datasets. Our code is available at https://github.com/ielab/dense-screening-feedback.

Related papers

• Quam: Adaptive Retrieval through Query Affinity Modelling [15.3583908068962]
  Building relevance models to rank documents based on user information needs is a central task in information retrieval and the NLP community. We propose a unifying view of the nascent area of adaptive retrieval by proposing, Quam. Our proposed approach, Quam improves the recall performance by up to 26% over the standard re-ranking baselines.
  arXiv  Detail & Related papers  (2024-10-26T22:52:12Z)
• Sample Complexity of Preference-Based Nonparametric Off-Policy Evaluation with Deep Networks [58.469818546042696]
  We study the sample efficiency of OPE with human preference and establish a statistical guarantee for it. By appropriately selecting the size of a ReLU network, we show that one can leverage any low-dimensional manifold structure in the Markov decision process.
  arXiv  Detail & Related papers  (2023-10-16T16:27:06Z)
• Generating Natural Language Queries for More Effective Systematic Review Screening Prioritisation [53.77226503675752]
  The current state of the art uses the final title of the review as a query to rank the documents using BERT-based neural rankers. In this paper, we explore alternative sources of queries for prioritising screening, such as the Boolean query used to retrieve the documents to be screened and queries generated by instruction-based large-scale language models such as ChatGPT and Alpaca. Our best approach is not only viable based on the information available at the time of screening, but also has similar effectiveness to the final title.
  arXiv  Detail & Related papers  (2023-09-11T05:12:14Z)
• Neural Rankers for Effective Screening Prioritisation in Medical Systematic Review Literature Search [31.797257552928336]
  We apply several pre-trained language models to the systematic review document ranking task. An empirical analysis compares how effective neural methods compare to traditional methods for this task. Our results show that BERT-based rankers outperform the current state-of-the-art screening prioritisation methods.
  arXiv  Detail & Related papers  (2022-12-18T05:26:40Z)
• Incorporating Relevance Feedback for Information-Seeking Retrieval using Few-Shot Document Re-Ranking [56.80065604034095]
  We introduce a kNN approach that re-ranks documents based on their similarity with the query and the documents the user considers relevant. To evaluate our different integration strategies, we transform four existing information retrieval datasets into the relevance feedback scenario.
  arXiv  Detail & Related papers  (2022-10-19T16:19:37Z)
• Sequential Learning Of Neural Networks for Prequential MDL [18.475866691786695]
  We evaluate approaches for computing prequential description lengths for image classification datasets with neural networks. Considering the computational cost, we find that online-learning with rehearsal has favorable performance. We present description lengths for a suite of image classification datasets that improve upon previously reported results by large margins.
  arXiv  Detail & Related papers  (2022-10-14T16:30:23Z)
• ReAct: Temporal Action Detection with Relational Queries [84.76646044604055]
  This work aims at advancing temporal action detection (TAD) using an encoder-decoder framework with action queries. We first propose a relational attention mechanism in the decoder, which guides the attention among queries based on their relations. Lastly, we propose to predict the localization quality of each action query at inference in order to distinguish high-quality queries.
  arXiv  Detail & Related papers  (2022-07-14T17:46:37Z)
• A Closer Look at Debiased Temporal Sentence Grounding in Videos: Dataset, Metric, and Approach [53.727460222955266]
  Temporal Sentence Grounding in Videos (TSGV) aims to ground a natural language sentence in an untrimmed video. Recent studies have found that current benchmark datasets may have obvious moment annotation biases. We introduce a new evaluation metric "dR@n,IoU@m" that discounts the basic recall scores to alleviate the inflating evaluation caused by biased datasets.
  arXiv  Detail & Related papers  (2022-03-10T08:58:18Z)
• Deep Reference Priors: What is the best way to pretrain a model? [27.705359364301458]
  This paper formalizes the question using the theory of reference priors. Reference priors are objective, uninformative Bayesian priors that maximize the mutual information between the task and the model. This paper presents the first demonstration of reference priors for medium-scale deep networks and image-based data.
  arXiv  Detail & Related papers  (2022-02-01T02:32:39Z)
• Modeling Online Behavior in Recommender Systems: The Importance of Temporal Context [30.894950420437926]
  We show how omitting temporal context when evaluating recommender system performance leads to false confidence. We propose a training procedure to further embed the temporal context in existing models. Results show that including our temporal objective can improve recall@20 by up to 20%.
  arXiv  Detail & Related papers  (2020-09-19T19:36:43Z)
• S^3-Rec: Self-Supervised Learning for Sequential Recommendation with Mutual Information Maximization [104.87483578308526]
  We propose the model S3-Rec, which stands for Self-Supervised learning for Sequential Recommendation. For our task, we devise four auxiliary self-supervised objectives to learn the correlations among attribute, item, subsequence, and sequence. Extensive experiments conducted on six real-world datasets demonstrate the superiority of our proposed method over existing state-of-the-art methods.
  arXiv  Detail & Related papers  (2020-08-18T11:44:10Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.