Towards Reducing Manual Workload in Technology-Assisted Reviews: Estimating Ranking Performance

• URL: http://arxiv.org/abs/2201.05648v1
• Date: Fri, 14 Jan 2022 19:48:45 GMT
• Title: Towards Reducing Manual Workload in Technology-Assisted Reviews: Estimating Ranking Performance
• Authors: Grace E. Lee and Aixin Sun
• Abstract summary: When researchers label studies, they can screen ranked documents where relevant documents are higher than irrelevant ones. This paper investigates the quality of document ranking of systematic reviews. After extensive analysis on SR document rankings, we hypothesize 'topic broadness' as a factor that affects the ranking quality of SR.
• Score: 30.29371206568408
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Conducting a systematic review (SR) is comprised of multiple tasks: (i) collect documents (studies) that are likely to be relevant from digital libraries (eg., PubMed), (ii) manually read and label the documents as relevant or irrelevant, (iii) extract information from the relevant studies, and (iv) analyze and synthesize the information and derive a conclusion of SR. When researchers label studies, they can screen ranked documents where relevant documents are higher than irrelevant ones. This practice, known as screening prioritization (ie., document ranking approach), speeds up the process of conducting a SR as the documents labelled as relevant can move to the next tasks earlier. However, the approach is limited in reducing the manual workload because the total number of documents to screen remains the same. Towards reducing the manual workload in the screening process, we investigate the quality of document ranking of SR. This can signal researchers whereabouts in the ranking relevant studies are located and let them decide where to stop the screening. After extensive analysis on SR document rankings from different ranking models, we hypothesize 'topic broadness' as a factor that affects the ranking quality of SR. Finally, we propose a measure that estimates the topic broadness and demonstrate that the proposed measure is a simple yet effective method to predict the qualities of document rankings for SRs.

Related papers

• JudgeRank: Leveraging Large Language Models for Reasoning-Intensive Reranking [81.88787401178378]
  We introduce JudgeRank, a novel agentic reranker that emulates human cognitive processes when assessing document relevance. We evaluate JudgeRank on the reasoning-intensive BRIGHT benchmark, demonstrating substantial performance improvements over first-stage retrieval methods. In addition, JudgeRank performs on par with fine-tuned state-of-the-art rerankers on the popular BEIR benchmark, validating its zero-shot generalization capability.
  arXiv  Detail & Related papers  (2024-10-31T18:43:12Z)
• Generating Natural Language Queries for More Effective Systematic Review Screening Prioritisation [53.77226503675752]
  The current state of the art uses the final title of the review as a query to rank the documents using BERT-based neural rankers. In this paper, we explore alternative sources of queries for prioritising screening, such as the Boolean query used to retrieve the documents to be screened and queries generated by instruction-based large-scale language models such as ChatGPT and Alpaca. Our best approach is not only viable based on the information available at the time of screening, but also has similar effectiveness to the final title.
  arXiv  Detail & Related papers  (2023-09-11T05:12:14Z)
• Open Domain Multi-document Summarization: A Comprehensive Study of Model Brittleness under Retrieval [42.73076855699184]
  Multi-document summarization (MDS) assumes a set of topic-related documents are provided as input. We study this more challenging setting by formalizing the task and bootstrapping it using existing datasets, retrievers and summarizers.
  arXiv  Detail & Related papers  (2022-12-20T18:41:38Z)
• Neural Rankers for Effective Screening Prioritisation in Medical Systematic Review Literature Search [31.797257552928336]
  We apply several pre-trained language models to the systematic review document ranking task. An empirical analysis compares how effective neural methods compare to traditional methods for this task. Our results show that BERT-based rankers outperform the current state-of-the-art screening prioritisation methods.
  arXiv  Detail & Related papers  (2022-12-18T05:26:40Z)
• Incorporating Relevance Feedback for Information-Seeking Retrieval using Few-Shot Document Re-Ranking [56.80065604034095]
  We introduce a kNN approach that re-ranks documents based on their similarity with the query and the documents the user considers relevant. To evaluate our different integration strategies, we transform four existing information retrieval datasets into the relevance feedback scenario.
  arXiv  Detail & Related papers  (2022-10-19T16:19:37Z)
• GERE: Generative Evidence Retrieval for Fact Verification [57.78768817972026]
  We propose GERE, the first system that retrieves evidences in a generative fashion. The experimental results on the FEVER dataset show that GERE achieves significant improvements over the state-of-the-art baselines.
  arXiv  Detail & Related papers  (2022-04-12T03:49:35Z)
• Augmenting Document Representations for Dense Retrieval with Interpolation and Perturbation [49.940525611640346]
  Document Augmentation for dense Retrieval (DAR) framework augments the representations of documents with their Dense Augmentation and perturbations. We validate the performance of DAR on retrieval tasks with two benchmark datasets, showing that the proposed DAR significantly outperforms relevant baselines on the dense retrieval of both the labeled and unlabeled documents.
  arXiv  Detail & Related papers  (2022-03-15T09:07:38Z)
• Mirror Matching: Document Matching Approach in Seed-driven Document Ranking for Medical Systematic Reviews [31.3220495275256]
  Document ranking is an approach for assisting researchers by providing document rankings where relevant documents are ranked higher than irrelevant ones. We propose a document matching measure named Mirror Matching, which calculates matching scores between medical abstract texts by incorporating common writing patterns.
  arXiv  Detail & Related papers  (2021-12-28T22:27:52Z)
• Automating Document Classification with Distant Supervision to Increase the Efficiency of Systematic Reviews [18.33687903724145]
  Well-done systematic reviews are expensive, time-demanding, and labor-intensive. We propose an automatic document classification approach to significantly reduce the effort in reviewing documents.
  arXiv  Detail & Related papers  (2020-12-09T22:45:40Z)
• Overview of the TREC 2019 Fair Ranking Track [65.15263872493799]
  The goal of the TREC Fair Ranking track was to develop a benchmark for evaluating retrieval systems in terms of fairness to different content providers. This paper presents an overview of the track, including the task definition, descriptions of the data and the annotation process.
  arXiv  Detail & Related papers  (2020-03-25T21:34:58Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.