Relevance feedback strategies for recall-oriented neural information retrieval

• URL: http://arxiv.org/abs/2311.15110v1
• Date: Sat, 25 Nov 2023 19:50:41 GMT
• Title: Relevance feedback strategies for recall-oriented neural information retrieval
• Authors: Timo Kats, Peter van der Putten, Jan Scholtes
• Abstract summary: This research proposes a more recall-oriented approach to reducing review effort. More specifically, through iteratively re-ranking the relevance rankings based on user feedback. Our results show that this method can reduce review effort between 17.85% and 59.04%, compared to a baseline approach.
• Score: 0.0
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: In a number of information retrieval applications (e.g., patent search, literature review, due diligence, etc.), preventing false negatives is more important than preventing false positives. However, approaches designed to reduce review effort (like "technology assisted review") can create false negatives, since they are often based on active learning systems that exclude documents automatically based on user feedback. Therefore, this research proposes a more recall-oriented approach to reducing review effort. More specifically, through iteratively re-ranking the relevance rankings based on user feedback, which is also referred to as relevance feedback. In our proposed method, the relevance rankings are produced by a BERT-based dense-vector search and the relevance feedback is based on cumulatively summing the queried and selected embeddings. Our results show that this method can reduce review effort between 17.85% and 59.04%, compared to a baseline approach (of no feedback), given a fixed recall target

Related papers

• Adaptive Querying for Reward Learning from Human Feedback [5.587293092389789]
  Learning from human feedback is a popular approach to train robots to adapt to user preferences and improve safety. We examine how to learn a penalty function associated with unsafe behaviors, such as side effects, using multiple forms of human feedback. We employ an iterative, two-phase approach which first selects critical states for querying, and then uses information gain to select a feedback format for querying.
  arXiv  Detail & Related papers  (2024-12-11T00:02:48Z)
• Real-time Factuality Assessment from Adversarial Feedback [11.742257531343814]
  We show that evaluations for assessing the factuality of news from conventional sources result in high accuracies over time for LLM-based detectors.<n>We argue that a proper factuality evaluation dataset should test a model's ability to reason about current events by retrieving and reading related evidence.
  arXiv  Detail & Related papers  (2024-10-18T17:47:11Z)
• Rethinking the Evaluation of Dialogue Systems: Effects of User Feedback on Crowdworkers and LLMs [57.16442740983528]
  In ad-hoc retrieval, evaluation relies heavily on user actions, including implicit feedback. The role of user feedback in annotators' assessment of turns in a conversational perception has been little studied. We focus on how the evaluation of task-oriented dialogue systems ( TDSs) is affected by considering user feedback, explicit or implicit, as provided through the follow-up utterance of a turn being evaluated.
  arXiv  Detail & Related papers  (2024-04-19T16:45:50Z)
• Exploiting Correlated Auxiliary Feedback in Parameterized Bandits [56.84649080789685]
  We study a novel variant of the parameterized bandits problem in which the learner can observe additional auxiliary feedback that is correlated with the observed reward. The auxiliary feedback is readily available in many real-life applications, e.g., an online platform that wants to recommend the best-rated services to its users can observe the user's rating of service (rewards) and collect additional information like service delivery time (auxiliary feedback)
  arXiv  Detail & Related papers  (2023-11-05T17:27:06Z)
• Click-Feedback Retrieval [10.203235400791845]
  We study a setting where the feedback is provided through users clicking liked and disliked searching results. We construct a new benchmark termed click-feedback retrieval based on a large-scale dataset in fashion domain.
  arXiv  Detail & Related papers  (2023-04-28T19:03:03Z)
• Recall, Robustness, and Lexicographic Evaluation [49.13362412522523]
  The application of recall without a formal evaluative motivation has led to criticism of recall as a vague or inappropriate measure. Our analysis is composed of three tenets: recall, robustness, and lexicographic evaluation.
  arXiv  Detail & Related papers  (2023-02-22T13:39:54Z)
• Semi-supervised Batch Learning From Logged Data [24.826544828460158]
  We build on the counterfactual risk minimization framework, which also assumes access to propensity scores. We propose learning methods for problems where feedback is missing for some samples, so there are samples with feedback and samples missing-feedback in the logged data.
  arXiv  Detail & Related papers  (2022-09-15T08:58:28Z)
• A Dataset on Malicious Paper Bidding in Peer Review [84.68308372858755]
  Malicious reviewers strategically bid in order to unethically manipulate the paper assignment. A critical impediment towards creating and evaluating methods to mitigate this issue is the lack of publicly-available data on malicious paper bidding. We release a novel dataset, collected from a mock conference activity where participants were instructed to bid either honestly or maliciously.
  arXiv  Detail & Related papers  (2022-06-24T20:23:33Z)
• Evaluating the Predictive Performance of Positive-Unlabelled Classifiers: a brief critical review and practical recommendations for improvement [77.34726150561087]
  Positive-Unlabelled (PU) learning is a growing area of machine learning. This paper critically reviews the main PU learning evaluation approaches and the choice of predictive accuracy measures in 51 articles proposing PU classifiers.
  arXiv  Detail & Related papers  (2022-06-06T08:31:49Z)
• LoL: A Comparative Regularization Loss over Query Reformulation Losses for Pseudo-Relevance Feedback [70.44530794897861]
  Pseudo-relevance feedback (PRF) has proven to be an effective query reformulation technique to improve retrieval accuracy. Existing PRF methods independently treat revised queries originating from the same query but using different numbers of feedback documents. We propose the Loss-over-Loss (LoL) framework to compare the reformulation losses between different revisions of the same query during training.
  arXiv  Detail & Related papers  (2022-04-25T10:42:50Z)
• Correcting the User Feedback-Loop Bias for Recommendation Systems [34.44834423714441]
  We propose a systematic and dynamic way to correct user feedback-loop bias in recommendation systems. Our method includes a deep-learning component to learn each user's dynamic rating history embedding. We empirically validated the existence of such user feedback-loop bias in real world recommendation systems.
  arXiv  Detail & Related papers  (2021-09-13T15:02:55Z)
• A Training-free and Reference-free Summarization Evaluation Metric via Centrality-weighted Relevance and Self-referenced Redundancy [60.419107377879925]
  We propose a training-free and reference-free summarization evaluation metric. Our metric consists of a centrality-weighted relevance score and a self-referenced redundancy score. Our methods can significantly outperform existing methods on both multi-document and single-document summarization evaluation.
  arXiv  Detail & Related papers  (2021-06-26T05:11:27Z)
• How Useful are Reviews for Recommendation? A Critical Review and Potential Improvements [8.471274313213092]
  We investigate a growing body of work that seeks to improve recommender systems through the use of review text. Our initial findings reveal several discrepancies in reported results, partly due to copying results across papers despite changes in experimental settings or data pre-processing. Further investigation calls for discussion on a much larger problem about the "importance" of user reviews for recommendation.
  arXiv  Detail & Related papers  (2020-05-25T16:30:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.