Streamlining Conformal Information Retrieval via Score Refinement

• URL: http://arxiv.org/abs/2410.02914v1
• Date: Thu, 3 Oct 2024 19:05:47 GMT
• Title: Streamlining Conformal Information Retrieval via Score Refinement
• Authors: Yotam Intrator, Ori Kelner, Regev Cohen, Roman Goldenberg, Ehud Rivlin, Daniel Freedman,
• Abstract summary: We introduce a score refinement method that applies a simple monotone transformation to retrieval scores, leading to significantly smaller conformal sets. Experiments on various BEIR benchmarks validate the effectiveness of our approach in producing compact sets.
• Score: 11.220373666664468
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Information retrieval (IR) methods, like retrieval augmented generation, are fundamental to modern applications but often lack statistical guarantees. Conformal prediction addresses this by retrieving sets guaranteed to include relevant information, yet existing approaches produce large-sized sets, incurring high computational costs and slow response times. In this work, we introduce a score refinement method that applies a simple monotone transformation to retrieval scores, leading to significantly smaller conformal sets while maintaining their statistical guarantees. Experiments on various BEIR benchmarks validate the effectiveness of our approach in producing compact sets containing relevant information.

Related papers

• CREAM: Continual Retrieval on Dynamic Streaming Corpora with Adaptive Soft Memory [19.64051996386645]
  CREAM is a self-supervised framework for memory-based continual retrieval.<n>It adapts to both seen and unseen topics in an unsupervised setting.<n> Experiments on two benchmark datasets demonstrate that CREAM exhibits superior adaptability and retrieval accuracy.
  arXiv  Detail & Related papers  (2026-01-06T04:47:49Z)
• AcuRank: Uncertainty-Aware Adaptive Computation for Listwise Reranking [25.459771464139855]
  Listwise reranking with large language models (LLMs) enhances top-ranked results in retrieval-based applications.<n>We propose AcuRank, an adaptive reranking framework that dynamically adjusts both the amount and target of computation based on uncertainty estimates over document relevance.<n>Results on the TREC-DL and BEIR benchmarks show that our method consistently achieves a superior accuracy-efficiency trade-off and scales better with compute than fixed-computation baselines.
  arXiv  Detail & Related papers  (2025-05-24T05:15:49Z)
• Breaking the Lens of the Telescope: Online Relevance Estimation over Large Retrieval Sets [15.549852480638066]
  We propose a novel paradigm for re-ranking called online relevance estimation. Online relevance estimation continuously updates relevance estimates for a query throughout the ranking process. We validate our approach on TREC benchmarks under two scenarios: hybrid retrieval and adaptive retrieval.
  arXiv  Detail & Related papers  (2025-04-12T22:05:50Z)
• Lightweight and Direct Document Relevance Optimization for Generative Information Retrieval [49.669503570350166]
  Generative information retrieval (GenIR) is a promising neural retrieval paradigm that formulates document retrieval as a document identifier (docid) generation task. Existing GenIR models suffer from token-level misalignment, where models trained to predict the next token often fail to capture document-level relevance effectively. We propose direct document relevance optimization (DDRO), which aligns token-level docid generation with document-level relevance estimation through direct optimization via pairwise ranking.
  arXiv  Detail & Related papers  (2025-04-07T15:27:37Z)
• Chain-of-Retrieval Augmented Generation [72.06205327186069]
  This paper introduces an approach for training o1-like RAG models that retrieve and reason over relevant information step by step before generating the final answer. Our proposed method, CoRAG, allows the model to dynamically reformulate the query based on the evolving state.
  arXiv  Detail & Related papers  (2025-01-24T09:12:52Z)
• Retriever-and-Memory: Towards Adaptive Note-Enhanced Retrieval-Augmented Generation [72.70046559930555]
  We propose a generic RAG approach called Adaptive Note-Enhanced RAG (Adaptive-Note) for complex QA tasks. Specifically, Adaptive-Note introduces an overarching view of knowledge growth, iteratively gathering new information in the form of notes. In addition, we employ an adaptive, note-based stop-exploration strategy to decide "what to retrieve and when to stop" to encourage sufficient knowledge exploration.
  arXiv  Detail & Related papers  (2024-10-11T14:03:29Z)
• GenCRF: Generative Clustering and Reformulation Framework for Enhanced Intent-Driven Information Retrieval [20.807374287510623]
  We propose GenCRF: a Generative Clustering and Reformulation Framework to capture diverse intentions adaptively. We show that GenCRF achieves state-of-the-art performance, surpassing previous query reformulation SOTAs by up to 12% on nDCG@10.
  arXiv  Detail & Related papers  (2024-09-17T05:59:32Z)
• Exploring Information Retrieval Landscapes: An Investigation of a Novel Evaluation Techniques and Comparative Document Splitting Methods [0.0]
  In this study, the structured nature of textbooks, the conciseness of articles, and the narrative complexity of novels are shown to require distinct retrieval strategies. A novel evaluation technique is introduced, utilizing an open-source model to generate a comprehensive dataset of question-and-answer pairs. The evaluation employs weighted scoring metrics, including SequenceMatcher, BLEU, METEOR, and BERT Score, to assess the system's accuracy and relevance.
  arXiv  Detail & Related papers  (2024-09-13T02:08:47Z)
• Concurrent Brainstorming & Hypothesis Satisfying: An Iterative Framework for Enhanced Retrieval-Augmented Generation (R2CBR3H-SR) [0.456877715768796]
  This study introduces an innovative, iterative retrieval-augmented generation system. Our approach uniquely integrates a vector-space driven re-ranking mechanism with concurrent brainstorming to expedite the retrieval of highly relevant documents. This research advances the state-of-the-art in intelligent retrieval systems, setting a new benchmark for resource-efficient information extraction and abstraction in knowledge-intensive applications.
  arXiv  Detail & Related papers  (2024-01-03T17:01:44Z)
• Noisy Correspondence Learning with Self-Reinforcing Errors Mitigation [63.180725016463974]
  Cross-modal retrieval relies on well-matched large-scale datasets that are laborious in practice. We introduce a novel noisy correspondence learning framework, namely textbfSelf-textbfReinforcing textbfErrors textbfMitigation (SREM)
  arXiv  Detail & Related papers  (2023-12-27T09:03:43Z)
• Large-scale Fully-Unsupervised Re-Identification [78.47108158030213]
  We propose two strategies to learn from large-scale unlabeled data. The first strategy performs a local neighborhood sampling to reduce the dataset size in each without violating neighborhood relationships. A second strategy leverages a novel Re-Ranking technique, which has a lower time upper bound complexity and reduces the memory complexity from O(n2) to O(kn) with k n.
  arXiv  Detail & Related papers  (2023-07-26T16:19:19Z)
• A robust method for reliability updating with equality information using sequential adaptive importance sampling [8.254850675268957]
  Reliability updating refers to a problem that integrates Bayesian updating technique with structural reliability analysis. This paper proposes an innovative method called RU-SAIS, which combines elements of sequential importance sampling and K-means clustering. Results show that RU-SAIS achieves a more accurate and robust estimator of the posterior failure probability than the existing methods.
  arXiv  Detail & Related papers  (2023-03-08T12:55:40Z)
• Conformal Frequency Estimation using Discrete Sketched Data with Coverage for Distinct Queries [35.67445122503686]
  This paper develops conformal inference methods to construct a confidence interval for the frequency of a queried object in a very large discrete data set. We show our methods have improved empirical performance compared to existing frequentist and Bayesian alternatives in simulations.
  arXiv  Detail & Related papers  (2022-11-09T00:05:29Z)
• Distributed Dynamic Safe Screening Algorithms for Sparse Regularization [73.85961005970222]
  We propose a new distributed dynamic safe screening (DDSS) method for sparsity regularized models and apply it on shared-memory and distributed-memory architecture respectively. We prove that the proposed method achieves the linear convergence rate with lower overall complexity and can eliminate almost all the inactive features in a finite number of iterations almost surely.
  arXiv  Detail & Related papers  (2022-04-23T02:45:55Z)
• Self-Certifying Classification by Linearized Deep Assignment [65.0100925582087]
  We propose a novel class of deep predictors for classifying metric data on graphs within PAC-Bayes risk certification paradigm. Building on the recent PAC-Bayes literature and data-dependent priors, this approach enables learning posterior distributions on the hypothesis space.
  arXiv  Detail & Related papers  (2022-01-26T19:59:14Z)
• Risk Minimization from Adaptively Collected Data: Guarantees for Supervised and Policy Learning [57.88785630755165]
  Empirical risk minimization (ERM) is the workhorse of machine learning, but its model-agnostic guarantees can fail when we use adaptively collected data. We study a generic importance sampling weighted ERM algorithm for using adaptively collected data to minimize the average of a loss function over a hypothesis class. For policy learning, we provide rate-optimal regret guarantees that close an open gap in the existing literature whenever exploration decays to zero.
  arXiv  Detail & Related papers  (2021-06-03T09:50:13Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.