Personalized Top-k Set Queries Over Predicted Scores

• URL: http://arxiv.org/abs/2502.12998v1
• Date: Tue, 18 Feb 2025 16:19:08 GMT
• Title: Personalized Top-k Set Queries Over Predicted Scores
• Authors: Sohrab Namazi Nia, Subhodeep Ghosh, Senjuti Basu Roy, Sihem Amer-Yahia,
• Abstract summary: This work studies the applicability of expensive external oracles in answering top-k queries over predicted scores.<n>We propose a generic computational framework that handles arbitrary set-based scoring functions.
• Score: 21.74740893966611
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: This work studies the applicability of expensive external oracles such as large language models in answering top-k queries over predicted scores. Such scores are incurred by user-defined functions to answer personalized queries over multi-modal data. We propose a generic computational framework that handles arbitrary set-based scoring functions, as long as the functions could be decomposed into constructs, each of which sent to an oracle (in our case an LLM) to predict partial scores. At a given point in time, the framework assumes a set of responses and their partial predicted scores, and it maintains a collection of possible sets that are likely to be the true top-k. Since calling oracles is costly, our framework judiciously identifies the next construct, i.e., the next best question to ask the oracle so as to maximize the likelihood of identifying the true top-k. We present a principled probabilistic model that quantifies that likelihood. We study efficiency opportunities in designing algorithms. We run an evaluation with three large scale datasets, scoring functions, and baselines. Experiments indicate the efficacy of our framework, as it achieves an order of magnitude improvement over baselines in requiring LLM calls while ensuring result accuracy. Scalability experiments further indicate that our framework could be used in large-scale applications.

Related papers

• Reinforcing Compositional Retrieval: Retrieving Step-by-Step for Composing Informative Contexts [67.67746334493302]
  Large Language Models (LLMs) have demonstrated remarkable capabilities across numerous tasks, yet they often rely on external context to handle complex tasks. We propose a tri-encoder sequential retriever that models this process as a Markov Decision Process (MDP) We show that our method consistently and significantly outperforms baselines, underscoring the importance of explicitly modeling inter-example dependencies.
  arXiv  Detail & Related papers  (2025-04-15T17:35:56Z)
• Universal Scalability in Declarative Program Analysis (with Choice-Based Combination Pruning) [1.3874486202578669]
  We show a near-universal construction that allows the choice construct to flexibly limit evaluation of predicates. We apply the technique to probably the largest, pre-existing Datalog analysis frameworks in existence: Doop (for Javacode) and the main client analyses from the Gigahorse framework.
  arXiv  Detail & Related papers  (2025-03-07T21:23:02Z)
• Large Language Model Can Be a Foundation for Hidden Rationale-Based Retrieval [12.83513794686623]
  In this paper, we propose and study a more challenging type of retrieval task, called hidden rationale retrieval.<n>To address such problems, an instruction-tuned Large language model (LLM) with a cross-encoder architecture could be a reasonable choice.<n>We name this retrieval framework by RaHoRe and verify its zero-shot and fine-tuned performance superiority on Emotional Support Conversation (ESC)
  arXiv  Detail & Related papers  (2024-12-21T13:19:15Z)
• LLM-Select: Feature Selection with Large Language Models [64.5099482021597]
  Large language models (LLMs) are capable of selecting the most predictive features, with performance rivaling the standard tools of data science. Our findings suggest that LLMs may be useful not only for selecting the best features for training but also for deciding which features to collect in the first place.
  arXiv  Detail & Related papers  (2024-07-02T22:23:40Z)
• Optimized Feature Generation for Tabular Data via LLMs with Decision Tree Reasoning [53.241569810013836]
  We propose a novel framework that utilizes large language models (LLMs) to identify effective feature generation rules. We use decision trees to convey this reasoning information, as they can be easily represented in natural language. OCTree consistently enhances the performance of various prediction models across diverse benchmarks.
  arXiv  Detail & Related papers  (2024-06-12T08:31:34Z)
• ERBench: An Entity-Relationship based Automatically Verifiable Hallucination Benchmark for Large Language Models [46.07900122810749]
  Large language models (LLMs) have achieved unprecedented performances in various applications, yet evaluating them is still challenging. We contend that utilizing existing relational databases is a promising approach for constructing benchmarks. We propose ERBench, which uses these integrity constraints to convert any database into an LLM benchmark.
  arXiv  Detail & Related papers  (2024-03-08T12:42:36Z)
• MinPrompt: Graph-based Minimal Prompt Data Augmentation for Few-shot Question Answering [64.6741991162092]
  We present MinPrompt, a minimal data augmentation framework for open-domain question answering. We transform the raw text into a graph structure to build connections between different factual sentences. We then apply graph algorithms to identify the minimal set of sentences needed to cover the most information in the raw text. We generate QA pairs based on the identified sentence subset and train the model on the selected sentences to obtain the final model.
  arXiv  Detail & Related papers  (2023-10-08T04:44:36Z)
• How Predictable Are Large Language Model Capabilities? A Case Study on BIG-bench [52.11481619456093]
  We study the performance prediction problem on experiment records from BIG-bench. An $R2$ score greater than 95% indicates the presence of learnable patterns within the experiment records. We find a subset as informative as BIG-bench Hard for evaluating new model families, while being $3times$ smaller.
  arXiv  Detail & Related papers  (2023-05-24T09:35:34Z)
• Variable Importance Matching for Causal Inference [73.25504313552516]
  We describe a general framework called Model-to-Match that achieves these goals. Model-to-Match uses variable importance measurements to construct a distance metric. We operationalize the Model-to-Match framework with LASSO.
  arXiv  Detail & Related papers  (2023-02-23T00:43:03Z)
• Interpretable by Design: Learning Predictors by Composing Interpretable Queries [8.054701719767293]
  We argue that machine learning algorithms should be interpretable by design. We minimize the expected number of queries needed for accurate prediction. Experiments on vision and NLP tasks demonstrate the efficacy of our approach.
  arXiv  Detail & Related papers  (2022-07-03T02:40:34Z)
• On Efficient Approximate Queries over Machine Learning Models [30.26180913049285]
  We develop a novel unified framework for approximate query answering by leveraging a proxy to minimize the oracle usage. Our framework uses a judicious combination of invoking the expensive oracle on data samples and applying the cheap proxy on the objects in the DB. Our algorithms outperform the state-of-the-art and achieve high result quality with provable statistical guarantees.
  arXiv  Detail & Related papers  (2022-06-06T18:35:19Z)
• Test Score Algorithms for Budgeted Stochastic Utility Maximization [12.360522095604983]
  We extend an existing scoring mechanism, namely the replication test scores, to incorporate heterogeneous item costs as well as item values. Our algorithms and approximation guarantees assume that test scores are noisy estimates of certain expected values. We show how our algorithm can be adapted to the setting where items arrive in a fashion while maintaining the same approximation guarantee.
  arXiv  Detail & Related papers  (2020-12-30T15:28:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.