Cost-aware LLM-based Online Dataset Annotation

• URL: http://arxiv.org/abs/2505.15101v1
• Date: Wed, 21 May 2025 04:49:44 GMT
• Title: Cost-aware LLM-based Online Dataset Annotation
• Authors: Eray Can Elumar, Cem Tekin, Osman Yagan,
• Abstract summary: CaMVo is an online framework for efficient and accurate dataset annotation.<n>It uses contextual embeddings, balancing confidence and cost without requiring pre-training or ground-truth labels.<n>It achieves comparable or superior accuracy to full majority voting while significantly reducing labeling costs.
• Score: 14.69261466438399
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Recent advances in large language models (LLMs) have enabled automated dataset labeling with minimal human supervision. While majority voting across multiple LLMs can improve label reliability by mitigating individual model biases, it incurs high computational costs due to repeated querying. In this work, we propose a novel online framework, Cost-aware Majority Voting (CaMVo), for efficient and accurate LLM-based dataset annotation. CaMVo adaptively selects a subset of LLMs for each data instance based on contextual embeddings, balancing confidence and cost without requiring pre-training or ground-truth labels. Leveraging a LinUCB-based selection mechanism and a Bayesian estimator over confidence scores, CaMVo estimates a lower bound on labeling accuracy for each LLM and aggregates responses through weighted majority voting. Our empirical evaluation on the MMLU and IMDB Movie Review datasets demonstrates that CaMVo achieves comparable or superior accuracy to full majority voting while significantly reducing labeling costs. This establishes CaMVo as a practical and robust solution for cost-efficient annotation in dynamic labeling environments.

Related papers

• Uncertainty-Aware Answer Selection for Improved Reasoning in Multi-LLM Systems [55.6590601898194]
  Large Language Models (LLMs) have demonstrated exceptional capabilities, yet selecting the most reliable response from multiple LLMs remains a challenge.<n>Existing approaches often depend on costly external verifiers, human evaluators, or self-consistency techniques that require multiple samples from a single model.<n>We propose a principled, novel and computationally efficient method to select the best response from multiple different LLMs using a calibrated log-likelihood score.
  arXiv  Detail & Related papers  (2025-09-30T01:25:19Z)
• LLM on a Budget: Active Knowledge Distillation for Efficient Classification of Large Text Corpora [0.1625256372381793]
  Large Language Models (LLMs) are highly accurate in classification tasks.<n> Knowledge Distillation (KD) where a LLM "teacher" trains a smaller and more efficient "student" model, offers a promising solution to this problem.<n>We introduce M-RARU (Multi-class Randomized Accept/Reject Uncertainty Sampling), a novel AL algorithm that significantly reduces training costs.
  arXiv  Detail & Related papers  (2025-09-17T18:38:56Z)
• Cut Costs, Not Accuracy: LLM-Powered Data Processing with Guarantees [10.940593916080276]
  Large Language Models (LLMs) are being increasingly used as a building block in data systems to process large text datasets.<n>To avoid high costs, more affordable but lower quality LLMs can be used to process records.<n>We present BARGAIN, a method that judiciously uses affordable LLMs in data processing to significantly reduce cost.
  arXiv  Detail & Related papers  (2025-09-02T23:41:50Z)
• Self-ensemble: Mitigating Confidence Distortion for Large Language Models [89.03110940871765]
  Large Language Models exhibit a confidence distortion problem on multi-choice question-answering.<n>We propose Self-ensemble to solve this problem.<n> Experimental results on three LLMs and datasets demonstrate that Self-ensemble comprehensively addresses the confidence distortion problem.
  arXiv  Detail & Related papers  (2025-06-02T17:59:29Z)
• Automatic Transmission for LLM Tiers: Optimizing Cost and Accuracy in Large Language Models [1.3791394805787949]
  We introduce LLM Automatic Transmission (LLM-AT) framework that automatically selects LLM tiers without training.<n>The starter selects the initial LLM tier expected to solve the given question.<n>The generator produces a response using the LLM of the selected tier, and the judge evaluates the validity of the response.
  arXiv  Detail & Related papers  (2025-05-27T09:11:00Z)
• MAPLE: Many-Shot Adaptive Pseudo-Labeling for In-Context Learning [53.02571749383208]
  In-Context Learning (ICL) empowers Large Language Models (LLMs) to tackle diverse tasks by incorporating multiple input-output examples.<n>Many-Shot Adaptive Pseudo-LabEling (MAPLE) is a novel influence-based many-shot ICL framework that utilizes pseudo-labeled samples to compensate for the lack of label information.
  arXiv  Detail & Related papers  (2025-05-22T04:54:27Z)
• LLM-Lasso: A Robust Framework for Domain-Informed Feature Selection and Regularization [59.75242204923353]
  We introduce LLM-Lasso, a framework that leverages large language models (LLMs) to guide feature selection in Lasso regression.<n>LLMs generate penalty factors for each feature, which are converted into weights for the Lasso penalty using a simple, tunable model.<n>Features identified as more relevant by the LLM receive lower penalties, increasing their likelihood of being retained in the final model.
  arXiv  Detail & Related papers  (2025-02-15T02:55:22Z)
• Labels Generated by Large Language Model Helps Measuring People's Empathy in Vitro [9.536979155245026]
  Large language models (LLMs) have revolutionised numerous fields.<n>This paper explores its potential in in-vitro applications.<n>We evaluate this approach in the emerging field of empathy computing.
  arXiv  Detail & Related papers  (2025-01-01T01:06:58Z)
• LLM-Forest: Ensemble Learning of LLMs with Graph-Augmented Prompts for Data Imputation [50.375567142250446]
  Large language models (LLMs), trained on vast corpora, have shown strong potential in data generation.<n>We propose a novel framework, LLM-Forest, which introduces a "forest" of few-shot prompt learning LLM "trees" with their outputs aggregated via confidence-based weighted voting.<n>This framework is established on a new concept of bipartite information graphs to identify high-quality relevant neighboring entries with both feature and value granularity.
  arXiv  Detail & Related papers  (2024-10-28T20:42:46Z)
• Are LLMs Better than Reported? Detecting Label Errors and Mitigating Their Effect on Model Performance [21.926934384262594]
  Large language models (LLMs) offer new opportunities to enhance the annotation process. We compare expert, crowd-sourced, and our LLM-based annotations in terms of agreement, label quality, and efficiency. Our findings reveal a substantial number of label errors, which, when corrected, induce a significant upward shift in reported model performance.
  arXiv  Detail & Related papers  (2024-10-24T16:27:03Z)
• MetaLLM: A High-performant and Cost-efficient Dynamic Framework for Wrapping LLMs [20.793892860721712]
  We introduce MetaLLM, a framework that dynamically and intelligently routes each query to the optimal large language models (LLMs)<n>By framing the selection problem as a multi-armed bandit, MetaLLM balances prediction accuracy and cost efficiency under uncertainty.<n>Our experiments, conducted on popular LLM platforms such as OpenAI and Together AI, showcase MetaLLM's efficacy in real-world scenarios.
  arXiv  Detail & Related papers  (2024-07-15T15:45:07Z)
• OptLLM: Optimal Assignment of Queries to Large Language Models [12.07164196530872]
  We propose a framework for addressing the cost-effective query allocation problem for large language models (LLMs) Our framework, named OptLLM, provides users with a range of optimal solutions to choose from, aligning with their budget constraints and performance preferences. To evaluate the effectiveness of OptLLM, we conduct extensive experiments on various types of tasks, including text classification, question answering, sentiment analysis, reasoning, and log parsing.
  arXiv  Detail & Related papers  (2024-05-24T01:05:37Z)
• RepEval: Effective Text Evaluation with LLM Representation [55.26340302485898]
  RepEval is a metric that leverages the projection of Large Language Models (LLMs) representations for evaluation. Our work underscores the richness of information regarding text quality embedded within LLM representations, offering insights for the development of new metrics.
  arXiv  Detail & Related papers  (2024-04-30T13:50:55Z)
• SMART: Automatically Scaling Down Language Models with Accuracy Guarantees for Reduced Processing Fees [21.801053526411415]
  Large Language Models (LLMs) have significantly boosted performance in natural language processing (NLP) tasks. The deployment of high-performance LLMs incurs substantial costs, primarily due to the increased number of parameters aimed at enhancing model performance. We introduce SMART, a novel framework designed to minimize the inference costs of NLP tasks while ensuring sufficient result quality.
  arXiv  Detail & Related papers  (2024-03-11T17:45:47Z)
• Self-Evaluation Improves Selective Generation in Large Language Models [54.003992911447696]
  We reformulate open-ended generation tasks into token-level prediction tasks. We instruct an LLM to self-evaluate its answers. We benchmark a range of scoring methods based on self-evaluation.
  arXiv  Detail & Related papers  (2023-12-14T19:09:22Z)
• From Quantity to Quality: Boosting LLM Performance with Self-Guided Data Selection for Instruction Tuning [52.257422715393574]
  We introduce a self-guided methodology for Large Language Models (LLMs) to autonomously discern and select cherry samples from open-source datasets. Our key innovation, the Instruction-Following Difficulty (IFD) metric, emerges as a pivotal metric to identify discrepancies between a model's expected responses and its intrinsic generation capability.
  arXiv  Detail & Related papers  (2023-08-23T09:45:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.