Pre-train or Annotate? Domain Adaptation with a Constrained Budget

• URL: http://arxiv.org/abs/2109.04711v1
• Date: Fri, 10 Sep 2021 07:28:26 GMT
• Title: Pre-train or Annotate? Domain Adaptation with a Constrained Budget
• Abstract summary: Given a fixed budget, what steps should an NLP practitioner take to maximize performance? We measure the annotation cost of three procedural text datasets and the pre-training cost of three in-domain language models. We find that, for small budgets, spending all funds on annotation leads to the best performance.
• Score: 25.44621972274297
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Recent work has demonstrated that pre-training in-domain language models can boost performance when adapting to a new domain. However, the costs associated with pre-training raise an important question: given a fixed budget, what steps should an NLP practitioner take to maximize performance? In this paper, we study domain adaptation under budget constraints, and approach it as a customer choice problem between data annotation and pre-training. Specifically, we measure the annotation cost of three procedural text datasets and the pre-training cost of three in-domain language models. Then we evaluate the utility of different combinations of pre-training and data annotation under varying budget constraints to assess which combination strategy works best. We find that, for small budgets, spending all funds on annotation leads to the best performance; once the budget becomes large enough, a combination of data annotation and in-domain pre-training works more optimally. We therefore suggest that task-specific data annotation should be part of an economical strategy when adapting an NLP model to a new domain.

Related papers

• Learning to Fine-tune Foundation Models under Resource Limitations [2.7071541526963805]
  We study the problem of optimal continual fine-tuning for a pre-trained Foundation Model deployed at a resource-limited device.<n>Our objective is to learn an optimal policy that determines emphwhen to fine-tune the model under a finite compute budget.<n> Experiments with a large pre-trained model on a widely-used text classification dataset demonstrate that our method consistently outperforms fine-tuning approaches with the same compute budget.
  arXiv  Detail & Related papers  (2026-07-12T10:32:32Z)
• Cost-Aware Learning [72.31444819326795]
  We consider the problem of Cost-Aware Learning, where sampling different component functions incur different costs.<n>We propose the Cost-Aware Descent algorithm for convex functions, and derive its cost complexity to attain an error of $$.<n>We introduce Cost-Aware GRPO, an algorithm designed to reduce the cost of policy optimization while preserving performance.
  arXiv  Detail & Related papers  (2026-04-30T15:39:09Z)
• Labels or Preferences? Budget-Constrained Learning with Human Judgments over AI-Generated Outputs [17.028710603629026]
  We show how to optimally allocate a fixed annotation budget between ground-truth labels and pairwise preferences in AI.<n>We introduce Preference-Calibrated Active Learning (PCAL), a novel robustness method that learns optimal data acquisition strategy.<n>This work provides a principled and statistically efficient approach for budget-constrained learning in modern AI.
  arXiv  Detail & Related papers  (2026-01-19T23:23:29Z)
• Computational Budget Should Be Considered in Data Selection [21.598075666695483]
  We argue that compute budget must be integral to data-selection strategies.<n>We propose a novel Computational budget-Aware Data Selection (CADS) method.<n>Our method achieves performance gains of up to 14.42% over baselines in vision and language benchmarks.
  arXiv  Detail & Related papers  (2025-10-19T12:16:43Z)
• Pre-training under infinite compute [87.02472603429936]
  We show that data-constrained approaches of increasing epoch count and parameter count eventually overfit.<n>We identify that ensembling independently trained models achieves a significantly lower loss asymptote than the regularized recipe.<n>Our results show that simple algorithmic improvements can enable significantly more data-efficient pre-training in a compute-rich future.
  arXiv  Detail & Related papers  (2025-09-18T09:36:23Z)
• Stepsize anything: A unified learning rate schedule for budgeted-iteration training [43.52874155421866]
  Budgeted-iteration training aims to achieve optimal learning within predetermined budgets.<n>While learning rate schedules govern the performance of different networks and tasks, their design remains largely lacking theoretical foundations.<n>We propose the Unified Budget-Aware (UBA) schedule, a theoretically grounded learning rate schedule that consistently outperforms commonly-used schedules.
  arXiv  Detail & Related papers  (2025-05-30T10:38:03Z)
• Balancing the Budget: Understanding Trade-offs Between Supervised and Preference-Based Finetuning [18.381178799923514]
  Post-training of Large Language Models often involves a pipeline of Supervised Finetuning (SFT) followed by Preference Finetuning (PFT) We study how to optimally allocate a fixed training data budget between the two stages.
  arXiv  Detail & Related papers  (2025-02-16T21:57:35Z)
• Compute-Constrained Data Selection [77.06528009072967]
  We find that many powerful data selection methods are almost never compute-optimal. For compute-optimal training, we find that perplexity and gradient data selection require training-to-selection model size ratios of 5x and 10x, respectively.
  arXiv  Detail & Related papers  (2024-10-21T17:11:21Z)
• Rational Metareasoning for Large Language Models [5.5539136805232205]
  Being prompted to engage in reasoning has emerged as a core technique for using large language models (LLMs) This work introduces a novel approach based on computational models of metareasoning used in cognitive science. We develop a reward function that incorporates the Value of Computation by penalizing unnecessary reasoning.
  arXiv  Detail & Related papers  (2024-10-07T23:48:52Z)
• In-Context Learning on a Budget: A Case Study in Token Classification [16.664472010838132]
  We study various methods for selecting samples to annotate within a predefined budget. We observe that no method significantly outperforms the others, with most yielding similar results. We demonstrate that a relatively small annotated sample pool can achieve performance comparable to using the entire training set.
  arXiv  Detail & Related papers  (2024-06-19T07:09:46Z)
• Metalearners for Ranking Treatment Effects [1.469168639465869]
  We show how learning to rank can maximize the area under a policy's incremental profit curve. We show how learning to rank can maximize the area under a policy's incremental profit curve.
  arXiv  Detail & Related papers  (2024-05-03T15:31:18Z)
• On the Necessity of Collaboration for Online Model Selection with Decentralized Data [53.244188985271606]
  We consider online model selection with decentralized data over $M$ clients, and study the necessity of collaboration among clients. Our results show (i) collaboration is unnecessary in the absence of computational constraints on clients; (ii) collaboration is necessary if the computational cost on each client is limited to $o(K)$, where $K$ is the number of candidate hypothesis spaces.
  arXiv  Detail & Related papers  (2024-04-15T06:32:28Z)
• How Much Data are Enough? Investigating Dataset Requirements for Patch-Based Brain MRI Segmentation Tasks [74.21484375019334]
  Training deep neural networks reliably requires access to large-scale datasets. To mitigate both the time and financial costs associated with model development, a clear understanding of the amount of data required to train a satisfactory model is crucial. This paper proposes a strategic framework for estimating the amount of annotated data required to train patch-based segmentation networks.
  arXiv  Detail & Related papers  (2024-04-04T13:55:06Z)
• Full or Weak annotations? An adaptive strategy for budget-constrained annotation campaigns [3.1318537187387787]
  We propose a novel approach to determine annotation strategies for segmentation datasets. Our method sequentially determines proportions of segmentation and classification annotations to collect for budget-fractions. We show in our experiments that our approach yields annotations that perform very close to the optimal for a number of different annotation budgets and datasets.
  arXiv  Detail & Related papers  (2023-03-21T08:41:54Z)
• The Power and Limitation of Pretraining-Finetuning for Linear Regression under Covariate Shift [127.21287240963859]
  We investigate a transfer learning approach with pretraining on the source data and finetuning based on the target data. For a large class of linear regression instances, transfer learning with $O(N2)$ source data is as effective as supervised learning with $N$ target data.
  arXiv  Detail & Related papers  (2022-08-03T05:59:49Z)
• How to distribute data across tasks for meta-learning? [59.608652082495624]
  We show that the optimal number of data points per task depends on the budget, but it converges to a unique constant value for large budgets. Our results suggest a simple and efficient procedure for data collection.
  arXiv  Detail & Related papers  (2021-03-15T15:38:47Z)
• Don't Stop Pretraining: Adapt Language Models to Domains and Tasks [81.99843216550306]
  We present a study across four domains (biomedical and computer science publications, news, and reviews) and eight classification tasks. A second phase of pretraining in-domain (domain-adaptive pretraining) leads to performance gains. Adapting to the task's unlabeled data (task-adaptive pretraining) improves performance even after domain-adaptive pretraining.
  arXiv  Detail & Related papers  (2020-04-23T04:21:19Z)
• Practical Annotation Strategies for Question Answering Datasets [44.405498237885354]
  We develop rules for annotating a subset of questions, so that the annotation cost is reduced while maintaining both in- and out-of-domain performance. Our work fulfills a demand in practice when labeling budgets are limited and where thus recommendations are needed for annotating QA datasets more cost-effectively.
  arXiv  Detail & Related papers  (2020-03-06T14:25:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.