Adaptive Fine-Tuning of Transformer-Based Language Models for Named Entity Recognition

• URL: http://arxiv.org/abs/2202.02617v1
• Date: Sat, 5 Feb 2022 19:20:03 GMT
• Title: Adaptive Fine-Tuning of Transformer-Based Language Models for Named Entity Recognition
• Authors: Felix Stollenwerk
• Abstract summary: The current standard approach for fine-tuning language models includes a fixed number of training epochs and a linear learning rate schedule. In this paper, we introduce adaptive fine-tuning, which is an alternative approach that uses early stopping and a custom learning rate schedule.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The current standard approach for fine-tuning transformer-based language models includes a fixed number of training epochs and a linear learning rate schedule. In order to obtain a near-optimal model for the given downstream task, a search in optimization hyperparameter space is usually required. In particular, the number of training epochs needs to be adjusted to the dataset size. In this paper, we introduce adaptive fine-tuning, which is an alternative approach that uses early stopping and a custom learning rate schedule to dynamically adjust the number of training epochs to the dataset size. For the example use case of named entity recognition, we show that our approach not only makes hyperparameter search with respect to the number of training epochs redundant, but also leads to improved results in terms of performance, stability and efficiency. This holds true especially for small datasets, where we outperform the state-of-the-art fine-tuning method by a large margin.

Related papers

• Adaptive Data Optimization: Dynamic Sample Selection with Scaling Laws [59.03420759554073]
  We introduce Adaptive Data Optimization (ADO), an algorithm that optimize data distributions in an online fashion, concurrent with model training. ADO does not require external knowledge, proxy models, or modifications to the model update. ADO uses per-domain scaling laws to estimate the learning potential of each domain during training and adjusts the data mixture accordingly.
  arXiv  Detail & Related papers  (2024-10-15T17:47:44Z)
• Towards An Online Incremental Approach to Predict Students Performance [0.8287206589886879]
  We propose a memory-based online incremental learning approach for updating an online classifier. Our approach achieves a notable improvement in model accuracy, with an enhancement of nearly 10% compared to the current state-of-the-art.
  arXiv  Detail & Related papers  (2024-05-03T17:13:26Z)
• TextGram: Towards a better domain-adaptive pretraining [0.3769303106863454]
  In NLP, pre-training involves using a large amount of text data to gain prior knowledge for performing downstream tasks. We propose our own domain-adaptive data selection method - TextGram. We show that the proposed strategy works better compared to other selection methods.
  arXiv  Detail & Related papers  (2024-04-28T15:44:57Z)
• Adaptive scheduling for adaptive sampling in POS taggers construction [0.27624021966289597]
  We introduce an adaptive scheduling for adaptive sampling as a novel way of machine learning in the construction of part-of-speech taggers. We analyze the shape of the learning curve geometrically in conjunction with a functional model to increase or decrease it at any time. We also improve the robustness of sampling by paying greater attention to those regions of the training data base subject to a temporary inflation in performance.
  arXiv  Detail & Related papers  (2024-02-04T15:02:17Z)
• Navigating Scaling Laws: Compute Optimality in Adaptive Model Training [39.96209967632896]
  In recent years, the state-of-the-art in deep learning has been dominated by very large models that have been pre-trained on vast amounts of data. We extend the concept of optimality by allowing for an adaptive' model, i.e. a model that can change its shape during training.
  arXiv  Detail & Related papers  (2023-11-06T16:20:28Z)
• Quick-Tune: Quickly Learning Which Pretrained Model to Finetune and How [62.467716468917224]
  We propose a methodology that jointly searches for the optimal pretrained model and the hyperparameters for finetuning it. Our method transfers knowledge about the performance of many pretrained models on a series of datasets. We empirically demonstrate that our resulting approach can quickly select an accurate pretrained model for a new dataset.
  arXiv  Detail & Related papers  (2023-06-06T16:15:26Z)
• Active Finetuning: Exploiting Annotation Budget in the Pretraining-Finetuning Paradigm [132.9949120482274]
  This paper focuses on the selection of samples for annotation in the pretraining-finetuning paradigm. We propose a novel method called ActiveFT for active finetuning task to select a subset of data distributing similarly with the entire unlabeled pool. Extensive experiments show the leading performance and high efficiency of ActiveFT superior to baselines on both image classification and semantic segmentation.
  arXiv  Detail & Related papers  (2023-03-25T07:17:03Z)
• Automatic Tuning of Stochastic Gradient Descent with Bayesian Optimisation [8.340191147575307]
  We introduce an original probabilistic model for traces of optimisers, based on latent Gaussian processes and an auto-/regressive formulation. It flexibly adjusts to abrupt changes of behaviours induced by new learning rate values. It is well-suited to tackle a set of problems: first, for the on-line adaptation of the learning rate for a cold-started run; then, for tuning the schedule for a set of similar tasks, as well as warm-starting it for a new task.
  arXiv  Detail & Related papers  (2020-06-25T13:18:18Z)
• Extrapolation for Large-batch Training in Deep Learning [72.61259487233214]
  We show that a host of variations can be covered in a unified framework that we propose. We prove the convergence of this novel scheme and rigorously evaluate its empirical performance on ResNet, LSTM, and Transformer.
  arXiv  Detail & Related papers  (2020-06-10T08:22:41Z)
• Don't Stop Pretraining: Adapt Language Models to Domains and Tasks [81.99843216550306]
  We present a study across four domains (biomedical and computer science publications, news, and reviews) and eight classification tasks. A second phase of pretraining in-domain (domain-adaptive pretraining) leads to performance gains. Adapting to the task's unlabeled data (task-adaptive pretraining) improves performance even after domain-adaptive pretraining.
  arXiv  Detail & Related papers  (2020-04-23T04:21:19Z)
• Tracking Performance of Online Stochastic Learners [57.14673504239551]
  Online algorithms are popular in large-scale learning settings due to their ability to compute updates on the fly, without the need to store and process data in large batches. When a constant step-size is used, these algorithms also have the ability to adapt to drifts in problem parameters, such as data or model properties, and track the optimal solution with reasonable accuracy. We establish a link between steady-state performance derived under stationarity assumptions and the tracking performance of online learners under random walk models.
  arXiv  Detail & Related papers  (2020-04-04T14:16:27Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.