Learning Curves for Decision Making in Supervised Machine Learning -- A Survey

• URL: http://arxiv.org/abs/2201.12150v1
• Date: Fri, 28 Jan 2022 14:34:32 GMT
• Title: Learning Curves for Decision Making in Supervised Machine Learning -- A Survey
• Authors: Felix Mohr, Jan N. van Rijn
• Abstract summary: Learning curves are a concept from social sciences that has been adopted in the context of machine learning. We contribute a framework that categorizes learning curve approaches using three criteria: the decision situation that they address, the intrinsic learning curve question that they answer and the type of resources that they use.
• Score: 9.994200032442413
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Learning curves are a concept from social sciences that has been adopted in the context of machine learning to assess the performance of a learning algorithm with respect to a certain resource, e.g. the number of training examples or the number of training iterations. Learning curves have important applications in several contexts of machine learning, most importantly for the context of data acquisition, early stopping of model training and model selection. For example, by modelling the learning curves, one can assess at an early stage whether the algorithm and hyperparameter configuration have the potential to be a suitable choice, often speeding up the algorithm selection process. A variety of approaches has been proposed to use learning curves for decision making. Some models answer the binary decision question of whether a certain algorithm at a certain budget will outperform a certain reference performance, whereas more complex models predict the entire learning curve of an algorithm. We contribute a framework that categorizes learning curve approaches using three criteria: the decision situation that they address, the intrinsic learning curve question that they answer and the type of resources that they use. We survey papers from literature and classify them into this framework.

Related papers

• Meta-Learning from Learning Curves for Budget-Limited Algorithm Selection [11.409496019407067]
  In a budget-limited scenario, it is crucial to carefully select an algorithm candidate and allocate a budget for training it. We propose a novel framework in which an agent must select in the process of learning the most promising algorithm without waiting until it is fully trained.
  arXiv  Detail & Related papers  (2024-10-10T08:09:58Z)
• A Unified Framework for Neural Computation and Learning Over Time [56.44910327178975]
  Hamiltonian Learning is a novel unified framework for learning with neural networks "over time" It is based on differential equations that: (i) can be integrated without the need of external software solvers; (ii) generalize the well-established notion of gradient-based learning in feed-forward and recurrent networks; (iii) open to novel perspectives.
  arXiv  Detail & Related papers  (2024-09-18T14:57:13Z)
• Strategies and impact of learning curve estimation for CNN-based image classification [0.2678472239880052]
  Learning curves are a measure for how the performance of machine learning models improves given a certain volume of training data. Over a wide variety of applications and models it was observed that learning curves follow -- to a large extent -- a power law behavior. By estimating the learning curve of a model from training on small subsets of data only the best models need to be considered for training on the full dataset.
  arXiv  Detail & Related papers  (2023-10-12T16:28:25Z)
• RetICL: Sequential Retrieval of In-Context Examples with Reinforcement Learning [53.52699766206808]
  We propose Retrieval for In-Context Learning (RetICL), a learnable method for modeling and optimally selecting examples sequentially for in-context learning. We evaluate RetICL on math word problem solving and scientific question answering tasks and show that it consistently outperforms or matches and learnable baselines.
  arXiv  Detail & Related papers  (2023-05-23T20:15:56Z)
• Towards Diverse Evaluation of Class Incremental Learning: A Representation Learning Perspective [67.45111837188685]
  Class incremental learning (CIL) algorithms aim to continually learn new object classes from incrementally arriving data. We experimentally analyze neural network models trained by CIL algorithms using various evaluation protocols in representation learning.
  arXiv  Detail & Related papers  (2022-06-16T11:44:11Z)
• Model-Based Deep Learning: On the Intersection of Deep Learning and Optimization [101.32332941117271]
  Decision making algorithms are used in a multitude of different applications. Deep learning approaches that use highly parametric architectures tuned from data without relying on mathematical models are becoming increasingly popular. Model-based optimization and data-centric deep learning are often considered to be distinct disciplines.
  arXiv  Detail & Related papers  (2022-05-05T13:40:08Z)
• Curriculum Learning: A Survey [65.31516318260759]
  Curriculum learning strategies have been successfully employed in all areas of machine learning. We construct a taxonomy of curriculum learning approaches by hand, considering various classification criteria. We build a hierarchical tree of curriculum learning methods using an agglomerative clustering algorithm.
  arXiv  Detail & Related papers  (2021-01-25T20:08:32Z)
• Model-Agnostic Explanations using Minimal Forcing Subsets [11.420687735660097]
  We propose a new model-agnostic algorithm to identify a minimal set of training samples that are indispensable for a given model's decision. Our algorithm identifies such a set of "indispensable" samples iteratively by solving a constrained optimization problem. Results show that our algorithm is an effective and easy-to-comprehend tool that helps to better understand local model behavior.
  arXiv  Detail & Related papers  (2020-11-01T22:45:16Z)
• Learning to Rank Learning Curves [15.976034696758148]
  We present a new method that saves computational budget by terminating poor configurations early on in the training. We show that our model is able to effectively rank learning curves without having to observe many or very long learning curves.
  arXiv  Detail & Related papers  (2020-06-05T10:49:52Z)
• Deep Reinforcement Learning for Adaptive Learning Systems [4.8685842576962095]
  We formulate the problem of how to find an individualized learning plan based on learner's latent traits. We apply a model-free deep reinforcement learning algorithm that can effectively find the optimal learning policy. We also develop a transition model estimator that emulates the learner's learning process using neural networks.
  arXiv  Detail & Related papers  (2020-04-17T18:04:03Z)
• Learning with Differentiable Perturbed Optimizers [54.351317101356614]
  We propose a systematic method to transform operations into operations that are differentiable and never locally constant. Our approach relies on perturbeds, and can be used readily together with existing solvers. We show how this framework can be connected to a family of losses developed in structured prediction, and give theoretical guarantees for their use in learning tasks.
  arXiv  Detail & Related papers  (2020-02-20T11:11:32Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.