Predicting Large Model Test Losses with a Noisy Quadratic System

• URL: http://arxiv.org/abs/2605.09154v1
• Date: Sat, 09 May 2026 20:35:09 GMT
• Title: Predicting Large Model Test Losses with a Noisy Quadratic System
• Abstract summary: We introduce a predictive model that estimates the pre-training loss of large models from model size (N), batch size (B) and number of weight updates (K)<n>This is the first loss prediction model that can handle changing batch size.
• Score: 18.22919317870566
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We introduce a predictive model that estimates the pre-training loss of large models from model size (N), batch size (B) and number of weight updates (K). This is the first loss prediction model that can handle changing batch size. The model outperforms Chinchilla's loss model, a model of the test loss using the batch size and number of tokens, in terms of projecting the loss at extrapolated compute budgets (up to 1000 folds). A natural use of the model is to find optimal N, B, K configurations under explicit and compound resource constraints like time, memory and compute. In our experiments, the model-selected configurations are close to ground-truth optimal. Our work advocates for loss prediction as a better alternative to heuristic-based laws, which are growing in complexity. The implementation is available on https://github.com/chuningxdy/Noisy-Quadratic-System.

Related papers

• Data-Constrained Language Model Pretraining: Improved Regularization and Scaling Laws [41.26396818761427]
  We study data-constrained pretraining along two axes, regularization and scaling.<n>We propose SoftQ, a scaling law that couples model size and data size to capture their interaction under repeated data.
  arXiv  Detail & Related papers  (2026-06-05T04:10:09Z)
• MERGETUNE: Continued fine-tuning of vision-language models [77.8627788911249]
  Fine-tuning vision-language models (VLMs) often leads to catastrophic forgetting of pretrained knowledge.<n>We introduce a novel paradigm, continued fine-tuning (CFT), which seeks to recover pretrained knowledge after a zero-shot model has already been adapted.
  arXiv  Detail & Related papers  (2026-01-15T15:15:53Z)
• Establishing Task Scaling Laws via Compute-Efficient Model Ladders [136.76316239300363]
  We develop task scaling laws and model ladders to predict the individual task performance of pretrained language models (LMs) in the overtrained setting.<n>We train a set of small-scale "ladder" models, collect data points to fit the parameterized functions of the two prediction steps, and make predictions for two target models.<n>On four multiple-choice tasks formatted as ranked classification, we can predict the accuracy of both target models within 2 points of absolute error.
  arXiv  Detail & Related papers  (2024-12-05T18:21:49Z)
• Coarse Graining with Neural Operators for Simulating Chaotic Systems [78.64101336150419]
  Predicting the long-term behavior of chaotic systems is crucial for various applications such as climate modeling.<n>An alternative approach to such a full-resolved simulation is using a coarse grid and then correcting its errors through a temporalittext model.<n>We propose an alternative end-to-end learning approach using a physics-informed neural operator (PINO) that overcomes this limitation.
  arXiv  Detail & Related papers  (2024-08-09T17:05:45Z)
• BayesBlend: Easy Model Blending using Pseudo-Bayesian Model Averaging, Stacking and Hierarchical Stacking in Python [0.0]
  We introduce the BayesBlend Python package to estimate weights and blend multiple (Bayesian) models' predictive distributions. BayesBlend implements pseudo-Bayesian model averaging, stacking and, uniquely, hierarchical Bayesian stacking to estimate model weights. We demonstrate the usage of BayesBlend with examples of insurance loss modeling.
  arXiv  Detail & Related papers  (2024-04-30T19:15:33Z)
• Language models scale reliably with over-training and on downstream tasks [121.69867718185125]
  Scaling laws are useful guides for derisking expensive training runs. However, there remain gaps between current studies and how language models are trained. In contrast, scaling laws mostly predict loss on inference, but models are usually compared on downstream task performance.
  arXiv  Detail & Related papers  (2024-03-13T13:54:00Z)
• nanoLM: an Affordable LLM Pre-training Benchmark via Accurate Loss Prediction across Scales [65.01417261415833]
  We present an approach to predict the pre-training loss based on our observations that Maximal Update Parametrization (muP) enables accurate fitting of scaling laws. With around 14% of the one-time pre-training cost, we can accurately forecast the loss for models up to 52B. Our goal with nanoLM is to empower researchers with limited resources to reach meaningful conclusions on large models.
  arXiv  Detail & Related papers  (2023-04-14T00:45:01Z)
• An Approximation Method for Fitted Random Forests [0.0]
  We study methods that approximate each fitted tree in the Random Forests model using the multinomial allocation of the data points to the leafs. Specifically, we begin by studying whether fitting a multinomial logistic regression helps reduce the size while preserving the prediction quality.
  arXiv  Detail & Related papers  (2022-07-05T17:28:52Z)
• Probabilistic Modeling for Human Mesh Recovery [73.11532990173441]
  This paper focuses on the problem of 3D human reconstruction from 2D evidence. We recast the problem as learning a mapping from the input to a distribution of plausible 3D poses.
  arXiv  Detail & Related papers  (2021-08-26T17:55:11Z)
• Scaling Laws for Acoustic Models [7.906034575114518]
  Recent work has shown that autoregressive generative models with cross-entropy objective functions exhibit smooth power-law relationships. We show that acoustic models trained with an auto-predictive coding loss behave as if they are subject to similar scaling laws.
  arXiv  Detail & Related papers  (2021-06-11T18:59:24Z)
• Bayes DistNet -- A Robust Neural Network for Algorithm Runtime Distribution Predictions [1.8275108630751844]
  Randomized algorithms are used in many state-of-the-art solvers for constraint satisfaction problems (CSP) and Boolean satisfiability (SAT) problems. Previous state-of-the-art methods directly try to predict a fixed parametric distribution that the input instance follows. This new model achieves robust predictive performance in the low observation setting, as well as handling censored observations.
  arXiv  Detail & Related papers  (2020-12-14T01:15:39Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.