Problems with Chinchilla Approach 2: Systematic Biases in IsoFLOP Parabola Fits

• URL: http://arxiv.org/abs/2603.22339v3
• Date: Sat, 28 Mar 2026 00:14:25 GMT
• Title: Problems with Chinchilla Approach 2: Systematic Biases in IsoFLOP Parabola Fits
• Abstract summary: Chinchilla Approach 2 biases imply a parameter underallocation corresponding to 6.5% of the $3.8times1025$ FLOP training budget.<n>Chinchilla Approach 3 largely eliminates these biases but is often regarded as less data-efficient, numerically unstable, and prone to local minima.<n>It may serve as a more convenient replacement for Approach 2 or a more scalable alternative for adaptations of Approach 3 to richer scaling law formulations.
• Score: 35.1311414689253
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Chinchilla Approach 2 is among the most widely used methods for fitting neural scaling laws. Its parabolic approximation introduces systematic biases in compute-optimal allocation estimates, even on noise-free synthetic data. Applied to published Llama 3 IsoFLOP data at open frontier compute scales, these biases imply a parameter underallocation corresponding to 6.5% of the $3.8\times10^{25}$ FLOP training budget and \$1.4M (90% CI: \$412K-\$2.9M) in unnecessary compute at 50% H100 MFU. Simulated multimodal model misallocations show even greater opportunity costs due to higher loss surface asymmetry. Three sources of this error are examined: IsoFLOP sampling grid width (Taylor approximation accuracy), uncentered IsoFLOP sampling, and loss surface asymmetry ($α\neq β$). Chinchilla Approach 3 largely eliminates these biases but is often regarded as less data-efficient, numerically unstable, prone to local minima, and harder to implement. Each concern is shown to be unfounded or addressable, especially when the partially linear structure of the objective is exploited via Variable Projection, enabling unbiased inference on all five loss surface parameters through a two-dimensional optimization that is well-conditioned, analytically differentiable, and amenable to dense, or even exhaustive, grid search. It may serve as a more convenient replacement for Approach 2 or a more scalable alternative for adaptations of Approach 3 to richer scaling law formulations. See https://github.com/Open-Athena/vpnls for details and https://openathena.ai/scaling-law-analysis for other results from this study.

Related papers

• Adaptive Nyström for Gaussian Process Regression [1.14219428942199]
  Low-rank Nystrm approximations can reduce this burden to $O(nm2)$, but their accuracy depends heavily on the selection of landmark points.<n>We propose an adaptive Nystrm approach that greedily selects landmarks to minimize the trace residual of the kernel approximation error.
  arXiv  Detail & Related papers  (2026-07-29T19:55:12Z)
• Level-Crossing Density as a Mesh-Free High-Frequency Auxiliary Loss for Implicit Neural Representations [0.0]
  We derive smooth Monte-Carlo estimators for a continuous neural field using no grid, no complex, no persistence.<n>The estimator is accurate to 1-3% against exact topology in 2D and 3D, and costs about 3 ms per iteration.<n>In 3D neural-SDF fitting, however, a failure mode we believe general to any sampled soft topology objective appears.
  arXiv  Detail & Related papers  (2026-07-07T04:23:23Z)
• Solve for the Hyperparameter, Skip the Search: Kolmogorov-Optimal Scaling Laws for Spline Regression [0.0]
  Optimal resolution can be solved for in closed form, to the accuracy an exhaustive search reaches, at a fraction of the compute.<n>KORE fits two pilot resolutions, solves a leverage-calibrated 2x2 system for the bias and noise scales, and evaluates the closed-form plug-in resolution with a tiny leave-one-out certificate.
  arXiv  Detail & Related papers  (2026-06-22T16:41:10Z)
• SA-CycleGAN-2.5D: Self-Attention CycleGAN with Tri-Planar Context for Multi-Site MRI Harmonization [3.5109108807229403]
  We propose SA-CycleGAN-2.5D, a domain adaptation framework motivated by the $HH$-divergence bound of Ben-David et al.<n>By bridging 2D efficiency and 3D consistency, our framework yields voxel-level harmonized images that preserve tumor pathophysiology.
  arXiv  Detail & Related papers  (2026-03-17T23:49:46Z)
• Closing the Approximation Gap of Partial AUC Optimization: A Tale of Two Formulations [121.39938773554523]
  The Area Under the ROC Curve (AUC) is a pivotal evaluation metric in real-world scenarios with both class imbalance and decision constraints.<n>We present two simple instance-wise minimax reformulations to close the approximation gap of PAUC optimization.<n>The resulting algorithms enjoy a linear per-iteration computational complexity w.r.t. the sample size and a convergence rate of $O(-2/3)$ for typical one-way and two-way PAUCs.
  arXiv  Detail & Related papers  (2025-12-01T02:52:33Z)
• Spline Dimensional Decomposition with Interpolation-based Optimal Knot Selection for Stochastic Dynamic Analysis [6.407952035735354]
  Spline dimensional (SDD) addresses non-smooth or locally nonlinear behaviors by partitioning input coordinates via knot placement.<n>We propose a computationally efficient, computationally expensive method for optimal knot selection in SDD.<n>A modal analysis of a lower control arm shows that SDD with the proposed knots yields higher accuracy than SDD with uniformly or randomly spaced knots.
  arXiv  Detail & Related papers  (2025-05-19T09:08:39Z)
• Enhancing Unsupervised Feature Selection via Double Sparsity Constrained Optimization [6.342485512772862]
  Unsupervised single feature selection (UFS) is widely applied in machine learning and pattern recognition.<n>Most of the existing methods only consider sparsity, which makes it difficult to select subsets and discriminative from the original.<n>In this paper, we propose a new method called DSCOFS to consider a subset and discriminative from the original.
  arXiv  Detail & Related papers  (2025-01-01T05:05:46Z)
• Two-Timescale Optimization Framework for Sparse-Feedback Linear-Quadratic Optimal Control [3.746304628644379]
  A $mathcalHfeedback$-guaranteed sparse-feedback linear-quadratic (LQ) optimal control with convex parameterization and convex-bounded uncertainty is studied.
  arXiv  Detail & Related papers  (2024-06-17T03:17:33Z)
• Scalable 3D Registration via Truncated Entry-wise Absolute Residuals [65.04922801371363]
  A $3$D registration approach can process more than ten million ($107$) point pairs with over $99%$ random outliers. We call our method TEAR, as it involves minimizing an outlier-robust loss that computes Truncated Entry-wise Absolute Residuals.
  arXiv  Detail & Related papers  (2024-04-01T04:43:39Z)
• DF2: Distribution-Free Decision-Focused Learning [30.288876294435294]
  Decision-focused learning (DFL) has emerged as a powerful approach for predict-then-optimize problems.<n>DFL faces three bottlenecks: model error, sample average approximation error, and approximation error.<n>We present DF2, the first decision-free learning method designed to mitigate these three bottlenecks.
  arXiv  Detail & Related papers  (2023-08-11T00:44:46Z)
• FeDXL: Provable Federated Learning for Deep X-Risk Optimization [105.17383135458897]
  We tackle a novel federated learning (FL) problem for optimizing a family of X-risks, to which no existing algorithms are applicable. The challenges for designing an FL algorithm for X-risks lie in the non-decomability of the objective over multiple machines and the interdependency between different machines.
  arXiv  Detail & Related papers  (2022-10-26T00:23:36Z)
• Distributed Sparse Regression via Penalization [5.990069843501885]
  We study linear regression over a network of agents, modeled as an undirected graph (with no centralized node) The estimation problem is formulated as the minimization of the sum of the local LASSO loss functions plus a quadratic penalty of the consensus constraint. We show that the proximal-gradient algorithm applied to the penalized problem converges linearly up to a tolerance of the order of the centralized statistical error.
  arXiv  Detail & Related papers  (2021-11-12T01:51:50Z)
• STORM+: Fully Adaptive SGD with Momentum for Nonconvex Optimization [74.1615979057429]
  We investigate non-batch optimization problems where the objective is an expectation over smooth loss functions. Our work builds on the STORM algorithm, in conjunction with a novel approach to adaptively set the learning rate and momentum parameters.
  arXiv  Detail & Related papers  (2021-11-01T15:43:36Z)
• A Retrospective Approximation Approach for Smooth Stochastic Optimization [0.2867517731896504]
  Gradient (SG) is the defactorandom iterative technique to solve optimization (SO) problems with a smooth (non-fimation) objective $imation.
  arXiv  Detail & Related papers  (2021-03-07T16:29:36Z)
• Canny-VO: Visual Odometry with RGB-D Cameras based on Geometric 3D-2D Edge Alignment [85.32080531133799]
  This paper reviews the classical problem of free-form curve registration and applies it to an efficient RGBD visual odometry system called Canny-VO. Two replacements for the distance transformation commonly used in edge registration are proposed: Approximate Nearest Neighbour Fields and Oriented Nearest Neighbour Fields. 3D2D edge alignment benefits from these alternative formulations in terms of both efficiency and accuracy.
  arXiv  Detail & Related papers  (2020-12-15T11:42:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.