Estimating the Local Learning Coefficient at Scale

• URL: http://arxiv.org/abs/2402.03698v2
• Date: Mon, 30 Sep 2024 05:40:51 GMT
• Title: Estimating the Local Learning Coefficient at Scale
• Abstract summary: The textitlocal learning coefficient (LLC) is a principled way of quantifying model complexity. We empirically show how the LLC may be measured accurately and self-consistently for deep linear networks (DLNs) up to 100M parameters.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The \textit{local learning coefficient} (LLC) is a principled way of quantifying model complexity, originally derived in the context of Bayesian statistics using singular learning theory (SLT). Several methods are known for numerically estimating the local learning coefficient, but so far these methods have not been extended to the scale of modern deep learning architectures or data sets. Using a method developed in {\tt arXiv:2308.12108 [stat.ML]} we empirically show how the LLC may be measured accurately and self-consistently for deep linear networks (DLNs) up to 100M parameters. We also show that the estimated LLC has the rescaling invariance that holds for the theoretical quantity.

Related papers

• Symmetries and Singularities [4.119847434567505]
  We study the local learning coefficient (LLC), which characterizes the effective complexity of a model near a given solution.<n>Existing methods for estimating the LLC often rely on posterior sampling, which can be computationally expensive for large neural networks.<n>In this work, we use known structures in the model to simplify the analysis and make LLC estimation more tractable.
  arXiv  Detail & Related papers  (2026-09-13T17:01:13Z)
• Exact Algebraic Computation of Learning Coefficients for Two-Dimensional Singular Models [0.5735035463793009]
  We present the first deterministic algorithm that computes local RLCT exactly for any two-dimensional computation whose Kullback-Leibler model is contact equivalent to a distance.<n>Our algorithm reveals exact structure in learning coefficients that sampling cannot and out-speeds it in the shallow regime.
  arXiv  Detail & Related papers  (2026-08-20T15:40:50Z)
• Estimation of the Learning Coefficient Using Empirical Loss [0.9208007322096532]
  The learning coefficient plays a crucial role in analyzing the performance of information criteria. We propose a novel numerical estimation method that fundamentally differs from previous approaches.
  arXiv  Detail & Related papers  (2025-02-14T08:30:04Z)
• The Statistical Accuracy of Neural Posterior and Likelihood Estimation [0.5825410941577592]
  We give an in-depth exploration on the statistical behavior of NPE and NLE. We prove that these methods have similar theoretical guarantees to common statistical methods like approximate Bayesian computation (ABC) and Bayesian synthetic likelihood (BSL) We prove that this accuracy can often be achieved at a vastly reduced computational cost, and will therefore deliver more attractive approximations than ABC and BSL in certain problems.
  arXiv  Detail & Related papers  (2024-11-18T21:25:32Z)
• Iterative Methods for Full-Scale Gaussian Process Approximations for Large Spatial Data [9.913418444556486]
  We show how iterative methods can be used to reduce the computational costs for calculating likelihoods, gradients, and predictive distributions with FSAs. We also present a novel, accurate, and fast way to calculate predictive variances relying on estimations and iterative methods. All methods are implemented in a free C++ software library with high-level Python and R packages.
  arXiv  Detail & Related papers  (2024-05-23T12:25:22Z)
• Querying Easily Flip-flopped Samples for Deep Active Learning [63.62397322172216]
  Active learning is a machine learning paradigm that aims to improve the performance of a model by strategically selecting and querying unlabeled data. One effective selection strategy is to base it on the model's predictive uncertainty, which can be interpreted as a measure of how informative a sample is. This paper proposes the it least disagree metric (LDM) as the smallest probability of disagreement of the predicted label.
  arXiv  Detail & Related papers  (2024-01-18T08:12:23Z)
• Unifying Synergies between Self-supervised Learning and Dynamic Computation [53.66628188936682]
  We present a novel perspective on the interplay between SSL and DC paradigms. We show that it is feasible to simultaneously learn a dense and gated sub-network from scratch in a SSL setting. The co-evolution during pre-training of both dense and gated encoder offers a good accuracy-efficiency trade-off.
  arXiv  Detail & Related papers  (2023-01-22T17:12:58Z)
• Adaptive neighborhood Metric learning [184.95321334661898]
  We propose a novel distance metric learning algorithm, named adaptive neighborhood metric learning (ANML) ANML can be used to learn both the linear and deep embeddings. The emphlog-exp mean function proposed in our method gives a new perspective to review the deep metric learning methods.
  arXiv  Detail & Related papers  (2022-01-20T17:26:37Z)
• Learning to Estimate Without Bias [57.82628598276623]
  Gauss theorem states that the weighted least squares estimator is a linear minimum variance unbiased estimation (MVUE) in linear models. In this paper, we take a first step towards extending this result to non linear settings via deep learning with bias constraints. A second motivation to BCE is in applications where multiple estimates of the same unknown are averaged for improved performance.
  arXiv  Detail & Related papers  (2021-10-24T10:23:51Z)
• Neural Networks for Parameter Estimation in Intractable Models [0.0]
  We show how to estimate parameters from max-stable processes, where inference is exceptionally challenging. We use data from model simulations as input and train deep neural networks to learn statistical parameters.
  arXiv  Detail & Related papers  (2021-07-29T21:59:48Z)
• Clustered Federated Learning via Generalized Total Variation Minimization [83.26141667853057]
  We study optimization methods to train local (or personalized) models for local datasets with a decentralized network structure. Our main conceptual contribution is to formulate federated learning as total variation minimization (GTV) Our main algorithmic contribution is a fully decentralized federated learning algorithm.
  arXiv  Detail & Related papers  (2021-05-26T18:07:19Z)
• Fast covariance parameter estimation of spatial Gaussian process models using neural networks [0.0]
  We train NNs to take moderate size spatial fields or variograms as input and return the range and noise-to-signal covariance parameters. Once trained, the NNs provide estimates with a similar accuracy compared to ML estimation and at a speedup by a factor of 100 or more. This work can be easily extended to other, more complex, spatial problems and provides a proof-of-concept for this use of machine learning in computational statistics.
  arXiv  Detail & Related papers  (2020-12-30T22:06:26Z)
• Efficient Debiased Evidence Estimation by Multilevel Monte Carlo Sampling [0.0]
  We propose a new optimization algorithm for Bayesian inference based multilevel Monte Carlo (MLMC) methods. Our numerical results confirm considerable computational savings compared to the conventional estimators.
  arXiv  Detail & Related papers  (2020-01-14T09:14:24Z)
• Localized Debiased Machine Learning: Efficient Inference on Quantile Treatment Effects and Beyond [69.83813153444115]
  We consider an efficient estimating equation for the (local) quantile treatment effect ((L)QTE) in causal inference. Debiased machine learning (DML) is a data-splitting approach to estimating high-dimensional nuisances. We propose localized debiased machine learning (LDML), which avoids this burdensome step.
  arXiv  Detail & Related papers  (2019-12-30T14:42:52Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.