Estimating the expected output of wide random MLPs more efficiently than sampling

• URL: http://arxiv.org/abs/2605.05179v1
• Date: Wed, 06 May 2026 17:46:12 GMT
• Title: Estimating the expected output of wide random MLPs more efficiently than sampling
• Abstract summary: We show how to estimate an expected loss in machine learning without running samples through the network at all.<n>We approximate representations of the distributions of activations at each layer, leveraging tools such as cumulants and Hermite expansions.<n>Our methods perform particularly well at estimating the probabilities of rare events, and additionally demonstrate how they can be used for model training.
• Score: 6.961302836244584
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: By far the most common way to estimate an expected loss in machine learning is to draw samples, compute the loss on each one, and take the empirical average. However, sampling is not necessarily optimal. Given an MLP at initialization, we show how to estimate its expected output over Gaussian inputs without running samples through the network at all. Instead, we produce approximate representations of the distributions of activations at each layer, leveraging tools such as cumulants and Hermite expansions. We show both theoretically and empirically that for sufficiently wide networks, our estimator achieves a target mean squared error using substantially fewer FLOPs than Monte Carlo sampling. We find moreover that our methods perform particularly well at estimating the probabilities of rare events, and additionally demonstrate how they can be used for model training. Together, these findings suggest a path to producing models with a greatly reduced probability of catastrophic tail risks.

Related papers

• UnpredictaBench: A Benchmark for Evaluating Distributional Randomness in LLMs [21.05820098186702]
  We introduce UnpredictaBench, an evaluation that tests the ability of large language models to capture true underlying distributions.<n>We find that even simple distributional simulation remains challenging, making it a necessary first step toward using LLMs as stand-ins for complex systems.
  arXiv  Detail & Related papers  (2026-06-04T18:18:02Z)
• Estimating the Probabilities of Rare Outputs in Language Models [8.585890569162267]
  We study low probability estimation in the context of argmax sampling from small transformer language models.<n>We find that importance sampling outperforms activation extrapolation, but both outperform naive sampling.<n>We argue that new methods for low probability estimation are needed to provide stronger guarantees about worst-case performance.
  arXiv  Detail & Related papers  (2024-10-17T04:31:18Z)
• Probabilistic Contrastive Learning for Long-Tailed Visual Recognition [78.70453964041718]
  Longtailed distributions frequently emerge in real-world data, where a large number of minority categories contain a limited number of samples. Recent investigations have revealed that supervised contrastive learning exhibits promising potential in alleviating the data imbalance. We propose a novel probabilistic contrastive (ProCo) learning algorithm that estimates the data distribution of the samples from each class in the feature space.
  arXiv  Detail & Related papers  (2024-03-11T13:44:49Z)
• Approximate Thompson Sampling via Epistemic Neural Networks [26.872304174606278]
  Epistemic neural networks (ENNs) are designed to produce accurate joint predictive distributions. We show that ENNs serve this purpose well and illustrate how the quality of joint predictive distributions drives performance.
  arXiv  Detail & Related papers  (2023-02-18T01:58:15Z)
• Rethinking Collaborative Metric Learning: Toward an Efficient Alternative without Negative Sampling [156.7248383178991]
  Collaborative Metric Learning (CML) paradigm has aroused wide interest in the area of recommendation systems (RS) We find that negative sampling would lead to a biased estimation of the generalization error. Motivated by this, we propose an efficient alternative without negative sampling for CML named textitSampling-Free Collaborative Metric Learning (SFCML)
  arXiv  Detail & Related papers  (2022-06-23T08:50:22Z)
• Low-variance estimation in the Plackett-Luce model via quasi-Monte Carlo sampling [58.14878401145309]
  We develop a novel approach to producing more sample-efficient estimators of expectations in the PL model. We illustrate our findings both theoretically and empirically using real-world recommendation data from Amazon Music and the Yahoo learning-to-rank challenge.
  arXiv  Detail & Related papers  (2022-05-12T11:15:47Z)
• Sampling from Arbitrary Functions via PSD Models [55.41644538483948]
  We take a two-step approach by first modeling the probability distribution and then sampling from that model. We show that these models can approximate a large class of densities concisely using few evaluations, and present a simple algorithm to effectively sample from these models.
  arXiv  Detail & Related papers  (2021-10-20T12:25:22Z)
• Unrolling Particles: Unsupervised Learning of Sampling Distributions [102.72972137287728]
  Particle filtering is used to compute good nonlinear estimates of complex systems. We show in simulations that the resulting particle filter yields good estimates in a wide range of scenarios.
  arXiv  Detail & Related papers  (2021-10-06T16:58:34Z)
• Efficiently Sampling Functions from Gaussian Process Posteriors [76.94808614373609]
  We propose an easy-to-use and general-purpose approach for fast posterior sampling. We demonstrate how decoupled sample paths accurately represent Gaussian process posteriors at a fraction of the usual cost.
  arXiv  Detail & Related papers  (2020-02-21T14:03:16Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.