We need to talk about random seeds

• URL: http://arxiv.org/abs/2210.13393v1
• Date: Mon, 24 Oct 2022 16:48:45 GMT
• Title: We need to talk about random seeds
• Authors: Steven Bethard
• Abstract summary: This opinion piece argues that there are some safe uses for random seeds. An analysis of 85 recent publications from the ACL Anthology finds that more than 50% contain risky uses of random seeds.
• Score: 16.33770822558325
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Modern neural network libraries all take as a hyperparameter a random seed, typically used to determine the initial state of the model parameters. This opinion piece argues that there are some safe uses for random seeds: as part of the hyperparameter search to select a good model, creating an ensemble of several models, or measuring the sensitivity of the training algorithm to the random seed hyperparameter. It argues that some uses for random seeds are risky: using a fixed random seed for "replicability" and varying only the random seed to create score distributions for performance comparison. An analysis of 85 recent publications from the ACL Anthology finds that more than 50% contain risky uses of random seeds.

Related papers

• Randomness control and reproducibility study of random forest algorithm in R and Python [0.0]
  We will discuss the strategy of integrating random forest intoocular tolerance assessment for toxicologists. We will compare four packages: randomForest and Ranger (R packages), adapted in Python via theSKRanger package, and the widely used Scikit-Learn with the RandomForestClassifier() function.
  arXiv  Detail & Related papers  (2024-08-22T07:59:49Z)
• Measuring model variability using robust non-parametric testing [5.519968037738177]
  This work attempts to describe the relationship between deep net models trained with different random seeds and the behavior of the expected model. We propose a novel summary statistic for network similarity, referred to as the $alpha$-trimming level. We show that the $alpha$-trimming level is more expressive than different performance metrics like validation accuracy, churn, or expected calibration error when taken alone.
  arXiv  Detail & Related papers  (2024-06-12T15:08:15Z)
• Optimal Multi-Distribution Learning [88.3008613028333]
  Multi-distribution learning seeks to learn a shared model that minimizes the worst-case risk across $k$ distinct data distributions. We propose a novel algorithm that yields an varepsilon-optimal randomized hypothesis with a sample complexity on the order of (d+k)/varepsilon2.
  arXiv  Detail & Related papers  (2023-12-08T16:06:29Z)
• Randomness: what is it and why does it matter? [0.0]
  A widely accepted definition of randomness lacks scientific rigor and its results are questionable. I propose an information-theory-based definition of randomness which focuses on the physical process of random number generation itself. A new quantity named "randomness deviation" allows for a practical measure of quality of a random number generating process or a device.
  arXiv  Detail & Related papers  (2023-03-14T16:38:16Z)
• Shortcomings of Top-Down Randomization-Based Sanity Checks for Evaluations of Deep Neural Network Explanations [67.40641255908443]
  We identify limitations of model-randomization-based sanity checks for the purpose of evaluating explanations. Top-down model randomization preserves scales of forward pass activations with high probability.
  arXiv  Detail & Related papers  (2022-11-22T18:52:38Z)
• Testing randomness of series generated in Bell's experiment [62.997667081978825]
  We use a toy fiber optic based setup to generate binary series, and evaluate their level of randomness according to Ville principle. Series are tested with a battery of standard statistical indicators, Hurst, Kolmogorov complexity, minimum entropy, Takensarity dimension of embedding, and Augmented Dickey Fuller and Kwiatkowski Phillips Schmidt Shin to check station exponent. The level of randomness of series obtained by applying Toeplitz extractor to rejected series is found to be indistinguishable from the level of non-rejected raw ones.
  arXiv  Detail & Related papers  (2022-08-31T17:39:29Z)
• Algorithms for Adaptive Experiments that Trade-off Statistical Analysis with Reward: Combining Uniform Random Assignment and Reward Maximization [50.725191156128645]
  Multi-armed bandit algorithms like Thompson Sampling can be used to conduct adaptive experiments. We present simulations for 2-arm experiments that explore two algorithms that combine the benefits of uniform randomization for statistical analysis.
  arXiv  Detail & Related papers  (2021-12-15T22:11:58Z)
• Generation of 1 Gb full entropy random numbers with the enhanced-NRBG method [0.13124513975412253]
  Experimental demonstration of a non-deterministic random bit generator from a quantum entropy source and a deterministic random bit generator mechanism. The extracted raw random numbers are further post-processed to generate a high-entropy seed for the hash based deterministic random bit generator.
  arXiv  Detail & Related papers  (2021-08-09T20:06:43Z)
• Modeling Sequences as Distributions with Uncertainty for Sequential Recommendation [63.77513071533095]
  Most existing sequential methods assume users are deterministic. Item-item transitions might fluctuate significantly in several item aspects and exhibit randomness of user interests. We propose a Distribution-based Transformer Sequential Recommendation (DT4SR) which injects uncertainties into sequential modeling.
  arXiv  Detail & Related papers  (2021-06-11T04:35:21Z)
• Improved device-independent randomness expansion rates using two sided randomness [3.4376560669160394]
  A device-independent randomness expansion protocol aims to take an initial random string and generate a longer one. We investigate the possible improvement that could be gained using the two-sided randomness. We also consider a modified protocol in which the input randomness is recycled.
  arXiv  Detail & Related papers  (2021-03-12T19:49:17Z)
• Generative Semantic Hashing Enhanced via Boltzmann Machines [61.688380278649056]
  Existing generative-hashing methods mostly assume a factorized form for the posterior distribution. We propose to employ the distribution of Boltzmann machine as the retrievalal posterior. We show that by effectively modeling correlations among different bits within a hash code, our model can achieve significant performance gains.
  arXiv  Detail & Related papers  (2020-06-16T01:23:39Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.