Modeling the Machine Learning Multiverse

• URL: http://arxiv.org/abs/2206.05985v1
• Date: Mon, 13 Jun 2022 09:11:48 GMT
• Title: Modeling the Machine Learning Multiverse
• Authors: Samuel J. Bell, Onno P. Kampman, Jesse Dodge and Neil D. Lawrence
• Abstract summary: We present a principled framework for making robust and generalizable claims in machine learning research. Our framework builds upon the psychology Multiverse Analysis introduced in response to psychology's own crisis. For the machine learning community, the Multiverse Analysis is a simple and effective technique for identifying robust claims.
• Score: 10.809039816152161
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Amid mounting concern about the reliability and credibility of machine learning research, we present a principled framework for making robust and generalizable claims: the Multiverse Analysis. Our framework builds upon the Multiverse Analysis (Steegen et al., 2016) introduced in response to psychology's own reproducibility crisis. To efficiently explore high-dimensional and often continuous ML search spaces, we model the multiverse with a Gaussian Process surrogate and apply Bayesian experimental design. Our framework is designed to facilitate drawing robust scientific conclusions about model performance, and thus our approach focuses on exploration rather than conventional optimization. In the first of two case studies, we investigate disputed claims about the relative merit of adaptive optimizers. Second, we synthesize conflicting research on the effect of learning rate on the large batch training generalization gap. For the machine learning community, the Multiverse Analysis is a simple and effective technique for identifying robust claims, for increasing transparency, and a step toward improved reproducibility.

Related papers

• Two-Stage Surrogate Modeling for Data-Driven Design Optimization with Application to Composite Microstructure Generation [1.912429179274357]
  This paper introduces a novel two-stage machine learning-based surrogate modeling framework to address inverse problems in scientific and engineering fields. In the first stage, a machine learning model termed the "learner" identifies a limited set of candidates within the input design space whose predicted outputs closely align with desired outcomes. In the second stage, a separate surrogate model, functioning as an "evaluator," is employed to assess the reduced candidate space generated in the first stage.
  arXiv  Detail & Related papers  (2024-01-04T00:25:12Z)
• Efficient Inverse Design Optimization through Multi-fidelity Simulations, Machine Learning, and Search Space Reduction Strategies [0.8646443773218541]
  This paper introduces a methodology designed to augment the inverse design optimization process in scenarios constrained by limited compute. The proposed methodology is analyzed on two distinct engineering inverse design problems: airfoil inverse design and the scalar field reconstruction problem. Notably, this method is adaptable across any inverse design application, facilitating a synergy between a representative low-fidelity ML model, and high-fidelity simulation, and can be seamlessly applied across any variety of population-based optimization algorithms.
  arXiv  Detail & Related papers  (2023-12-06T18:20:46Z)
• Topology-aware Robust Optimization for Out-of-distribution Generalization [18.436575017126323]
  Out-of-distribution (OOD) generalization is a challenging machine learning problem yet highly desirable in many high-stake applications. We propose topology-aware robust optimization (TRO) that seamlessly integrates distributional topology in a principled optimization framework. We theoretically demonstrate the effectiveness of our approach and empirically show that it significantly outperforms the state of the arts in a wide range of tasks including classification, regression, and semantic segmentation.
  arXiv  Detail & Related papers  (2023-07-26T03:48:37Z)
• End-to-End Meta-Bayesian Optimisation with Transformer Neural Processes [52.818579746354665]
  This paper proposes the first end-to-end differentiable meta-BO framework that generalises neural processes to learn acquisition functions via transformer architectures. We enable this end-to-end framework with reinforcement learning (RL) to tackle the lack of labelled acquisition data.
  arXiv  Detail & Related papers  (2023-05-25T10:58:46Z)
• IRGen: Generative Modeling for Image Retrieval [82.62022344988993]
  In this paper, we present a novel methodology, reframing image retrieval as a variant of generative modeling. We develop our model, dubbed IRGen, to address the technical challenge of converting an image into a concise sequence of semantic units. Our model achieves state-of-the-art performance on three widely-used image retrieval benchmarks and two million-scale datasets.
  arXiv  Detail & Related papers  (2023-03-17T17:07:36Z)
• Latent Variable Representation for Reinforcement Learning [131.03944557979725]
  It remains unclear theoretically and empirically how latent variable models may facilitate learning, planning, and exploration to improve the sample efficiency of model-based reinforcement learning. We provide a representation view of the latent variable models for state-action value functions, which allows both tractable variational learning algorithm and effective implementation of the optimism/pessimism principle. In particular, we propose a computationally efficient planning algorithm with UCB exploration by incorporating kernel embeddings of latent variable models.
  arXiv  Detail & Related papers  (2022-12-17T00:26:31Z)
• USCO-Solver: Solving Undetermined Stochastic Combinatorial Optimization Problems [9.015720257837575]
  We consider the regression between spaces, aiming to infer high-quality optimization solutions from samples of input-solution pairs. For learning foundations, we present learning-error analysis under the PAC-Bayesian framework. We obtain highly encouraging experimental results for several classic problems on both synthetic and real-world datasets.
  arXiv  Detail & Related papers  (2021-07-15T17:59:08Z)
• Harnessing Heterogeneity: Learning from Decomposed Feedback in Bayesian Modeling [68.69431580852535]
  We introduce a novel GP regression to incorporate the subgroup feedback. Our modified regression has provably lower variance -- and thus a more accurate posterior -- compared to previous approaches. We execute our algorithm on two disparate social problems.
  arXiv  Detail & Related papers  (2021-07-07T03:57:22Z)
• Machine Learning Framework for Quantum Sampling of Highly-Constrained, Continuous Optimization Problems [101.18253437732933]
  We develop a generic, machine learning-based framework for mapping continuous-space inverse design problems into surrogate unconstrained binary optimization problems. We showcase the framework's performance on two inverse design problems by optimizing thermal emitter topologies for thermophotovoltaic applications and (ii) diffractive meta-gratings for highly efficient beam steering.
  arXiv  Detail & Related papers  (2021-05-06T02:22:23Z)
• An Empirical Study of Assumptions in Bayesian Optimisation [61.19427472792523]
  In this work we rigorously analyse conventional and non-conventional assumptions inherent to Bayesian optimisation. We conclude that the majority of hyper- parameter tuning tasks exhibit heteroscedasticity and non-stationarity. We hope these findings may serve as guiding principles, both for practitioners and for further research in the field.
  arXiv  Detail & Related papers  (2020-12-07T16:21:12Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.