Mean-field methods and algorithmic perspectives for high-dimensional machine learning

• URL: http://arxiv.org/abs/2103.05945v1
• Date: Wed, 10 Mar 2021 09:02:36 GMT
• Title: Mean-field methods and algorithmic perspectives for high-dimensional machine learning
• Authors: Benjamin Aubin
• Abstract summary: We revisit an approach based on the tools of statistical physics of disordered systems. We capitalize on the deep connection between the replica method and message passing algorithms in order to shed light on the phase diagrams of various theoretical models.
• Score: 5.406386303264086
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The main difficulty that arises in the analysis of most machine learning algorithms is to handle, analytically and numerically, a large number of interacting random variables. In this Ph.D manuscript, we revisit an approach based on the tools of statistical physics of disordered systems. Developed through a rich literature, they have been precisely designed to infer the macroscopic behavior of a large number of particles from their microscopic interactions. At the heart of this work, we strongly capitalize on the deep connection between the replica method and message passing algorithms in order to shed light on the phase diagrams of various theoretical models, with an emphasis on the potential differences between statistical and algorithmic thresholds. We essentially focus on synthetic tasks and data generated in the teacher-student paradigm. In particular, we apply these mean-field methods to the Bayes-optimal analysis of committee machines, to the worst-case analysis of Rademacher generalization bounds for perceptrons, and to empirical risk minimization in the context of generalized linear models. Finally, we develop a framework to analyze estimation models with structured prior informations, produced for instance by deep neural networks based generative models with random weights.

Related papers

• High-dimensional learning of narrow neural networks [1.7094064195431147]
  This manuscript reviews the tools and ideas underlying recent progress in machine learning. We introduce a generic model -- the sequence multi-index model -- which encompasses numerous previously studied models as special instances. We explicate in full detail the analysis of the learning of sequence multi-index models, using statistical physics techniques such as the replica method and approximate message-passing algorithms.
  arXiv  Detail & Related papers  (2024-09-20T21:20:04Z)
• Unified Explanations in Machine Learning Models: A Perturbation Approach [0.0]
  Inconsistencies between XAI and modeling techniques can have the undesirable effect of casting doubt upon the efficacy of these explainability approaches. We propose a systematic, perturbation-based analysis against a popular, model-agnostic method in XAI, SHapley Additive exPlanations (Shap) We devise algorithms to generate relative feature importance in settings of dynamic inference amongst a suite of popular machine learning and deep learning methods, and metrics that allow us to quantify how well explanations generated under the static case hold.
  arXiv  Detail & Related papers  (2024-05-30T16:04:35Z)
• Enhancing Multiscale Simulations with Constitutive Relations-Aware Deep Operator Networks [0.7946947383637114]
  Multiscale finite element computations are commended for their ability to integrate micro-structural properties into macroscopic computational analyses. We propose a hybrid method in which we utilize deep operator networks for surrogate modeling of the microscale physics.
  arXiv  Detail & Related papers  (2024-05-22T15:40:05Z)
• Discovering Interpretable Physical Models using Symbolic Regression and Discrete Exterior Calculus [55.2480439325792]
  We propose a framework that combines Symbolic Regression (SR) and Discrete Exterior Calculus (DEC) for the automated discovery of physical models. DEC provides building blocks for the discrete analogue of field theories, which are beyond the state-of-the-art applications of SR to physical problems. We prove the effectiveness of our methodology by re-discovering three models of Continuum Physics from synthetic experimental data.
  arXiv  Detail & Related papers  (2023-10-10T13:23:05Z)
• Capturing dynamical correlations using implicit neural representations [85.66456606776552]
  We develop an artificial intelligence framework which combines a neural network trained to mimic simulated data from a model Hamiltonian with automatic differentiation to recover unknown parameters from experimental data. In doing so, we illustrate the ability to build and train a differentiable model only once, which then can be applied in real-time to multi-dimensional scattering data.
  arXiv  Detail & Related papers  (2023-04-08T07:55:36Z)
• RandomSCM: interpretable ensembles of sparse classifiers tailored for omics data [59.4141628321618]
  We propose an ensemble learning algorithm based on conjunctions or disjunctions of decision rules. The interpretability of the models makes them useful for biomarker discovery and patterns discovery in high dimensional data.
  arXiv  Detail & Related papers  (2022-08-11T13:55:04Z)
• Model-Based Deep Learning: On the Intersection of Deep Learning and Optimization [101.32332941117271]
  Decision making algorithms are used in a multitude of different applications. Deep learning approaches that use highly parametric architectures tuned from data without relying on mathematical models are becoming increasingly popular. Model-based optimization and data-centric deep learning are often considered to be distinct disciplines.
  arXiv  Detail & Related papers  (2022-05-05T13:40:08Z)
• Mixed Effects Neural ODE: A Variational Approximation for Analyzing the Dynamics of Panel Data [50.23363975709122]
  We propose a probabilistic model called ME-NODE to incorporate (fixed + random) mixed effects for analyzing panel data. We show that our model can be derived using smooth approximations of SDEs provided by the Wong-Zakai theorem. We then derive Evidence Based Lower Bounds for ME-NODE, and develop (efficient) training algorithms.
  arXiv  Detail & Related papers  (2022-02-18T22:41:51Z)
• Structured learning of rigid-body dynamics: A survey and unified view from a robotics perspective [5.597839822252915]
  We study supervised regression models that combine rigid-body mechanics with data-driven modelling techniques. We provide a unified view on the combination of data-driven regression models, such as neural networks and Gaussian processes, with analytical model priors.
  arXiv  Detail & Related papers  (2020-12-11T11:26:48Z)
• Multiplicative noise and heavy tails in stochastic optimization [62.993432503309485]
  empirical optimization is central to modern machine learning, but its role in its success is still unclear. We show that it commonly arises in parameters of discrete multiplicative noise due to variance. A detailed analysis is conducted in which we describe on key factors, including recent step size, and data, all exhibit similar results on state-of-the-art neural network models.
  arXiv  Detail & Related papers  (2020-06-11T09:58:01Z)
• Information-theoretic limits of a multiview low-rank symmetric spiked matrix model [19.738567726658875]
  We consider a generalization of an important class of high-dimensional inference problems, namely spiked symmetric matrix models. We rigorously establish the information-theoretic limits through the proof of single-letter formulas. We improve the recently introduced adaptive method, so that it can be used to study low-rank models.
  arXiv  Detail & Related papers  (2020-05-16T15:31:07Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.