Information theory for data-driven model reduction in physics and biology

• URL: http://arxiv.org/abs/2312.06608v2
• Date: Wed, 17 Apr 2024 16:58:36 GMT
• Title: Information theory for data-driven model reduction in physics and biology
• Authors: Matthew S. Schmitt, Maciej Koch-Janusz, Michel Fruchart, Daniel S. Seara, Michael Rust, Vincenzo Vitelli,
• Abstract summary: We develop a systematic approach based on the information bottleneck to identify the relevant variables. We show that in the limit of high compression, the relevant variables are directly determined by the slowest-decaying eigenfunctions. It provides a firm foundation to construct interpretable deep learning tools that perform model reduction.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Model reduction is the construction of simple yet predictive descriptions of the dynamics of many-body systems in terms of a few relevant variables. A prerequisite to model reduction is the identification of these relevant variables, a task for which no general method exists. Here, we develop a systematic approach based on the information bottleneck to identify the relevant variables, defined as those most predictive of the future. We elucidate analytically the relation between these relevant variables and the eigenfunctions of the transfer operator describing the dynamics. Further, we show that in the limit of high compression, the relevant variables are directly determined by the slowest-decaying eigenfunctions. Our information-based approach indicates when to optimally stop increasing the complexity of the reduced model. Furthermore, it provides a firm foundation to construct interpretable deep learning tools that perform model reduction. We illustrate how these tools work in practice by considering uncurated videos of atmospheric flows from which our algorithms automatically extract the dominant slow collective variables, as well as experimental videos of cyanobacteria colonies in which we discover an emergent synchronization order parameter.

Related papers

• Learning Causal Structure Distributions for Robust Planning [53.753366558072806]
  We find that learning the functional relationships while accounting for the uncertainty about the structural information leads to more robust dynamics models.<n>This in contrast with common model-learning methods that ignore the causal structure and fail to leverage the sparsity of interactions in robotic systems.<n>We show that our model can be used to learn the dynamics of a robot, which together with a sampling-based planner can be used to perform new tasks in novel environments.
  arXiv  Detail & Related papers  (2025-08-08T22:43:17Z)
• Neural Network Reprogrammability: A Unified Theme on Model Reprogramming, Prompt Tuning, and Prompt Instruction [55.914891182214475]
  We introduce neural network reprogrammability as a unifying framework for model adaptation.<n>We present a taxonomy that categorizes such information manipulation approaches across four key dimensions.<n>We also analyze remaining technical challenges and ethical considerations.
  arXiv  Detail & Related papers  (2025-06-05T05:42:27Z)
• Learning Structural Causal Models from Ordering: Identifiable Flow Models [19.99352354910655]
  We introduce a set of flow models that can recover component-wise, invertible transformation of variables. We propose design improvements that enable simultaneous learning of all causal mechanisms. Our method achieves a significant reduction in computational time compared to existing diffusion-based techniques.
  arXiv  Detail & Related papers  (2024-12-13T04:25:56Z)
• Identifiable Representation and Model Learning for Latent Dynamic Systems [0.0]
  We study the problem of identifiable representation and model learning for latent dynamic systems.<n>We prove that, for linear and affine nonlinear latent dynamic systems with sparse input matrices, it is possible to identify the latent variables up to scaling.
  arXiv  Detail & Related papers  (2024-10-23T13:55:42Z)
• Neural Persistence Dynamics [8.197801260302642]
  We consider the problem of learning the dynamics in the topology of time-evolving point clouds. Our proposed model - $textitNeural Persistence Dynamics$ - substantially outperforms the state-of-the-art across a diverse set of parameter regression tasks.
  arXiv  Detail & Related papers  (2024-05-24T17:20:18Z)
• REMEDI: Corrective Transformations for Improved Neural Entropy Estimation [0.7488108981865708]
  We introduce $textttREMEDI$ for efficient and accurate estimation of differential entropy. Our approach demonstrates improvement across a broad spectrum of estimation tasks. It can be naturally extended to information theoretic supervised learning models.
  arXiv  Detail & Related papers  (2024-02-08T14:47:37Z)
• Learning minimal representations of stochastic processes with variational autoencoders [52.99137594502433]
  We introduce an unsupervised machine learning approach to determine the minimal set of parameters required to describe a process. Our approach enables for the autonomous discovery of unknown parameters describing processes.
  arXiv  Detail & Related papers  (2023-07-21T14:25:06Z)
• Learning Latent Dynamics via Invariant Decomposition and (Spatio-)Temporal Transformers [0.6767885381740952]
  We propose a method for learning dynamical systems from high-dimensional empirical data. We focus on the setting in which data are available from multiple different instances of a system. We study behaviour through simple theoretical analyses and extensive experiments on synthetic and real-world datasets.
  arXiv  Detail & Related papers  (2023-06-21T07:52:07Z)
• Capturing dynamical correlations using implicit neural representations [85.66456606776552]
  We develop an artificial intelligence framework which combines a neural network trained to mimic simulated data from a model Hamiltonian with automatic differentiation to recover unknown parameters from experimental data. In doing so, we illustrate the ability to build and train a differentiable model only once, which then can be applied in real-time to multi-dimensional scattering data.
  arXiv  Detail & Related papers  (2023-04-08T07:55:36Z)
• Scientific Machine Learning for Modeling and Simulating Complex Fluids [0.0]
  rheological equations relate internal stresses and deformations in complex fluids. Data-driven models provide accessible alternatives to expensive first-principles models. Development of similar models for complex fluids has lagged.
  arXiv  Detail & Related papers  (2022-10-10T04:35:31Z)
• Dynamic Latent Separation for Deep Learning [67.62190501599176]
  A core problem in machine learning is to learn expressive latent variables for model prediction on complex data. Here, we develop an approach that improves expressiveness, provides partial interpretation, and is not restricted to specific applications.
  arXiv  Detail & Related papers  (2022-10-07T17:56:53Z)
• On the Influence of Enforcing Model Identifiability on Learning dynamics of Gaussian Mixture Models [14.759688428864159]
  We propose a technique for extracting submodels from singular models. Our method enforces model identifiability during training. We show how the method can be applied to more complex models like deep neural networks.
  arXiv  Detail & Related papers  (2022-06-17T07:50:22Z)
• On the Generalization and Adaption Performance of Causal Models [99.64022680811281]
  Differentiable causal discovery has proposed to factorize the data generating process into a set of modules. We study the generalization and adaption performance of such modular neural causal models. Our analysis shows that the modular neural causal models outperform other models on both zero and few-shot adaptation in low data regimes.
  arXiv  Detail & Related papers  (2022-06-09T17:12:32Z)
• Learning Interacting Dynamical Systems with Latent Gaussian Process ODEs [13.436770170612295]
  We study for the first time uncertainty-aware modeling of continuous-time dynamics of interacting objects. Our model infers both independent dynamics and their interactions with reliable uncertainty estimates.
  arXiv  Detail & Related papers  (2022-05-24T08:36:25Z)
• Capturing Actionable Dynamics with Structured Latent Ordinary Differential Equations [68.62843292346813]
  We propose a structured latent ODE model that captures system input variations within its latent representation. Building on a static variable specification, our model learns factors of variation for each input to the system, thus separating the effects of the system inputs in the latent space.
  arXiv  Detail & Related papers  (2022-02-25T20:00:56Z)
• Towards Robust and Adaptive Motion Forecasting: A Causal Representation Perspective [72.55093886515824]
  We introduce a causal formalism of motion forecasting, which casts the problem as a dynamic process with three groups of latent variables. We devise a modular architecture that factorizes the representations of invariant mechanisms and style confounders to approximate a causal graph. Experiment results on synthetic and real datasets show that our three proposed components significantly improve the robustness and reusability of the learned motion representations.
  arXiv  Detail & Related papers  (2021-11-29T18:59:09Z)
• Discovery of Nonlinear Dynamical Systems using a Runge-Kutta Inspired Dictionary-based Sparse Regression Approach [9.36739413306697]
  We blend machine learning and dictionary-based learning with numerical analysis tools to discover governing differential equations. We obtain interpretable and parsimonious models which are prone to generalize better beyond the sampling regime. We discuss its extension to governing equations, containing rational nonlinearities that typically appear in biological networks.
  arXiv  Detail & Related papers  (2021-05-11T08:46:51Z)
• Triplot: model agnostic measures and visualisations for variable importance in predictive models that take into account the hierarchical correlation structure [3.0036519884678894]
  We propose new methods to support model analysis by exploiting the information about the correlation between variables. We show how to analyze groups of variables (aspects) both when they are proposed by the user and when they should be determined automatically. We also present the new type of model visualisation, triplot, which exploits a hierarchical structure of variable grouping to produce a high information density model visualisation.
  arXiv  Detail & Related papers  (2021-04-07T21:29:03Z)
• Beyond Trivial Counterfactual Explanations with Diverse Valuable Explanations [64.85696493596821]
  In computer vision applications, generative counterfactual methods indicate how to perturb a model's input to change its prediction. We propose a counterfactual method that learns a perturbation in a disentangled latent space that is constrained using a diversity-enforcing loss. Our model improves the success rate of producing high-quality valuable explanations when compared to previous state-of-the-art methods.
  arXiv  Detail & Related papers  (2021-03-18T12:57:34Z)
• Using Data Assimilation to Train a Hybrid Forecast System that Combines Machine-Learning and Knowledge-Based Components [52.77024349608834]
  We consider the problem of data-assisted forecasting of chaotic dynamical systems when the available data is noisy partial measurements. We show that by using partial measurements of the state of the dynamical system, we can train a machine learning model to improve predictions made by an imperfect knowledge-based model.
  arXiv  Detail & Related papers  (2021-02-15T19:56:48Z)
• Leveraging Global Parameters for Flow-based Neural Posterior Estimation [90.21090932619695]
  Inferring the parameters of a model based on experimental observations is central to the scientific method. A particularly challenging setting is when the model is strongly indeterminate, i.e., when distinct sets of parameters yield identical observations. We present a method for cracking such indeterminacy by exploiting additional information conveyed by an auxiliary set of observations sharing global parameters.
  arXiv  Detail & Related papers  (2021-02-12T12:23:13Z)
• Variational Hyper RNN for Sequence Modeling [69.0659591456772]
  We propose a novel probabilistic sequence model that excels at capturing high variability in time series data. Our method uses temporal latent variables to capture information about the underlying data pattern. The efficacy of the proposed method is demonstrated on a range of synthetic and real-world sequential data.
  arXiv  Detail & Related papers  (2020-02-24T19:30:32Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.