Univariate Skeleton Prediction in Multivariate Systems Using Transformers

• URL: http://arxiv.org/abs/2406.17834v1
• Date: Tue, 25 Jun 2024 15:07:06 GMT
• Title: Univariate Skeleton Prediction in Multivariate Systems Using Transformers
• Authors: Giorgio Morales, John W. Sheppard,
• Abstract summary: We propose an explainable neural SR method that generates univariate symbolic skeletons that aim to explain how each variable influences the system's response. Experimental results demonstrate that this method learns skeleton expressions matching the underlying functions and outperforms two GP-based and two neural SR methods.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Symbolic regression (SR) methods attempt to learn mathematical expressions that approximate the behavior of an observed system. However, when dealing with multivariate systems, they often fail to identify the functional form that explains the relationship between each variable and the system's response. To begin to address this, we propose an explainable neural SR method that generates univariate symbolic skeletons that aim to explain how each variable influences the system's response. By analyzing multiple sets of data generated artificially, where one input variable varies while others are fixed, relationships are modeled separately for each input variable. The response of such artificial data sets is estimated using a regression neural network (NN). Finally, the multiple sets of input-response pairs are processed by a pre-trained Multi-Set Transformer that solves a problem we termed Multi-Set Skeleton Prediction and outputs a univariate symbolic skeleton. Thus, such skeletons represent explanations of the function approximated by the regression NN. Experimental results demonstrate that this method learns skeleton expressions matching the underlying functions and outperforms two GP-based and two neural SR methods.

Related papers

• On the Mechanism and Dynamics of Modular Addition: Fourier Features, Lottery Ticket, and Grokking [49.1352577985191]
  We present a comprehensive analysis of how two-layer neural networks learn features to solve the modular addition task.<n>Our work provides a full mechanistic interpretation of the learned model and a theoretical explanation of its training dynamics.
  arXiv  Detail & Related papers  (2026-02-18T20:25:13Z)
• Decomposable Neuro Symbolic Regression [8.450904497835262]
  We present a decomposable Symbolic Regression (SR) method that generates interpretable multivariate expressions.<n>Our method distills a trained opaque'' regression model into mathematical expressions that serve as explanations of its computed function.
  arXiv  Detail & Related papers  (2025-11-06T07:17:36Z)
• PROSE: Predicting Operators and Symbolic Expressions using Multimodal Transformers [5.263113622394007]
  We develop a new neural network framework for predicting differential equations. By using a transformer structure and a feature fusion approach, our network can simultaneously embed sets of solution operators for various parametric differential equations. The network is shown to be able to handle noise in the data and errors in the symbolic representation, including noisy numerical values, model misspecification, and erroneous addition or deletion of terms.
  arXiv  Detail & Related papers  (2023-09-28T19:46:07Z)
• Unsupervised Learning of Invariance Transformations [105.54048699217668]
  We develop an algorithmic framework for finding approximate graph automorphisms. We discuss how this framework can be used to find approximate automorphisms in weighted graphs in general.
  arXiv  Detail & Related papers  (2023-07-24T17:03:28Z)
• Scalable Neural Symbolic Regression using Control Variables [7.725394912527969]
  We propose ScaleSR, a scalable symbolic regression model that leverages control variables to enhance both accuracy and scalability. The proposed method involves a four-step process. First, we learn a data generator from observed data using deep neural networks (DNNs) Experimental results demonstrate that the proposed ScaleSR significantly outperforms state-of-the-art baselines in discovering mathematical expressions with multiple variables.
  arXiv  Detail & Related papers  (2023-06-07T18:30:25Z)
• Toward Physically Plausible Data-Driven Models: A Novel Neural Network Approach to Symbolic Regression [2.7071541526963805]
  This paper proposes a novel neural network-based symbolic regression method. It constructs physically plausible models based on even very small training data sets and prior knowledge about the system. We experimentally evaluate the approach on four test systems: the TurtleBot 2 mobile robot, the magnetic manipulation system, the equivalent resistance of two resistors in parallel, and the longitudinal force of the anti-lock braking system.
  arXiv  Detail & Related papers  (2023-02-01T22:05:04Z)
• A predictive physics-aware hybrid reduced order model for reacting flows [65.73506571113623]
  A new hybrid predictive Reduced Order Model (ROM) is proposed to solve reacting flow problems. The number of degrees of freedom is reduced from thousands of temporal points to a few POD modes with their corresponding temporal coefficients. Two different deep learning architectures have been tested to predict the temporal coefficients.
  arXiv  Detail & Related papers  (2023-01-24T08:39:20Z)
• Capturing Actionable Dynamics with Structured Latent Ordinary Differential Equations [68.62843292346813]
  We propose a structured latent ODE model that captures system input variations within its latent representation. Building on a static variable specification, our model learns factors of variation for each input to the system, thus separating the effects of the system inputs in the latent space.
  arXiv  Detail & Related papers  (2022-02-25T20:00:56Z)
• Identification of Dynamical Systems using Symbolic Regression [0.0]
  We describe a method for the identification of models for dynamical systems from observational data. The novelty is that we add a step of gradient-based optimization of the ODE parameters. We find that gradient-based optimization of parameters improves predictive accuracy of the models.
  arXiv  Detail & Related papers  (2021-07-06T11:41:10Z)
• The MELODIC family for simultaneous binary logistic regression in a reduced space [0.5330240017302619]
  We propose the MELODIC family for simultaneous binary logistic regression modeling. The model may be interpreted in terms of logistic regression coefficients or in terms of a biplot. Two applications are shown in detail: one relating personality characteristics to drug consumption profiles and one relating personality characteristics to depressive and anxiety disorders.
  arXiv  Detail & Related papers  (2021-02-16T15:47:20Z)
• Provably Efficient Neural Estimation of Structural Equation Model: An Adversarial Approach [144.21892195917758]
  We study estimation in a class of generalized Structural equation models (SEMs) We formulate the linear operator equation as a min-max game, where both players are parameterized by neural networks (NNs), and learn the parameters of these neural networks using a gradient descent. For the first time we provide a tractable estimation procedure for SEMs based on NNs with provable convergence and without the need for sample splitting.
  arXiv  Detail & Related papers  (2020-07-02T17:55:47Z)
• Multipole Graph Neural Operator for Parametric Partial Differential Equations [57.90284928158383]
  One of the main challenges in using deep learning-based methods for simulating physical systems is formulating physics-based data. We propose a novel multi-level graph neural network framework that captures interaction at all ranges with only linear complexity. Experiments confirm our multi-graph network learns discretization-invariant solution operators to PDEs and can be evaluated in linear time.
  arXiv  Detail & Related papers  (2020-06-16T21:56:22Z)
• Variational Transformers for Diverse Response Generation [71.53159402053392]
  Variational Transformer (VT) is a variational self-attentive feed-forward sequence model. VT combines the parallelizability and global receptive field computation of the Transformer with the variational nature of the CVAE. We explore two types of VT: 1) modeling the discourse-level diversity with a global latent variable; and 2) augmenting the Transformer decoder with a sequence of finegrained latent variables.
  arXiv  Detail & Related papers  (2020-03-28T07:48:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.