Deep Generative Symbolic Regression
- URL: http://arxiv.org/abs/2401.00282v1
- Date: Sat, 30 Dec 2023 17:05:31 GMT
- Title: Deep Generative Symbolic Regression
- Authors: Samuel Holt, Zhaozhi Qian, Mihaela van der Schaar
- Abstract summary: Symbolic regression aims to discover concise closed-form mathematical equations from data.
Existing methods, ranging from search to reinforcement learning, fail to scale with the number of input variables.
We propose an instantiation of our framework, Deep Generative Symbolic Regression.
- Score: 83.04219479605801
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: Symbolic regression (SR) aims to discover concise closed-form mathematical
equations from data, a task fundamental to scientific discovery. However, the
problem is highly challenging because closed-form equations lie in a complex
combinatorial search space. Existing methods, ranging from heuristic search to
reinforcement learning, fail to scale with the number of input variables. We
make the observation that closed-form equations often have structural
characteristics and invariances (e.g., the commutative law) that could be
further exploited to build more effective symbolic regression solutions.
Motivated by this observation, our key contribution is to leverage pre-trained
deep generative models to capture the intrinsic regularities of equations,
thereby providing a solid foundation for subsequent optimization steps. We show
that our novel formalism unifies several prominent approaches of symbolic
regression and offers a new perspective to justify and improve on the previous
ad hoc designs, such as the usage of cross-entropy loss during pre-training.
Specifically, we propose an instantiation of our framework, Deep Generative
Symbolic Regression (DGSR). In our experiments, we show that DGSR achieves a
higher recovery rate of true equations in the setting of a larger number of
input variables, and it is more computationally efficient at inference time
than state-of-the-art RL symbolic regression solutions.
Related papers
- Discovering symbolic expressions with parallelized tree search [59.92040079807524]
Symbolic regression plays a crucial role in scientific research thanks to its capability of discovering concise and interpretable mathematical expressions from data.
Existing algorithms have faced a critical bottleneck of accuracy and efficiency over a decade when handling problems of complexity.
We introduce a parallelized tree search (PTS) model to efficiently distill generic mathematical expressions from limited data.
arXiv Detail & Related papers (2024-07-05T10:41:15Z) - Out of the Ordinary: Spectrally Adapting Regression for Covariate Shift [12.770658031721435]
We propose a method for adapting the weights of the last layer of a pre-trained neural regression model to perform better on input data originating from a different distribution.
We demonstrate how this lightweight spectral adaptation procedure can improve out-of-distribution performance for synthetic and real-world datasets.
arXiv Detail & Related papers (2023-12-29T04:15:58Z) - A Transformer Model for Symbolic Regression towards Scientific Discovery [11.827358526480323]
Symbolic Regression (SR) searches for mathematical expressions which best describe numerical datasets.
We propose a new Transformer model aiming at Symbolic Regression particularly focused on its application for Scientific Discovery.
We apply our best model to the SRSD datasets which yields state-of-the-art results using the normalized tree-based edit distance.
arXiv Detail & Related papers (2023-12-07T06:27:48Z) - Understanding Augmentation-based Self-Supervised Representation Learning
via RKHS Approximation and Regression [53.15502562048627]
Recent work has built the connection between self-supervised learning and the approximation of the top eigenspace of a graph Laplacian operator.
This work delves into a statistical analysis of augmentation-based pretraining.
arXiv Detail & Related papers (2023-06-01T15:18:55Z) - Theoretical Characterization of the Generalization Performance of
Overfitted Meta-Learning [70.52689048213398]
This paper studies the performance of overfitted meta-learning under a linear regression model with Gaussian features.
We find new and interesting properties that do not exist in single-task linear regression.
Our analysis suggests that benign overfitting is more significant and easier to observe when the noise and the diversity/fluctuation of the ground truth of each training task are large.
arXiv Detail & Related papers (2023-04-09T20:36:13Z) - Mutual Information Learned Regressor: an Information-theoretic Viewpoint
of Training Regression Systems [10.314518385506007]
An existing common practice for solving regression problems is the mean square error (MSE) minimization approach.
Recently, Yi et al., proposed a mutual information based supervised learning framework where they introduced a label entropy regularization.
In this paper, we investigate the regression under the mutual information based supervised learning framework.
arXiv Detail & Related papers (2022-11-23T03:43:22Z) - Neural Symbolic Regression that Scales [58.45115548924735]
We introduce the first symbolic regression method that leverages large scale pre-training.
We procedurally generate an unbounded set of equations, and simultaneously pre-train a Transformer to predict the symbolic equation from a corresponding set of input-output-pairs.
arXiv Detail & Related papers (2021-06-11T14:35:22Z) - A Hypergradient Approach to Robust Regression without Correspondence [85.49775273716503]
We consider a variant of regression problem, where the correspondence between input and output data is not available.
Most existing methods are only applicable when the sample size is small.
We propose a new computational framework -- ROBOT -- for the shuffled regression problem.
arXiv Detail & Related papers (2020-11-30T21:47:38Z) - AI Feynman 2.0: Pareto-optimal symbolic regression exploiting graph
modularity [8.594811303203581]
We present an improved method for symbolic regression that seeks to fit data to formulas that are Pareto-optimal.
It improves on the previous state-of-the-art by typically being orders of magnitude more robust toward noise and bad data.
We develop a method for discovering generalized symmetries from gradient properties of a neural network fit.
arXiv Detail & Related papers (2020-06-18T18:01:19Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.