Differentiable Genetic Programming for High-dimensional Symbolic Regression

• URL: http://arxiv.org/abs/2304.08915v1
• Date: Tue, 18 Apr 2023 11:39:45 GMT
• Title: Differentiable Genetic Programming for High-dimensional Symbolic Regression
• Authors: Peng Zeng, Xiaotian Song, Andrew Lensen, Yuwei Ou, Yanan Sun, Mengjie Zhang, Jiancheng Lv
• Abstract summary: Symbolic regression (SR) is considered an effective way to reach interpretable machine learning (ML) Genetic programming (GP) has been the dominator in solving SR problems. We propose a differentiable approach named DGP to construct GP trees towards high-dimensional SR.
• Score: 13.230237932229052
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Symbolic regression (SR) is the process of discovering hidden relationships from data with mathematical expressions, which is considered an effective way to reach interpretable machine learning (ML). Genetic programming (GP) has been the dominator in solving SR problems. However, as the scale of SR problems increases, GP often poorly demonstrates and cannot effectively address the real-world high-dimensional problems. This limitation is mainly caused by the stochastic evolutionary nature of traditional GP in constructing the trees. In this paper, we propose a differentiable approach named DGP to construct GP trees towards high-dimensional SR for the first time. Specifically, a new data structure called differentiable symbolic tree is proposed to relax the discrete structure to be continuous, thus a gradient-based optimizer can be presented for the efficient optimization. In addition, a sampling method is proposed to eliminate the discrepancy caused by the above relaxation for valid symbolic expressions. Furthermore, a diversification mechanism is introduced to promote the optimizer escaping from local optima for globally better solutions. With these designs, the proposed DGP method can efficiently search for the GP trees with higher performance, thus being capable of dealing with high-dimensional SR. To demonstrate the effectiveness of DGP, we conducted various experiments against the state of the arts based on both GP and deep neural networks. The experiment results reveal that DGP can outperform these chosen peer competitors on high-dimensional regression benchmarks with dimensions varying from tens to thousands. In addition, on the synthetic SR problems, the proposed DGP method can also achieve the best recovery rate even with different noisy levels. It is believed this work can facilitate SR being a powerful alternative to interpretable ML for a broader range of real-world problems.

Related papers

• A Functional Analysis Approach to Symbolic Regression [0.990319860068191]
  Symbolic regression (SR) poses a significant challenge for randomized searchs. Traditional genetic programming (GP) algorithms exhibit limited performance when tree-based representations are used for SR. We introduce a novel SR approach that draws insights from functional analysis.
  arXiv  Detail & Related papers  (2024-02-09T10:24:47Z)
• Domain Invariant Learning for Gaussian Processes and Bayesian Exploration [39.83530605880014]
  We propose a domain invariant learning algorithm for Gaussian processes (DIL-GP) with a min-max optimization on the likelihood. Numerical experiments demonstrate the superiority of DIL-GP for predictions on several synthetic and real-world datasets.
  arXiv  Detail & Related papers  (2023-12-18T16:13:34Z)
• GFN-SR: Symbolic Regression with Generative Flow Networks [0.9208007322096533]
  In recent years, deep symbolic regression (DSR) has emerged as a popular method in the field. We propose an alternative framework (GFN-SR) to approach SR with deep learning. GFN-SR is capable of generating a diverse set of best-fitting expressions.
  arXiv  Detail & Related papers  (2023-12-01T07:38:05Z)
• Model-Based Reparameterization Policy Gradient Methods: Theory and Practical Algorithms [88.74308282658133]
  Reization (RP) Policy Gradient Methods (PGMs) have been widely adopted for continuous control tasks in robotics and computer graphics. Recent studies have revealed that, when applied to long-term reinforcement learning problems, model-based RP PGMs may experience chaotic and non-smooth optimization landscapes. We propose a spectral normalization method to mitigate the exploding variance issue caused by long model unrolls.
  arXiv  Detail & Related papers  (2023-10-30T18:43:21Z)
• ParFam -- (Neural Guided) Symbolic Regression Based on Continuous Global Optimization [14.146976111782466]
  We present our new approach ParFam to translate the discrete symbolic regression problem into a continuous one. In combination with a global, this approach results in a highly effective method to tackle the problem of SR. We also present an extension incorporating a pre-trained transformer network DL-ParFam to guide ParFam.
  arXiv  Detail & Related papers  (2023-10-09T09:01:25Z)
• An Optimization-based Deep Equilibrium Model for Hyperspectral Image Deconvolution with Convergence Guarantees [71.57324258813675]
  We propose a novel methodology for addressing the hyperspectral image deconvolution problem. A new optimization problem is formulated, leveraging a learnable regularizer in the form of a neural network. The derived iterative solver is then expressed as a fixed-point calculation problem within the Deep Equilibrium framework.
  arXiv  Detail & Related papers  (2023-06-10T08:25:16Z)
• Sparse high-dimensional linear regression with a partitioned empirical Bayes ECM algorithm [62.997667081978825]
  We propose a computationally efficient and powerful Bayesian approach for sparse high-dimensional linear regression. Minimal prior assumptions on the parameters are used through the use of plug-in empirical Bayes estimates. The proposed approach is implemented in the R package probe.
  arXiv  Detail & Related papers  (2022-09-16T19:15:50Z)
• GSR: A Generalized Symbolic Regression Approach [13.606672419862047]
  Generalized Symbolic Regression presented in this paper. We show that our GSR method outperforms several state-of-the-art methods on the well-known Symbolic Regression benchmark problem sets. We highlight the strengths of GSR by introducing SymSet, a new SR benchmark set which is more challenging relative to the existing benchmarks.
  arXiv  Detail & Related papers  (2022-05-31T07:20:17Z)
• Incremental Ensemble Gaussian Processes [53.3291389385672]
  We propose an incremental ensemble (IE-) GP framework, where an EGP meta-learner employs an it ensemble of GP learners, each having a unique kernel belonging to a prescribed kernel dictionary. With each GP expert leveraging the random feature-based approximation to perform online prediction and model update with it scalability, the EGP meta-learner capitalizes on data-adaptive weights to synthesize the per-expert predictions. The novel IE-GP is generalized to accommodate time-varying functions by modeling structured dynamics at the EGP meta-learner and within each GP learner.
  arXiv  Detail & Related papers  (2021-10-13T15:11:25Z)
• Harnessing Heterogeneity: Learning from Decomposed Feedback in Bayesian Modeling [68.69431580852535]
  We introduce a novel GP regression to incorporate the subgroup feedback. Our modified regression has provably lower variance -- and thus a more accurate posterior -- compared to previous approaches. We execute our algorithm on two disparate social problems.
  arXiv  Detail & Related papers  (2021-07-07T03:57:22Z)
• Deep Representational Similarity Learning for analyzing neural signatures in task-based fMRI dataset [81.02949933048332]
  This paper develops Deep Representational Similarity Learning (DRSL), a deep extension of Representational Similarity Analysis (RSA) DRSL is appropriate for analyzing similarities between various cognitive tasks in fMRI datasets with a large number of subjects.
  arXiv  Detail & Related papers  (2020-09-28T18:30:14Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.