SymFormer: End-to-end symbolic regression using transformer-based architecture

• URL: http://arxiv.org/abs/2205.15764v2
• Date: Wed, 1 Jun 2022 10:46:37 GMT
• Title: SymFormer: End-to-end symbolic regression using transformer-based architecture
• Authors: Martin Vastl, Jon\'a\v{s} Kulh\'anek, Ji\v{r}\'i Kubal\'ik, Erik Derner, Robert Babu\v{s}ka
• Abstract summary: We propose a transformer-based approach called SymFormer, which predicts the formula by outputting the individual symbols and the corresponding constants simultaneously. We show on a set of benchmarks that SymFormer outperforms two state-of-the-art methods while having faster inference.
• Score: 2.2049183478692584
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Many real-world problems can be naturally described by mathematical formulas. The task of finding formulas from a set of observed inputs and outputs is called symbolic regression. Recently, neural networks have been applied to symbolic regression, among which the transformer-based ones seem to be the most promising. After training the transformer on a large number of formulas (in the order of days), the actual inference, i.e., finding a formula for new, unseen data, is very fast (in the order of seconds). This is considerably faster than state-of-the-art evolutionary methods. The main drawback of transformers is that they generate formulas without numerical constants, which have to be optimized separately, so yielding suboptimal results. We propose a transformer-based approach called SymFormer, which predicts the formula by outputting the individual symbols and the corresponding constants simultaneously. This leads to better performance in terms of fitting the available data. In addition, the constants provided by SymFormer serve as a good starting point for subsequent tuning via gradient descent to further improve the performance. We show on a set of benchmarks that SymFormer outperforms two state-of-the-art methods while having faster inference.

Related papers

• Paraformer: Fast and Accurate Parallel Transformer for Non-autoregressive End-to-End Speech Recognition [62.83832841523525]
  We propose a fast and accurate parallel transformer, termed Paraformer. It accurately predicts the number of output tokens and extract hidden variables. It can attain comparable performance to the state-of-the-art AR transformer, with more than 10x speedup.
  arXiv  Detail & Related papers  (2022-06-16T17:24:14Z)
• Temporal Latent Bottleneck: Synthesis of Fast and Slow Processing Mechanisms in Sequence Learning [85.95599675484341]
  Recurrent neural networks have a strong inductive bias towards learning temporally compressed representations. Transformers have little inductive bias towards learning temporally compressed representations.
  arXiv  Detail & Related papers  (2022-05-30T00:12:33Z)
• End-to-end symbolic regression with transformers [20.172752966322214]
  Symbolic magnitude regression is a difficult task which usually involves predicting the two-step procedure faster. We show that our model approaches the end-to-end approach Neural the constants as an informed Transformer.
  arXiv  Detail & Related papers  (2022-04-22T06:55:43Z)
• Iterative Decoding for Compositional Generalization in Transformers [5.269770493488338]
  In sequence-to-sequence learning, transformers are often unable to predict correct outputs for even marginally longer examples. This paper introduces iterative decoding, an alternative to seq2seq learning. We show that transfomers trained via iterative decoding outperform their seq2seq counterparts on the PCFG dataset.
  arXiv  Detail & Related papers  (2021-10-08T14:52:25Z)
• nnFormer: Interleaved Transformer for Volumetric Segmentation [50.10441845967601]
  We introduce nnFormer, a powerful segmentation model with an interleaved architecture based on empirical combination of self-attention and convolution. nnFormer achieves tremendous improvements over previous transformer-based methods on two commonly used datasets Synapse and ACDC.
  arXiv  Detail & Related papers  (2021-09-07T17:08:24Z)
• Neural Symbolic Regression that Scales [58.45115548924735]
  We introduce the first symbolic regression method that leverages large scale pre-training. We procedurally generate an unbounded set of equations, and simultaneously pre-train a Transformer to predict the symbolic equation from a corresponding set of input-output-pairs.
  arXiv  Detail & Related papers  (2021-06-11T14:35:22Z)
• High-performance symbolic-numerics via multiple dispatch [52.77024349608834]
  Symbolics.jl is an extendable symbolic system which uses dynamic multiple dispatch to change behavior depending on the domain needs. We show that by formalizing a generic API on actions independent of implementation, we can retroactively add optimized data structures to our system. We demonstrate the ability to swap between classical term-rewriting simplifiers and e-graph-based term-rewriting simplifiers.
  arXiv  Detail & Related papers  (2021-05-09T14:22:43Z)
• Finetuning Pretrained Transformers into RNNs [81.72974646901136]
  Transformers have outperformed recurrent neural networks (RNNs) in natural language generation. A linear-complexity recurrent variant has proven well suited for autoregressive generation. This work aims to convert a pretrained transformer into its efficient recurrent counterpart.
  arXiv  Detail & Related papers  (2021-03-24T10:50:43Z)
• AI Feynman 2.0: Pareto-optimal symbolic regression exploiting graph modularity [8.594811303203581]
  We present an improved method for symbolic regression that seeks to fit data to formulas that are Pareto-optimal. It improves on the previous state-of-the-art by typically being orders of magnitude more robust toward noise and bad data. We develop a method for discovering generalized symmetries from gradient properties of a neural network fit.
  arXiv  Detail & Related papers  (2020-06-18T18:01:19Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.