In-Context Symbolic Regression for Robustness-Improved Kolmogorov-Arnold Networks

• URL: http://arxiv.org/abs/2603.15250v1
• Date: Mon, 16 Mar 2026 13:21:26 GMT
• Title: In-Context Symbolic Regression for Robustness-Improved Kolmogorov-Arnold Networks
• Abstract summary: Symbolic regression aims to replace blackbox predictors with analytical expressions that can be inspected and validated in machine learning.<n> Kolmogorov-Arnold Pursuit Networks (KANs) are well suited to this goal because each connection between adjacent units (an "edge") ismetd by a univariate function.<n>In practice, however, symbolic extraction is a bottleneck: the standard KAN-to-symbol approach fits operators to each learned edge function in fine isolation.
• Score: 3.7163623763519733
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Symbolic regression aims to replace black-box predictors with concise analytical expressions that can be inspected and validated in scientific machine learning. Kolmogorov-Arnold Networks (KANs) are well suited to this goal because each connection between adjacent units (an "edge") is parametrised by a learnable univariate function that can, in principle, be replaced by a symbolic operator. In practice, however, symbolic extraction is a bottleneck: the standard KAN-to-symbol approach fits operators to each learned edge function in isolation, making the discrete choice sensitive to initialisation and non-convex parameter fitting, and ignoring how local substitutions interact through the full network. We study in-context symbolic regression for operator extraction in KANs, and present two complementary instantiations. Greedy in-context Symbolic Regression (GSR) performs greedy, in-context selection by choosing edge replacements according to end-to-end loss improvement after brief fine-tuning. Gated Matching Pursuit (GMP) amortises this in-context selection by training a differentiable gated operator layer that places an operator library behind sparse gates on each edge; after convergence, gates are discretised (optionally followed by a short in-context greedy refinement pass). We quantify robustness via one-factor-at-a-time (OFAT) hyper-parameter sweeps and assess both predictive error and qualitative consistency of recovered formulas. Across several experiments, greedy in-context symbolic regression achieves up to 99.8% reduction in median OFAT test MSE.

Related papers

• Phase Transition in Binary Compressed Sensing via Annealing with Adaptive Regularization [0.7734726150561088]
  Regularization choice changes the recovery phase diagrams of binary compressed sensing.<n>We develop a regularization-selection method that combines systematic parameter search with random forest regression.
  arXiv  Detail & Related papers  (2026-09-15T06:35:58Z)
• Bayesian Symbolic Regression with Entropic Reinforcement Learning [40.36303059396686]
  We propose ERRLESS, a scalable approach for sampling from the posterior distribution over expressions given data.<n>ERRLESS learns a neural policy that constructs expressions sequentially by building up their abstract syntax trees.<n>We demonstrate that ERRLESS achieves competitive results on the Feynman benchmark while producing short and interpretable expressions.
  arXiv  Detail & Related papers  (2026-08-10T13:56:45Z)
• Sparse Weight Decomposition for Efficient Circuit Extraction [36.87892448834478]
  Dense pretrained transformers do not naturally expose interpretable units for circuit extraction.<n>We propose Sparse Weight Decomposition, which factorizes pretrained projections by factorizing each weight matrix into two sparse factors.<n>We show that SWD remains effective for full-model replacement of all attention and weight matrices after fine-tuning the nonzero factor values.
  arXiv  Detail & Related papers  (2026-08-04T16:40:48Z)
• C$^{2}$R: Cross-sample Consistency Regularization Mitigates Feature Splitting and Absorption in Sparse Autoencoders [56.67585330425431]
  Sparse Autoencoders (SAEs) are widely used to interpret large language models by decomposing activations into sparse, human-understandable features.<n>These issues stem from inconsistent latent assignment across samples.<n>We introduce C$2$R, which penalizes the co-activation of directionally similar latents.
  arXiv  Detail & Related papers  (2026-06-29T17:45:31Z)
• Symbolic Regression via Latent Iterative Refinement [8.016092717336457]
  We propose Latent Equation Embedding (LEE), a framework that closes the gap between a one-shot prediction and the true posterior.<n>LEE learns a shared latent space Z equipped with three components.<n>LEE produces expressions 2--10x simpler than the strongest accuracy-oriented baselines.
  arXiv  Detail & Related papers  (2026-05-26T16:25:05Z)
• GENSR: Symbolic Regression Based in Equation Generative Space [15.186848349610363]
  GenSR is a generative latent space-based SR framework.<n>From a Bayesian perspective, GenSR reframes the SR task as maximizing the conditional distribution $p(mathrmEqu. mid mathrmNum.)$.
  arXiv  Detail & Related papers  (2026-02-24T05:14:34Z)
• Gaussian Match-and-Copy: A Minimalist Benchmark for Studying Transformer Induction [44.83333974000826]
  We introduce a minimalist benchmark that isolates long-range retrieval through pure second-order correlation signals.<n> Numerical investigations show that this task retains key qualitative aspects of how Transformers develop match-and-copy circuits.<n>We prove this max-margin alignment for GD trajectories that reach vanishing empirical loss under explicit technical conditions.
  arXiv  Detail & Related papers  (2026-02-07T14:18:11Z)
• Verifying Closed-Loop Contractivity of Learning-Based Controllers via Partitioning [52.23804865017831]
  We address the problem of verifying closed-loop contraction in nonlinear control systems whose controller and contraction metric are both parameterized by neural networks.<n>We derive a tractable and scalable sufficient condition for closed-loop contractivity that reduces to checking that the dominant eigenvalue of a symmetric Metzler matrix is nonpositive.
  arXiv  Detail & Related papers  (2025-12-01T23:06:56Z)
• Parameter-Aware Ensemble SINDy for Interpretable Symbolic SGS Closure [0.0]
  This work designs a scalable, parameter-aware sparse regression framework for discovering interpretable partial differential equations and subgridscale closures.<n>Building on SINDy (Sparse Identification of Dynamics), the approach addresses key limitations through four enhancements.
  arXiv  Detail & Related papers  (2025-08-13T09:37:57Z)
• PseudoNeg-MAE: Self-Supervised Point Cloud Learning using Conditional Pseudo-Negative Embeddings [55.55445978692678]
  PseudoNeg-MAE enhances global feature representation of point cloud masked autoencoders by making them both discriminative and sensitive to transformations.<n>We propose a novel loss that explicitly penalizes invariant collapse, enabling the network to capture richer transformation cues while preserving discriminative representations.
  arXiv  Detail & Related papers  (2024-09-24T07:57:21Z)
• Parseval Convolution Operators and Neural Networks [16.78532039510369]
  We first identify the Parseval convolution operators as the class of energy-preserving filterbanks. We then present a constructive approach for the design/specification of such filterbanks via the chaining of elementary Parseval modules. We demonstrate the usage of those tools with the design of a CNN-based algorithm for the iterative reconstruction of biomedical images.
  arXiv  Detail & Related papers  (2024-08-19T13:31:16Z)
• ISR: Invertible Symbolic Regression [7.499800486499609]
  Invertible Symbolic Regression is a machine learning technique that generates analytical relationships between inputs and outputs of a given dataset via invertible maps. We transform the affine coupling blocks of INNs into a symbolic framework, resulting in an end-to-end differentiable symbolic invertible architecture. We show that ISR can serve as a (symbolic) normalizing flow for density estimation tasks.
  arXiv  Detail & Related papers  (2024-05-10T23:20:46Z)
• On the Effective Number of Linear Regions in Shallow Univariate ReLU Networks: Convergence Guarantees and Implicit Bias [50.84569563188485]
  We show that gradient flow converges in direction when labels are determined by the sign of a target network with $r$ neurons. Our result may already hold for mild over- parameterization, where the width is $tildemathcalO(r)$ and independent of the sample size.
  arXiv  Detail & Related papers  (2022-05-18T16:57:10Z)
• Autoencoding Variational Autoencoder [56.05008520271406]
  We study the implications of this behaviour on the learned representations and also the consequences of fixing it by introducing a notion of self consistency. We show that encoders trained with our self-consistency approach lead to representations that are robust (insensitive) to perturbations in the input introduced by adversarial attacks.
  arXiv  Detail & Related papers  (2020-12-07T14:16:14Z)
• Pairwise Supervised Hashing with Bernoulli Variational Auto-Encoder and Self-Control Gradient Estimator [62.26981903551382]
  Variational auto-encoders (VAEs) with binary latent variables provide state-of-the-art performance in terms of precision for document retrieval. We propose a pairwise loss function with discrete latent VAE to reward within-class similarity and between-class dissimilarity for supervised hashing. This new semantic hashing framework achieves superior performance compared to the state-of-the-arts.
  arXiv  Detail & Related papers  (2020-05-21T06:11:33Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.