Sparse-Aware Neural Networks for Nonlinear Functionals: Mitigating the Exponential Dependence on Dimension

• URL: http://arxiv.org/abs/2604.06774v1
• Date: Wed, 08 Apr 2026 07:42:02 GMT
• Title: Sparse-Aware Neural Networks for Nonlinear Functionals: Mitigating the Exponential Dependence on Dimension
• Abstract summary: We propose a framework that employs convolutional architectures to extract sparse features from a finite number of samples.<n>We show that sparse approximators enable stable recovery from discrete samples.<n>They also provide new theoretical insights into how sparsity can alleviate the curse of dimensionality in functional learning.
• Score: 27.6052958612001
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Deep neural networks have emerged as powerful tools for learning operators defined over infinite-dimensional function spaces. However, existing theories frequently encounter difficulties related to dimensionality and limited interpretability. This work investigates how sparsity can help address these challenges in functional learning, a central ingredient in operator learning. We propose a framework that employs convolutional architectures to extract sparse features from a finite number of samples, together with deep fully connected networks to effectively approximate nonlinear functionals. Using universal discretization methods, we show that sparse approximators enable stable recovery from discrete samples. In addition, both the deterministic and the random sampling schemes are sufficient for our analysis. These findings lead to improved approximation rates and reduced sample sizes in various function spaces, including those with fast frequency decay and mixed smoothness. They also provide new theoretical insights into how sparsity can alleviate the curse of dimensionality in functional learning.

Related papers

• Functional Equivalence and Geometric Diversity in Neural Network Approximations: An Empirical Characterization [0.0]
  We analyze functional equivalence and geometric diversity of neural network approximations to a few elementary mathematical functions.<n>The study reveals large equivalence classes of functionally indistinguishable yet geometrically diverse networks.<n>A model select criterion is proposed for identifying optimal models based on parsimony, ease of estimation, and inference efficiency.
  arXiv  Detail & Related papers  (2026-07-21T10:12:03Z)
• Asymptotic Learning Curves for Diffusion Models with Random Features Score and Manifold Data [7.047721358838963]
  We study the theoretical behavior of denoising score matching.<n>We derive exact expressions for the test, train, and score errors in the high-dimensional limit.
  arXiv  Detail & Related papers  (2026-03-24T08:58:12Z)
• Neural Approximation and Its Applications [73.11802084858294]
  We introduce the neural basis function by leveraging an untrained neural network as the basis function.<n>We suggest the neural approximation (NeuApprox) paradigm for multivariate function approximation.<n>Experiments on diverse multi-dimensional images, light field data, videos, traffic data, and point cloud data demonstrate the promising performance of NeuApprox.
  arXiv  Detail & Related papers  (2026-03-04T06:05:43Z)
• Global Convergence and Rich Feature Learning in $L$-Layer Infinite-Width Neural Networks under $μ$P Parametrization [66.03821840425539]
  In this paper, we investigate the training dynamics of $L$-layer neural networks using the tensor gradient program (SGD) framework.<n>We show that SGD enables these networks to learn linearly independent features that substantially deviate from their initial values.<n>This rich feature space captures relevant data information and ensures that any convergent point of the training process is a global minimum.
  arXiv  Detail & Related papers  (2025-03-12T17:33:13Z)
• Explainable Neural Networks with Guarantees: A Sparse Estimation Approach [11.142723510517778]
  This paper introduces a novel approach to constructing an explainable neural network that harmonizes predictiveness and explainability.<n>Our model, termed SparXnet, is designed as a linear combination of a sparse set of jointly learned features.<n>Our research paves the way for further research on sparse and explainable neural networks with guarantee.
  arXiv  Detail & Related papers  (2025-01-02T12:10:17Z)
• Spherical Analysis of Learning Nonlinear Functionals [10.785977740158193]
  In this paper, we consider functionals defined on sets of functions on spheres. The approximation ability of deep ReLU neural networks is investigated using an encoder-decoder framework.
  arXiv  Detail & Related papers  (2024-10-01T20:10:00Z)
• Nonlinear functional regression by functional deep neural network with kernel embedding [18.927592350748682]
  We introduce a functional deep neural network with an adaptive and discretization-invariant dimension reduction method.<n>Explicit rates of approximating nonlinear smooth functionals across various input function spaces are derived.<n>We conduct numerical experiments on both simulated and real datasets to demonstrate the effectiveness and benefits of our functional net.
  arXiv  Detail & Related papers  (2024-01-05T16:43:39Z)
• Provable Multi-Task Representation Learning by Two-Layer ReLU Neural Networks [69.38572074372392]
  We present the first results proving that feature learning occurs during training with a nonlinear model on multiple tasks. Our key insight is that multi-task pretraining induces a pseudo-contrastive loss that favors representations that align points that typically have the same label across tasks.
  arXiv  Detail & Related papers  (2023-07-13T16:39:08Z)
• Nonparametric Classification on Low Dimensional Manifolds using Overparameterized Convolutional Residual Networks [78.11734286268455]
  We study the performance of ConvResNeXts, trained with weight decay from the perspective of nonparametric classification.<n>Our analysis allows for infinitely many building blocks in ConvResNeXts, and shows that weight decay implicitly enforces sparsity on these blocks.
  arXiv  Detail & Related papers  (2023-07-04T11:08:03Z)
• Offline Reinforcement Learning with Differentiable Function Approximation is Provably Efficient [65.08966446962845]
  offline reinforcement learning, which aims at optimizing decision-making strategies with historical data, has been extensively applied in real-life applications. We take a step by considering offline reinforcement learning with differentiable function class approximation (DFA) Most importantly, we show offline differentiable function approximation is provably efficient by analyzing the pessimistic fitted Q-learning algorithm.
  arXiv  Detail & Related papers  (2022-10-03T07:59:42Z)
• Stabilizing Q-learning with Linear Architectures for Provably Efficient Learning [53.17258888552998]
  This work proposes an exploration variant of the basic $Q$-learning protocol with linear function approximation. We show that the performance of the algorithm degrades very gracefully under a novel and more permissive notion of approximation error.
  arXiv  Detail & Related papers  (2022-06-01T23:26:51Z)
• Generalization of Neural Combinatorial Solvers Through the Lens of Adversarial Robustness [68.97830259849086]
  Most datasets only capture a simpler subproblem and likely suffer from spurious features. We study adversarial robustness - a local generalization property - to reveal hard, model-specific instances and spurious features. Unlike in other applications, where perturbation models are designed around subjective notions of imperceptibility, our perturbation models are efficient and sound. Surprisingly, with such perturbations, a sufficiently expressive neural solver does not suffer from the limitations of the accuracy-robustness trade-off common in supervised learning.
  arXiv  Detail & Related papers  (2021-10-21T07:28:11Z)
• Function Approximation via Sparse Random Features [23.325877475827337]
  This paper introduces the sparse random feature method that learns parsimonious random feature models utilizing techniques from compressive sensing. We show that the sparse random feature method outperforms shallow networks for well-structured functions and applications to scientific machine learning tasks.
  arXiv  Detail & Related papers  (2021-03-04T17:53:54Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.