Selection, Representation, and Execution in Sparse Fourier Neural Operators

• URL: http://arxiv.org/abs/2608.30070v1
• Date: Sun, 30 Aug 2026 22:26:59 GMT
• Title: Selection, Representation, and Execution in Sparse Fourier Neural Operators
• Abstract summary: Sparse representations are often expected to make models smaller and also reduce inference cost.<n>We distinguish sparsity in the representation, in the stored parameters, in the theoretical operation count, and runtime.<n>We present an empirical study of several routes toward sparse FNOs that tests each transition between them separately.
• Score: 1.8119312186036625
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Sparse representations are often expected to make models smaller and also reduce inference cost. For Fourier Neural Operators (FNOs), these objectives are not equivalent or do not always align: removing parts of the learned operator can leave the underlying transforms and dense computations unchanged, while changing the grid on which the model is evaluated can introduce overhead of its own. We therefore distinguish sparsity in the representation, in the stored parameters, in the theoretical operation count, and in measured runtime, and present an empirical study of several routes toward sparse FNOs that tests each transition between them separately. Coarsening the execution grid reduces the theoretical cost without reducing measured latency, and adding a correction term recovers accuracy at the cost of making the model slower. Even an 83\% parameter reduction remains slower than the dense baseline under ordinary execution. These results motivate a stricter definition of useful sparsity: the deployed operator must preserve solution accuracy and map its reduced support to a genuinely cheaper execution path.

Related papers

• Penelope: Localized Latent Recurrence for Efficient Structured Reasoning [26.549641831847982]
  Penelope is an efficient latent-reasoning framework for pretrained decoder-only Transformers.<n>It localizes recurrent computation to a selected decoder interval.<n>It attains competitive accuracy relative to established latent-reasoning models.
  arXiv  Detail & Related papers  (2026-07-28T16:06:46Z)
• Randomized neural operator for parametric PDEs with fast training and conformal uncertainty quantification [47.720075875232226]
  We introduce PCA--RaNN, a neural operator that combines a PCA-based dimensionality reduction with fixed random features.<n>It recasts operator learning as fixed regression, reducing latent regression to three orders of magnitude.
  arXiv  Detail & Related papers  (2026-06-28T15:02:26Z)
• Train Less, Infer Faster: Efficient Model Finetuning and Compression via Structured Sparsity [21.090365337326414]
  Finetuning foundation language models (LMs) with billions of parameters is often impractical due to high computational costs, memory requirements, and the risk of overfitting.<n>We propose a scheme for effective finetuning via sparsification using training gates, which requires minimal trainable parameters.<n> Empirical results show it outperforms recent finetuning baselines in efficiency and performance.
  arXiv  Detail & Related papers  (2026-02-09T20:20:29Z)
• Repulsor: Accelerating Generative Modeling with a Contrastive Memory Bank [65.00301565190824]
  mname is a plug-and-play training framework that requires no external encoders.<n>mname achieves a state-of-the-art FID of textbf2.40 within 400k steps, significantly outperforming comparable methods.
  arXiv  Detail & Related papers  (2025-12-09T14:39:26Z)
• Sparse Attention Post-Training for Mechanistic Interpretability [55.030850996535776]
  We introduce a simple post-training method that makes transformer attention sparse without sacrificing performance.<n>Applying a flexible sparsity regularisation under a constrained-loss objective, we show on models up to 1B parameters that it is possible to retain the original pretraining loss while reducing attention connectivity to $approx 0.3 % of its edges.
  arXiv  Detail & Related papers  (2025-12-05T16:40:08Z)
• Accelerating PDE Solvers with Equation-Recast Neural Operator Preconditioning [9.178290601589365]
  Minimal-Data Parametric Neural Operator Preconditioning (MD-PNOP) is a new paradigm for accelerating parametric PDE solvers.<n>It recasts the residual from parameter deviation as additional source term, where trained neural operators can be used to refine the solution in an offline fashion.<n>It consistently achieves 50% reduction in computational time while maintaining full order fidelity for fixed-source, single-group eigenvalue, and multigroup coupled eigenvalue problems.
  arXiv  Detail & Related papers  (2025-09-01T12:14:58Z)
• Guaranteed Approximation Bounds for Mixed-Precision Neural Operators [83.64404557466528]
  We build on intuition that neural operator learning inherently induces an approximation error. We show that our approach reduces GPU memory usage by up to 50% and improves throughput by 58% with little or no reduction in accuracy.
  arXiv  Detail & Related papers  (2023-07-27T17:42:06Z)
• Learning Decorrelated Representations Efficiently Using Fast Fourier Transform [3.932322649674071]
  We propose a relaxed decorrelating regularizer that can be computed in O(n d log d) time by Fast Fourier Transform. The proposed regularizer exhibits accuracy comparable to that of existing regularizers in downstream tasks, whereas their training requires less memory and is faster for large d.
  arXiv  Detail & Related papers  (2023-01-04T12:38:08Z)
• FasterPose: A Faster Simple Baseline for Human Pose Estimation [65.8413964785972]
  We propose a design paradigm for cost-effective network with LR representation for efficient pose estimation, named FasterPose. We study the training behavior of FasterPose, and formulate a novel regressive cross-entropy (RCE) loss function for accelerating the convergence. Compared with the previously dominant network of pose estimation, our method reduces 58% of the FLOPs and simultaneously gains 1.3% improvement of accuracy.
  arXiv  Detail & Related papers  (2021-07-07T13:39:08Z)
• Age-Based Coded Computation for Bias Reduction in Distributed Learning [57.9123881133818]
  Coded computation can be used to speed up distributed learning in the presence of straggling workers. Partial recovery of the gradient vector can further reduce the computation time at each iteration. Estimator bias will be particularly prevalent when the straggling behavior is correlated over time.
  arXiv  Detail & Related papers  (2020-06-02T17:51:11Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.