An Attention-based Spatio-Temporal Neural Operator for Evolving Physics

• URL: http://arxiv.org/abs/2506.11328v1
• Date: Thu, 12 Jun 2025 21:54:47 GMT
• Title: An Attention-based Spatio-Temporal Neural Operator for Evolving Physics
• Authors: Vispi Karkaria, Doksoo Lee, Yi-Ping Chen, Yue Yu, Wei Chen,
• Abstract summary: Key challenge in machine learning is learning unknown physical processes and making predictions across real-scales.<n>We propose Attention-based S-Temporal Neural Operator (ASNO), a novel architecture that combines separable attention mechanisms for spatial and temporal interactions.<n>ASNO learns a transformer for temporal prediction and adapts to unseen physical parameters, enhancing interpretability by historical state contributions.
• Score: 15.451874867285957
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: In scientific machine learning (SciML), a key challenge is learning unknown, evolving physical processes and making predictions across spatio-temporal scales. For example, in real-world manufacturing problems like additive manufacturing, users adjust known machine settings while unknown environmental parameters simultaneously fluctuate. To make reliable predictions, it is desired for a model to not only capture long-range spatio-temporal interactions from data but also adapt to new and unknown environments; traditional machine learning models excel at the first task but often lack physical interpretability and struggle to generalize under varying environmental conditions. To tackle these challenges, we propose the Attention-based Spatio-Temporal Neural Operator (ASNO), a novel architecture that combines separable attention mechanisms for spatial and temporal interactions and adapts to unseen physical parameters. Inspired by the backward differentiation formula (BDF), ASNO learns a transformer for temporal prediction and extrapolation and an attention-based neural operator for handling varying external loads, enhancing interpretability by isolating historical state contributions and external forces, enabling the discovery of underlying physical laws and generalizability to unseen physical environments. Empirical results on SciML benchmarks demonstrate that ASNO outperforms over existing models, establishing its potential for engineering applications, physics discovery, and interpretable machine learning.

Related papers

• Langevin Flows for Modeling Neural Latent Dynamics [81.81271685018284]
  We introduce LangevinFlow, a sequential Variational Auto-Encoder where the time evolution of latent variables is governed by the underdamped Langevin equation.<n>Our approach incorporates physical priors -- such as inertia, damping, a learned potential function, and forces -- to represent both autonomous and non-autonomous processes in neural systems.<n>Our method outperforms state-of-the-art baselines on synthetic neural populations generated by a Lorenz attractor.
  arXiv  Detail & Related papers  (2025-07-15T17:57:48Z)
• PI-WAN: A Physics-Informed Wind-Adaptive Network for Quadrotor Dynamics Prediction in Unknown Environments [3.4802474792943805]
  We introduce the Physics-Informed Wind-Adaptive Network (PI-WAN), which embeds physical constraints directly into the training process for robust quadrotor dynamics learning.<n>Specifically, PI-WAN employs a Temporal Convolutional Network (TCN) architecture that efficiently captures temporal dependencies from historical flight data.<n>By incorporating real-time prediction results into a model predictive control (MPC) framework, we achieve improvements in closed-loop tracking performance.
  arXiv  Detail & Related papers  (2025-07-01T14:48:22Z)
• ENMA: Tokenwise Autoregression for Generative Neural PDE Operators [12.314585849869797]
  We introduce ENMA, a generative neural-temporal operator designed to model dynamics arising from physical phenomena.<n>ENMA predicts future dynamics compressed latent space using a generative masked autoregressive transformer trained with flow matching loss.<n>The framework generalizes to new PDE regimes and supports one-shot surrogate modeling of time-dependent parametric PDEs.
  arXiv  Detail & Related papers  (2025-06-06T15:25:14Z)
• Neural Operators for Accelerating Scientific Simulations and Design [85.89660065887956]
  An AI framework, known as Neural Operators, presents a principled framework for learning mappings between functions defined on continuous domains. Neural Operators can augment or even replace existing simulators in many applications, such as computational fluid dynamics, weather forecasting, and material modeling.
  arXiv  Detail & Related papers  (2023-09-27T00:12:07Z)
• Spherical Fourier Neural Operators: Learning Stable Dynamics on the Sphere [53.63505583883769]
  We introduce Spherical FNOs (SFNOs) for learning operators on spherical geometries. SFNOs have important implications for machine learning-based simulation of climate dynamics.
  arXiv  Detail & Related papers  (2023-06-06T16:27:17Z)
• Physics-constrained deep learning postprocessing of temperature and humidity [0.0]
  We propose to achieve physical consistency in deep learning-based postprocessing models. We find that constraining a neural network to enforce thermodynamic state equations yields physically-consistent predictions.
  arXiv  Detail & Related papers  (2022-12-07T09:31:25Z)
• Stabilizing Machine Learning Prediction of Dynamics: Noise and Noise-inspired Regularization [58.720142291102135]
  Recent has shown that machine learning (ML) models can be trained to accurately forecast the dynamics of chaotic dynamical systems. In the absence of mitigating techniques, this technique can result in artificially rapid error growth, leading to inaccurate predictions and/or climate instability. We introduce Linearized Multi-Noise Training (LMNT), a regularization technique that deterministically approximates the effect of many small, independent noise realizations added to the model input during training.
  arXiv  Detail & Related papers  (2022-11-09T23:40:52Z)
• Physics-informed machine learning with differentiable programming for heterogeneous underground reservoir pressure management [64.17887333976593]
  Avoiding over-pressurization in subsurface reservoirs is critical for applications like CO2 sequestration and wastewater injection. Managing the pressures by controlling injection/extraction are challenging because of complex heterogeneity in the subsurface. We use differentiable programming with a full-physics model and machine learning to determine the fluid extraction rates that prevent over-pressurization.
  arXiv  Detail & Related papers  (2022-06-21T20:38:13Z)
• Interpreting Machine Learning Models for Room Temperature Prediction in Non-domestic Buildings [0.0]
  This work presents an interpretable machine learning model aimed at predicting room temperature in non-domestic buildings. We demonstrate experimentally that the proposed model can accurately forecast room temperatures eight hours ahead in real-time.
  arXiv  Detail & Related papers  (2021-11-23T11:16:35Z)
• Physics-Integrated Variational Autoencoders for Robust and Interpretable Generative Modeling [86.9726984929758]
  We focus on the integration of incomplete physics models into deep generative models. We propose a VAE architecture in which a part of the latent space is grounded by physics. We demonstrate generative performance improvements over a set of synthetic and real-world datasets.
  arXiv  Detail & Related papers  (2021-02-25T20:28:52Z)
• On the application of Physically-Guided Neural Networks with Internal Variables to Continuum Problems [0.0]
  We present Physically-Guided Neural Networks with Internal Variables (PGNNIV) universal physical laws are used as constraints in the neural network, in such a way that some neuron values can be interpreted as internal state variables of the system. This endows the network with unraveling capacity, as well as better predictive properties such as faster convergence, fewer data needs and additional noise filtering. We extend this new methodology to continuum physical problems, showing again its predictive and explanatory capacities when only using measurable values in the training set.
  arXiv  Detail & Related papers  (2020-11-23T13:06:52Z)
• Identification of state functions by physically-guided neural networks with physically-meaningful internal layers [0.0]
  We use the concept of physically-constrained neural networks (PCNN) to predict the input-output relation in a physical system. We show that this approach, besides getting physically-based predictions, accelerates the training process.
  arXiv  Detail & Related papers  (2020-11-17T11:26:37Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.