Spatially-informed transformers: Injecting geostatistical covariance biases into self-attention for spatio-temporal forecasting

• URL: http://arxiv.org/abs/2512.17696v1
• Date: Fri, 19 Dec 2025 15:32:24 GMT
• Title: Spatially-informed transformers: Injecting geostatistical covariance biases into self-attention for spatio-temporal forecasting
• Abstract summary: We propose a hybrid architecture that injects a geostatistic inductive bias directly into the decomposing self-attention mechanism via a learnable costatistics kernel.<n>We demonstrate the phenomenon of Deep Variography'', where the network successfully recovers the true spatial parameters of the underlying process end-to-end via backpropagation.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The modeling of high-dimensional spatio-temporal processes presents a fundamental dichotomy between the probabilistic rigor of classical geostatistics and the flexible, high-capacity representations of deep learning. While Gaussian processes offer theoretical consistency and exact uncertainty quantification, their prohibitive computational scaling renders them impractical for massive sensor networks. Conversely, modern transformer architectures excel at sequence modeling but inherently lack a geometric inductive bias, treating spatial sensors as permutation-invariant tokens without a native understanding of distance. In this work, we propose a spatially-informed transformer, a hybrid architecture that injects a geostatistical inductive bias directly into the self-attention mechanism via a learnable covariance kernel. By formally decomposing the attention structure into a stationary physical prior and a non-stationary data-driven residual, we impose a soft topological constraint that favors spatially proximal interactions while retaining the capacity to model complex dynamics. We demonstrate the phenomenon of ``Deep Variography'', where the network successfully recovers the true spatial decay parameters of the underlying process end-to-end via backpropagation. Extensive experiments on synthetic Gaussian random fields and real-world traffic benchmarks confirm that our method outperforms state-of-the-art graph neural networks. Furthermore, rigorous statistical validation confirms that the proposed method delivers not only superior predictive accuracy but also well-calibrated probabilistic forecasts, effectively bridging the gap between physics-aware modeling and data-driven learning.

Related papers

• Out-of-distribution Neural Inference in Dynamical Ising Models [0.30586855806896035]
  We show that data-driven training produces distinct statistical strategies under topology and temperature shifts.<n>High in-distribution accuracy and apparent out-of-distribution do not necessarily imply a learned dynamics-to-structure rule.<n>Our results identify a concrete failure mode of standard data-driven learning in physical inverse problems.
  arXiv  Detail & Related papers  (2026-07-03T07:31:30Z)
• Representation Gap: Explaining the Unreasonable Effectiveness of Neural Networks from a Geometric Perspective [1.8618013369281987]
  We introduce the Representation Gap, a metric closely related to the generalization error, but admitting better-behaved dynamics.<n>We derive a precise equivalent of the Representation Gap and show that it is governed by a single parameter, the textitintrinsic dimension of the task.<n>We show that this dynamic also extends to a broader range of tasks and training algorithms.
  arXiv  Detail & Related papers  (2026-05-20T19:51:25Z)
• Adaptive Diffusion Posterior Sampling for Data and Model Fusion of Complex Nonlinear Dynamical Systems [1.9800280967139285]
  High-fidelity numerical simulations of chaotic, high dimensional nonlinear dynamical systems are computationally expensive.<n>Most surrogate models for such systems are deterministic, for example when neural operators are involved.<n>This work presents a surrogate modeling formulation that leverages generative machine learning.
  arXiv  Detail & Related papers  (2026-03-13T04:11:37Z)
• Emergent Manifold Separability during Reasoning in Large Language Models [46.78826734548872]
  Chain-of-Thought prompting significantly improves reasoning in Large Language Models.<n>We quantify the linear separability of latent representations without the confounding factors of probe training.
  arXiv  Detail & Related papers  (2026-02-23T20:36:17Z)
• Leveraging generative adversarial networks with spatially adaptive denormalization for multivariate stochastic seismic data inversion [0.0]
  We propose an iterative geostatistical inversion algorithm, SPADE-GANInv, for the prediction of facies and multiple correlated continuous properties from seismic data.<n>The SPADE-GAN is trained to reproduce realistic geometries, while sequential co-simulation predicts the spatial variability of the facies-dependent continuous properties.<n>The method is demonstrated on both 2-D synthetic scenarios and field data, targeting the prediction of facies, porosity, and acoustic impedance from full-stack seismic data.
  arXiv  Detail & Related papers  (2025-12-02T15:25:22Z)
• VIKING: Deep variational inference with stochastic projections [48.946143517489496]
  Variational mean field approximations tend to struggle with contemporary overparametrized deep neural networks.<n>We propose a simple variational family that considers two independent linear subspaces of the parameter space.<n>This allows us to build a fully-correlated approximate posterior reflecting the overparametrization.
  arXiv  Detail & Related papers  (2025-10-27T15:38:35Z)
• Deep operator network for surrogate modeling of poroelasticity with random permeability fields [3.7214007898390196]
  Poroelasticity -- coupled fluid flow and elastic deformation in porous media -- often involves spatially variable permeability.<n>In this study, we propose a surrogate modeling framework based on the deep operator network (DeepONet), a neural architecture designed to learn mappings between infinite-dimensional function spaces.<n>To enhance predictive accuracy and stability, we integrate three strategies: nondimensionalization of the governing equations, input dimensionality reduction via Karhunen--Lo'eve expansion, and a two-step training procedure that decouples the optimization of branch and trunk networks.
  arXiv  Detail & Related papers  (2025-09-15T14:18:49Z)
• Multidimensional Distributional Neural Network Output Demonstrated in Super-Resolution of Surface Wind Speed [0.0]
  We present a framework for training neural networks with a multidimensional Gaussian loss.<n>This framework generates closed-form predictive distributions over outputs with non-identically distributed and heteroscedastic structure.<n>We discuss its broader applicability to uncertainty-aware prediction in scientific models.
  arXiv  Detail & Related papers  (2025-08-21T18:22:44Z)
• Certified Neural Approximations of Nonlinear Dynamics [51.01318247729693]
  In safety-critical contexts, the use of neural approximations requires formal bounds on their closeness to the underlying system.<n>We propose a novel, adaptive, and parallelizable verification method based on certified first-order models.
  arXiv  Detail & Related papers  (2025-05-21T13:22:20Z)
• Efficient Transformed Gaussian Process State-Space Models for Non-Stationary High-Dimensional Dynamical Systems [49.819436680336786]
  We propose an efficient transformed Gaussian process state-space model (ETGPSSM) for scalable and flexible modeling of high-dimensional, non-stationary dynamical systems.<n>Specifically, our ETGPSSM integrates a single shared GP with input-dependent normalizing flows, yielding an expressive implicit process prior that captures complex, non-stationary transition dynamics.<n>Our ETGPSSM outperforms existing GPSSMs and neural network-based SSMs in terms of computational efficiency and accuracy.
  arXiv  Detail & Related papers  (2025-03-24T03:19:45Z)
• A Generalized Unified Skew-Normal Process with Neural Bayes Inference [1.5388334141379898]
  In recent decades, statisticians have been increasingly encountering spatial data that exhibit non-Gaussian behaviors such as asymmetry and heavy-tailedness.<n>To address the limitations of the Gaussian models, a variety of skewed models has been proposed.<n>Among various proposals in the literature, unified skewed distributions, such as the Unified Skew-Normal (SUN), have received considerable attention.
  arXiv  Detail & Related papers  (2024-11-26T13:00:39Z)
• Uncertainty quantification and out-of-distribution detection using surjective normalizing flows [46.51077762143714]
  We propose a simple approach using surjective normalizing flows to identify out-of-distribution data sets in deep neural network models. We show that our method can reliably discern out-of-distribution data from in-distribution data.
  arXiv  Detail & Related papers  (2023-11-01T09:08:35Z)
• Dynamic Kernel-Based Adaptive Spatial Aggregation for Learned Image Compression [63.56922682378755]
  We focus on extending spatial aggregation capability and propose a dynamic kernel-based transform coding. The proposed adaptive aggregation generates kernel offsets to capture valid information in the content-conditioned range to help transform. Experimental results demonstrate that our method achieves superior rate-distortion performance on three benchmarks compared to the state-of-the-art learning-based methods.
  arXiv  Detail & Related papers  (2023-08-17T01:34:51Z)
• Physics-informed UNets for Discovering Hidden Elasticity in Heterogeneous Materials [0.0]
  We develop a novel UNet-based neural network model for inversion in elasticity (El-UNet) We show superior performance, both in terms of accuracy and computational cost, by El-UNet compared to fully-connected physics-informed neural networks.
  arXiv  Detail & Related papers  (2023-06-01T23:35:03Z)
• Capturing dynamical correlations using implicit neural representations [85.66456606776552]
  We develop an artificial intelligence framework which combines a neural network trained to mimic simulated data from a model Hamiltonian with automatic differentiation to recover unknown parameters from experimental data. In doing so, we illustrate the ability to build and train a differentiable model only once, which then can be applied in real-time to multi-dimensional scattering data.
  arXiv  Detail & Related papers  (2023-04-08T07:55:36Z)
• Data-driven emergence of convolutional structure in neural networks [83.4920717252233]
  We show how fully-connected neural networks solving a discrimination task can learn a convolutional structure directly from their inputs. By carefully designing data models, we show that the emergence of this pattern is triggered by the non-Gaussian, higher-order local structure of the inputs.
  arXiv  Detail & Related papers  (2022-02-01T17:11:13Z)
• Multiplicative noise and heavy tails in stochastic optimization [62.993432503309485]
  empirical optimization is central to modern machine learning, but its role in its success is still unclear. We show that it commonly arises in parameters of discrete multiplicative noise due to variance. A detailed analysis is conducted in which we describe on key factors, including recent step size, and data, all exhibit similar results on state-of-the-art neural network models.
  arXiv  Detail & Related papers  (2020-06-11T09:58:01Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.