Decoupled-Value Attention for Prior-Data Fitted Networks: GP Inference for Physical Equations

• URL: http://arxiv.org/abs/2509.20950v1
• Date: Thu, 25 Sep 2025 09:32:42 GMT
• Title: Decoupled-Value Attention for Prior-Data Fitted Networks: GP Inference for Physical Equations
• Abstract summary: Prior-data fitted networks (PFNs) are a promising alternative to time-consuming Gaussian Process (GP) inference for creating fast surrogates of physical systems.<n>We introduce Decoupled-Value Attention (DVA), motivated by the GP property that the function space is fully characterized by the kernel over inputs.<n>We demonstrate that the crucial factor for scaling PFNs is the attention rule rather than the architecture itself.
• Score: 0.3823356975862005
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Prior-data fitted networks (PFNs) are a promising alternative to time-consuming Gaussian Process (GP) inference for creating fast surrogates of physical systems. PFN reduces the computational burden of GP-training by replacing Bayesian inference in GP with a single forward pass of a learned prediction model. However, with standard Transformer attention, PFNs show limited effectiveness on high-dimensional regression tasks. We introduce Decoupled-Value Attention (DVA)-- motivated by the GP property that the function space is fully characterized by the kernel over inputs and the predictive mean is a weighted sum of training targets. DVA computes similarities from inputs only and propagates labels solely through values. Thus, the proposed DVA mirrors the Gaussian-process update while remaining kernel-free. We demonstrate that the crucial factor for scaling PFNs is the attention rule rather than the architecture itself. Specifically, our results demonstrate that (a) localized attention consistently reduces out-of-sample validation loss in PFNs across different dimensional settings, with validation loss reduced by more than 50% in five- and ten-dimensional cases, and (b) the role of attention is more decisive than the choice of backbone architecture, showing that CNN-based PFNs can perform at par with their Transformer-based counterparts. The proposed PFNs provide 64-dimensional power flow equation approximations with a mean absolute error of the order of 1E-3, while being over 80x faster than exact GP inference.

Related papers

• Physics-Informed Laplace Neural Operator for Solving Partial Differential Equations [11.064132774859553]
  Physics-Informed Laplace Neural Operator (PILNO) is a fast surrogate solver for partial differential equations.<n>It embeds physics into training through PDE, boundary condition, and initial condition residuals.<n>PILNO consistently improves accuracy in small-data settings, reduces run-to-run variability across random seeds, and achieves stronger generalization than purely data-driven baselines.
  arXiv  Detail & Related papers  (2026-02-13T08:19:40Z)
• Scalable Gaussian Processes with Low-Rank Deep Kernel Decomposition [7.532273334759435]
  Kernels are key to encoding prior beliefs and data structures in Gaussian process (GP) models.<n>Deep kernel learning enhances kernel flexibility by feeding inputs through a neural network before applying a standard parametric form.<n>We introduce a fully data-driven, scalable deep kernel representation where a neural network directly represents a low-rank kernel.
  arXiv  Detail & Related papers  (2025-05-24T05:42:11Z)
• Neighbour-Driven Gaussian Process Variational Autoencoders for Scalable Structured Latent Modelling [14.358070928996069]
  Gaussian Process (GP) Variational Autoencoders (VAEs) extend standard VAEs by replacing the fully factorised Gaussian prior with a GP prior.<n> performing exact GP inference in large-scale GPVAEs is computationally prohibitive, often forcing existing approaches to rely on restrictive kernel assumptions.<n>We propose a neighbour-driven approximation strategy that exploits local adjacencies in the latent space to achieve scalable GPVAE inference.
  arXiv  Detail & Related papers  (2025-05-22T10:07:33Z)
• DeepRV: Accelerating spatiotemporal inference with pre-trained neural priors [0.0]
  We introduce DeepRV, a neural-network surrogate that closely matches full GP accuracy including hyperRV estimates.<n>Across simulated benchmarks, DeepRV achieves non-separable programming, and a real-world application to education deprivation in London.<n>All experiments run on a single consumer-grade GPU to ensure accessibility for practitioners.
  arXiv  Detail & Related papers  (2025-03-27T13:04:41Z)
• Fixed-Mean Gaussian Processes for Post-hoc Bayesian Deep Learning [11.22428369342346]
  We introduce a novel family of sparse variational Gaussian processes (GPs), where the posterior mean is fixed to any continuous function when using a universal kernel.<n>Specifically, we fix the mean of this GP to the output of the pre-trained DNN, allowing our approach to effectively fit the GP's predictive variances to estimate the prediction uncertainty.<n> Experimental results demonstrate that FMGP improves both uncertainty estimation and computational efficiency when compared to state-of-the-art methods.
  arXiv  Detail & Related papers  (2024-12-05T14:17:16Z)
• Chasing Fairness in Graphs: A GNN Architecture Perspective [73.43111851492593]
  We propose textsfFair textsfMessage textsfPassing (FMP) designed within a unified optimization framework for graph neural networks (GNNs) In FMP, the aggregation is first adopted to utilize neighbors' information and then the bias mitigation step explicitly pushes demographic group node presentation centers together. Experiments on node classification tasks demonstrate that the proposed FMP outperforms several baselines in terms of fairness and accuracy on three real-world datasets.
  arXiv  Detail & Related papers  (2023-12-19T18:00:15Z)
• Sparse is Enough in Fine-tuning Pre-trained Large Language Models [98.46493578509039]
  We propose a gradient-based sparse fine-tuning algorithm, named Sparse Increment Fine-Tuning (SIFT) We validate its effectiveness on a range of tasks including the GLUE Benchmark and Instruction-tuning.
  arXiv  Detail & Related papers  (2023-12-19T06:06:30Z)
• Sample-Efficient Optimisation with Probabilistic Transformer Surrogates [66.98962321504085]
  This paper investigates the feasibility of employing state-of-the-art probabilistic transformers in Bayesian optimisation. We observe two drawbacks stemming from their training procedure and loss definition, hindering their direct deployment as proxies in black-box optimisation. We introduce two components: 1) a BO-tailored training prior supporting non-uniformly distributed points, and 2) a novel approximate posterior regulariser trading-off accuracy and input sensitivity to filter favourable stationary points for improved predictive performance.
  arXiv  Detail & Related papers  (2022-05-27T11:13:17Z)
• Transformers Can Do Bayesian Inference [56.99390658880008]
  We present Prior-Data Fitted Networks (PFNs) PFNs leverage in-context learning in large-scale machine learning techniques to approximate a large set of posteriors. We demonstrate that PFNs can near-perfectly mimic Gaussian processes and also enable efficient Bayesian inference for intractable problems.
  arXiv  Detail & Related papers  (2021-12-20T13:07:39Z)
• Robust and Adaptive Temporal-Difference Learning Using An Ensemble of Gaussian Processes [70.80716221080118]
  The paper takes a generative perspective on policy evaluation via temporal-difference (TD) learning. The OS-GPTD approach is developed to estimate the value function for a given policy by observing a sequence of state-reward pairs. To alleviate the limited expressiveness associated with a single fixed kernel, a weighted ensemble (E) of GP priors is employed to yield an alternative scheme.
  arXiv  Detail & Related papers  (2021-12-01T23:15:09Z)
• Learning to Solve the AC-OPF using Sensitivity-Informed Deep Neural Networks [52.32646357164739]
  We propose a deep neural network (DNN) to solve the solutions of the optimal power flow (ACOPF) The proposed SIDNN is compatible with a broad range of OPF schemes. It can be seamlessly integrated in other learning-to-OPF schemes.
  arXiv  Detail & Related papers  (2021-03-27T00:45:23Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.