Out-of-Support Generalisation via Weight Space Sequence Modelling

• URL: http://arxiv.org/abs/2602.13550v1
• Date: Sat, 14 Feb 2026 01:51:54 GMT
• Title: Out-of-Support Generalisation via Weight Space Sequence Modelling
• Authors: Roussel Desmond Nzoyem,
• Abstract summary: We reformulate the OoS generalisation problem as a sequence modelling task in the weight space.<n>Our framework yields plausible, interpretable, and uncertainty-aware predictions without necessitating explicit inductive biases.<n>These results hold significant implications for the wider adoption of artificial intelligence in safety-critical applications.
• Score: 1.6921396880325779
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: As breakthroughs in deep learning transform key industries, models are increasingly required to extrapolate on datapoints found outside the range of the training set, a challenge we coin as out-of-support (OoS) generalisation. However, neural networks frequently exhibit catastrophic failure on OoS samples, yielding unrealistic but overconfident predictions. We address this challenge by reformulating the OoS generalisation problem as a sequence modelling task in the weight space, wherein the training set is partitioned into concentric shells corresponding to discrete sequential steps. Our WeightCaster framework yields plausible, interpretable, and uncertainty-aware predictions without necessitating explicit inductive biases, all the while maintaining high computational efficiency. Emprical validation on a synthetic cosine dataset and real-world air quality sensor readings demonstrates performance competitive or superior to the state-of-the-art. By enhancing reliability beyond in-distribution scenarios, these results hold significant implications for the wider adoption of artificial intelligence in safety-critical applications.

Related papers

• Sycophancy Mitigation Through Reinforcement Learning with Uncertainty-Aware Adaptive Reasoning Trajectories [58.988535279557546]
  We introduce textbf sycophancy Mitigation through Adaptive Reasoning Trajectories.<n>We show that SMART significantly reduces sycophantic behavior while preserving strong performance on out-of-distribution inputs.
  arXiv  Detail & Related papers  (2025-09-20T17:09:14Z)
• Generalization vs. Memorization in Autoregressive Deep Learning: Or, Examining Temporal Decay of Gradient Coherence [0.1286280695561924]
  We apply influence function formalism to characterize how autoregressive PDE surrogates assimilate and propagate information derived from diverse physical scenarios.<n>We provide actionable insights regarding the design of improved surrogates.
  arXiv  Detail & Related papers  (2025-08-18T20:29:34Z)
• Robust Molecular Property Prediction via Densifying Scarce Labeled Data [53.24886143129006]
  In drug discovery, compounds most critical for advancing research often lie beyond the training set.<n>We propose a novel bilevel optimization approach that leverages unlabeled data to interpolate between in-distribution (ID) and out-of-distribution (OOD) data.
  arXiv  Detail & Related papers  (2025-06-13T15:27:40Z)
• Uncertainty-Aware Trajectory Prediction via Rule-Regularized Heteroscedastic Deep Classification [3.126303871979975]
  SHIFT (Spectral Heteroscedastic Informed Forecasting for Trajectories) is a novel framework that combines well-calibrated uncertainty modeling with informative priors.<n>Our model excels in complex scenarios, such as intersections, where uncertainty is inherently higher.
  arXiv  Detail & Related papers  (2025-04-17T17:24:50Z)
• A generative foundation model for an all-in-one seismic processing framework [2.499907423888049]
  generative seismic foundation model (GSFM) is a unified framework based on generative diffusion models (GDMs)<n>By adopting a target-oriented diffusion process prediction, GSFM improves computational efficiency without compromising accuracy.
  arXiv  Detail & Related papers  (2025-02-03T07:01:36Z)
• On the KL-Divergence-based Robust Satisficing Model [2.425685918104288]
  robustness satisficing framework has attracted increasing attention from academia. We present analytical interpretations, diverse performance guarantees, efficient and stable numerical methods, convergence analysis, and an extension tailored for hierarchical data structures. We demonstrate the superior performance of our model compared to state-of-the-art benchmarks.
  arXiv  Detail & Related papers  (2024-08-17T10:05:05Z)
• On the Generalization Ability of Unsupervised Pretraining [53.06175754026037]
  Recent advances in unsupervised learning have shown that unsupervised pre-training, followed by fine-tuning, can improve model generalization. This paper introduces a novel theoretical framework that illuminates the critical factor influencing the transferability of knowledge acquired during unsupervised pre-training to the subsequent fine-tuning phase. Our results contribute to a better understanding of unsupervised pre-training and fine-tuning paradigm, and can shed light on the design of more effective pre-training algorithms.
  arXiv  Detail & Related papers  (2024-03-11T16:23:42Z)
• A PAC-Bayesian Perspective on the Interpolating Information Criterion [54.548058449535155]
  We show how a PAC-Bayes bound is obtained for a general class of models, characterizing factors which influence performance in the interpolating regime. We quantify how the test error for overparameterized models achieving effectively zero training error depends on the quality of the implicit regularization imposed by e.g. the combination of model, parameter-initialization scheme.
  arXiv  Detail & Related papers  (2023-11-13T01:48:08Z)
• Hyperspectral Benchmark: Bridging the Gap between HSI Applications through Comprehensive Dataset and Pretraining [11.935879491267634]
  Hyperspectral Imaging (HSI) serves as a non-destructive spatial spectroscopy technique with a multitude of potential applications. A recurring challenge lies in the limited size of the target datasets, impeding exhaustive architecture search. This study introduces an innovative benchmark dataset encompassing three markedly distinct HSI applications.
  arXiv  Detail & Related papers  (2023-09-20T08:08:34Z)
• Toward Certified Robustness Against Real-World Distribution Shifts [65.66374339500025]
  We train a generative model to learn perturbations from data and define specifications with respect to the output of the learned model. A unique challenge arising from this setting is that existing verifiers cannot tightly approximate sigmoid activations. We propose a general meta-algorithm for handling sigmoid activations which leverages classical notions of counter-example-guided abstraction refinement.
  arXiv  Detail & Related papers  (2022-06-08T04:09:13Z)
• Generalization of Neural Combinatorial Solvers Through the Lens of Adversarial Robustness [68.97830259849086]
  Most datasets only capture a simpler subproblem and likely suffer from spurious features. We study adversarial robustness - a local generalization property - to reveal hard, model-specific instances and spurious features. Unlike in other applications, where perturbation models are designed around subjective notions of imperceptibility, our perturbation models are efficient and sound. Surprisingly, with such perturbations, a sufficiently expressive neural solver does not suffer from the limitations of the accuracy-robustness trade-off common in supervised learning.
  arXiv  Detail & Related papers  (2021-10-21T07:28:11Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.