Order-Agnostic Autoregressive Modelling with Missing Data

• URL: http://arxiv.org/abs/2605.06355v1
• Date: Thu, 07 May 2026 14:34:08 GMT
• Title: Order-Agnostic Autoregressive Modelling with Missing Data
• Abstract summary: We show that order-agnostic autoregressive models implicitly perform imputation under a missing completely at random mechanism.<n>We introduce the first principled framework for training them directly on incomplete datasets under general missingness mechanisms.<n>Across a suite of real-world benchmarks, our Missingness-Aware Order-Agnostic Autoregressive Model consistently outperforms established imputation baselines.
• Score: 12.971361761803891
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Order-Agnostic autoregressive models have demonstrated strong performance in deep generative modeling, yet their use in settings with incomplete data remains largely unexplored. In this work, we reinterpret them through the lens of missing data. First, we show that their standard training procedure on fully observed data implicitly performs imputation under a missing completely at random mechanism, resulting in robust out-of-sample imputation performance in settings with high missingness. Second, we introduce the first principled framework for training them directly on incomplete datasets under general missingness mechanisms. Third, we leverage their amortized conditional density estimation to perform active information acquisition, i.e., sequentially selecting the most informative missing variables for downstream prediction or inference. Across a suite of real-world benchmarks, our Missingness-Aware Order-Agnostic Autoregressive Model (MO-ARM) consistently outperforms established imputation baselines.

Related papers

• From Data to Behavior: Predicting Unintended Model Behaviors Before Training [78.37660873165284]
  We introduce Data2Behavior, a new task for predicting unintended model behaviors prior to training.<n>We also propose Manipulating Data Features (MDF), a lightweight approach that summarizes candidate data through their mean representations.<n>Experiments on Qwen3-14B, Qwen2.5-32B-Instruct, and Gemma-3-12b-it confirm that MDF can anticipate unintended behaviors and provide insight into pre-training vulnerabilities.
  arXiv  Detail & Related papers  (2026-02-04T16:37:17Z)
• Revisiting Multivariate Time Series Forecasting with Missing Values [65.30332997607141]
  Missing values are common in real-world time series.<n>Current approaches have developed an imputation-then-prediction framework that uses imputation modules to fill in missing values, followed by forecasting on the imputed data.<n>This framework overlooks a critical issue: there is no ground truth for the missing values, making the imputation process susceptible to errors that can degrade prediction accuracy.<n>We introduce Consistency-Regularized Information Bottleneck (CRIB), a novel framework built on the Information Bottleneck principle.
  arXiv  Detail & Related papers  (2025-09-27T20:57:48Z)
• Principled Input-Output-Conditioned Post-Hoc Uncertainty Estimation for Regression Networks [1.4671424999873808]
  Uncertainty is critical in safety-sensitive applications but is often omitted from off-the-shelf neural networks due to adverse effects on predictive performance.<n>We propose a theoretically grounded framework for post-hoc uncertainty estimation in regression tasks by fitting an auxiliary model to both original inputs and frozen model outputs.
  arXiv  Detail & Related papers  (2025-06-01T09:13:27Z)
• Continuous Visual Autoregressive Generation via Score Maximization [69.67438563485887]
  We introduce a Continuous VAR framework that enables direct visual autoregressive generation without vector quantization.<n>Within this framework, all we need is to select a strictly proper score and set it as the training objective to optimize.
  arXiv  Detail & Related papers  (2025-05-12T17:58:14Z)
• Self-attention-based Diffusion Model for Time-series Imputation in Partial Blackout Scenarios [23.160007389272575]
  Missing values in time series data can harm machine learning performance and introduce bias.<n>Previous work has tackled the imputation of missing data in random, complete blackouts and forecasting scenarios.<n>We introduce a two-stage imputation process using self-attention and diffusion processes to model feature and temporal correlations.
  arXiv  Detail & Related papers  (2025-03-03T16:58:15Z)
• What Do Learning Dynamics Reveal About Generalization in LLM Reasoning? [83.83230167222852]
  We find that a model's generalization behavior can be effectively characterized by a training metric we call pre-memorization train accuracy. By connecting a model's learning behavior to its generalization, pre-memorization train accuracy can guide targeted improvements to training strategies.
  arXiv  Detail & Related papers  (2024-11-12T09:52:40Z)
• Synthetic data, real errors: how (not) to publish and use synthetic data [86.65594304109567]
  We show how the generative process affects the downstream ML task. We introduce Deep Generative Ensemble (DGE) to approximate the posterior distribution over the generative process model parameters.
  arXiv  Detail & Related papers  (2023-05-16T07:30:29Z)
• Imputation-Free Learning from Incomplete Observations [73.15386629370111]
  We introduce the importance of guided gradient descent (IGSGD) method to train inference from inputs containing missing values without imputation. We employ reinforcement learning (RL) to adjust the gradients used to train the models via back-propagation. Our imputation-free predictions outperform the traditional two-step imputation-based predictions using state-of-the-art imputation methods.
  arXiv  Detail & Related papers  (2021-07-05T12:44:39Z)
• Training Deep Normalizing Flow Models in Highly Incomplete Data Scenarios with Prior Regularization [13.985534521589257]
  We propose a novel framework to facilitate the learning of data distributions in high paucity scenarios. The proposed framework naturally stems from posing the process of learning from incomplete data as a joint optimization task.
  arXiv  Detail & Related papers  (2021-04-03T20:57:57Z)
• MAIN: Multihead-Attention Imputation Networks [4.427447378048202]
  We propose a novel mechanism based on multi-head attention which can be applied effortlessly in any model. Our method inductively models patterns of missingness in the input data in order to increase the performance of the downstream task.
  arXiv  Detail & Related papers  (2021-02-10T13:50:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.