Architectural and Inferential Inductive Biases For Exchangeable Sequence Modeling

• URL: http://arxiv.org/abs/2503.01215v1
• Date: Mon, 03 Mar 2025 06:25:44 GMT
• Title: Architectural and Inferential Inductive Biases For Exchangeable Sequence Modeling
• Authors: Daksh Mittal, Ang Li, Tzu-Ching Yen, Daniel Guetta, Hongseok Namkoong,
• Abstract summary: Autoregressive models have emerged as a powerful framework for modeling exchangeable sequences.<n>We study the inferential and architectural inductive biases that are most effective for exchangeable sequence modeling.
• Score: 11.018120203982546
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Autoregressive models have emerged as a powerful framework for modeling exchangeable sequences - i.i.d. observations when conditioned on some latent factor - enabling direct modeling of uncertainty from missing data (rather than a latent). Motivated by the critical role posterior inference plays as a subroutine in decision-making (e.g., active learning, bandits), we study the inferential and architectural inductive biases that are most effective for exchangeable sequence modeling. For the inference stage, we highlight a fundamental limitation of the prevalent single-step generation approach: inability to distinguish between epistemic and aleatoric uncertainty. Instead, a long line of works in Bayesian statistics advocates for multi-step autoregressive generation; we demonstrate this "correct approach" enables superior uncertainty quantification that translates into better performance on downstream decision-making tasks. This naturally leads to the next question: which architectures are best suited for multi-step inference? We identify a subtle yet important gap between recently proposed Transformer architectures for exchangeable sequences (Muller et al., 2022; Nguyen & Grover, 2022; Ye & Namkoong, 2024), and prove that they in fact cannot guarantee exchangeability despite introducing significant computational overhead. We illustrate our findings using controlled synthetic settings, demonstrating how custom architectures can significantly underperform standard causal masks, underscoring the need for new architectural innovations.

Related papers

• Variational Bayesian Bow tie Neural Networks with Shrinkage [0.276240219662896]
  We build a relaxed version of the standard feed-forward rectified neural network. We employ Polya-Gamma data augmentation tricks to render a conditionally linear and Gaussian model. We derive a variational inference algorithm that avoids distributional assumptions and independence across layers.
  arXiv  Detail & Related papers  (2024-11-17T17:36:30Z)
• Continuous Bayesian Model Selection for Multivariate Causal Discovery [22.945274948173182]
  Current causal discovery approaches require restrictive model assumptions or assume access to interventional data to ensure structure identifiability. Recent work has shown that Bayesian model selection can greatly improve accuracy by exchanging restrictive modelling for more flexible assumptions. We demonstrate the competitiveness of our approach on both synthetic and real-world datasets.
  arXiv  Detail & Related papers  (2024-11-15T12:55:05Z)
• Exchangeable Sequence Models Quantify Uncertainty Over Latent Concepts [6.256239986541708]
  We show that pre-trained sequence models are naturally capable of probabilistic reasoning over exchangeable data points.<n>A sequence model learns the relationship between observations, which differs from typical Bayesian models.<n>We show the sequence prediction loss controls the quality of uncertainty quantification.
  arXiv  Detail & Related papers  (2024-08-06T17:16:10Z)
• Rigorous Probabilistic Guarantees for Robust Counterfactual Explanations [80.86128012438834]
  We show for the first time that computing the robustness of counterfactuals with respect to plausible model shifts is NP-complete. We propose a novel probabilistic approach which is able to provide tight estimates of robustness with strong guarantees.
  arXiv  Detail & Related papers  (2024-07-10T09:13:11Z)
• When Does Confidence-Based Cascade Deferral Suffice? [69.28314307469381]
  Cascades are a classical strategy to enable inference cost to vary adaptively across samples. A deferral rule determines whether to invoke the next classifier in the sequence, or to terminate prediction. Despite being oblivious to the structure of the cascade, confidence-based deferral often works remarkably well in practice.
  arXiv  Detail & Related papers  (2023-07-06T04:13:57Z)
• Advancing Counterfactual Inference through Nonlinear Quantile Regression [77.28323341329461]
  We propose a framework for efficient and effective counterfactual inference implemented with neural networks. The proposed approach enhances the capacity to generalize estimated counterfactual outcomes to unseen data. Empirical results conducted on multiple datasets offer compelling support for our theoretical assertions.
  arXiv  Detail & Related papers  (2023-06-09T08:30:51Z)
• Effects of Architectures on Continual Semantic Segmentation [0.0]
  We study how the choice of neural network architecture affects catastrophic forgetting in class- and domain-incremental semantic segmentation. We find that traditional CNNs like ResNet have high plasticity but low stability, while transformer architectures are much more stable.
  arXiv  Detail & Related papers  (2023-02-21T15:12:01Z)
• Slimmable Domain Adaptation [112.19652651687402]
  We introduce a simple framework, Slimmable Domain Adaptation, to improve cross-domain generalization with a weight-sharing model bank. Our framework surpasses other competing approaches by a very large margin on multiple benchmarks.
  arXiv  Detail & Related papers  (2022-06-14T06:28:04Z)
• Training and Inference on Any-Order Autoregressive Models the Right Way [97.39464776373902]
  A family of Any-Order Autoregressive Models (AO-ARMs) has shown breakthrough performance in arbitrary conditional tasks. We identify significant improvements to be made to previous formulations of AO-ARMs. Our method leads to improved performance with no compromises on tractability.
  arXiv  Detail & Related papers  (2022-05-26T18:00:02Z)
• Variational Causal Networks: Approximate Bayesian Inference over Causal Structures [132.74509389517203]
  We introduce a parametric variational family modelled by an autoregressive distribution over the space of discrete DAGs. In experiments, we demonstrate that the proposed variational posterior is able to provide a good approximation of the true posterior.
  arXiv  Detail & Related papers  (2021-06-14T17:52:49Z)
• Self-Reflective Variational Autoencoder [21.054722609128525]
  Variational Autoencoder (VAE) is a powerful framework for learning latent variable generative models. We introduce a solution, which we call self-reflective inference. We empirically demonstrate the clear advantages of matching the variational posterior to the exact posterior.
  arXiv  Detail & Related papers  (2020-07-10T05:05:26Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.