Predictive Sampling with Forecasting Autoregressive Models

• URL: http://arxiv.org/abs/2002.09928v2
• Date: Wed, 8 Jul 2020 10:02:57 GMT
• Title: Predictive Sampling with Forecasting Autoregressive Models
• Authors: Auke Wiggers, Emiel Hoogeboom
• Abstract summary: Autoregressive models (ARMs) currently hold state-of-the-art performance in likelihood-based modeling of image and audio data. We introduce the predictive sampling algorithm: a procedure that exploits the fast inference property of ARMs in order to speed up sampling, while keeping the model intact.
• Score: 13.021014899410684
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Autoregressive models (ARMs) currently hold state-of-the-art performance in likelihood-based modeling of image and audio data. Generally, neural network based ARMs are designed to allow fast inference, but sampling from these models is impractically slow. In this paper, we introduce the predictive sampling algorithm: a procedure that exploits the fast inference property of ARMs in order to speed up sampling, while keeping the model intact. We propose two variations of predictive sampling, namely sampling with ARM fixed-point iteration and learned forecasting modules. Their effectiveness is demonstrated in two settings: i) explicit likelihood modeling on binary MNIST, SVHN and CIFAR10, and ii) discrete latent modeling in an autoencoder trained on SVHN, CIFAR10 and Imagenet32. Empirically, we show considerable improvements over baselines in number of ARM inference calls and sampling speed.

Related papers

• A-SDM: Accelerating Stable Diffusion through Model Assembly and Feature Inheritance Strategies [51.7643024367548]
  Stable Diffusion Model is a prevalent and effective model for text-to-image (T2I) and image-to-image (I2I) generation. This study focuses on reducing redundant computation in SDM and optimizing the model through both tuning and tuning-free methods.
  arXiv  Detail & Related papers  (2024-05-31T21:47:05Z)
• Fast Sampling of Diffusion Models via Operator Learning [74.37531458470086]
  We use neural operators, an efficient method to solve the probability flow differential equations, to accelerate the sampling process of diffusion models. Compared to other fast sampling methods that have a sequential nature, we are the first to propose a parallel decoding method. We show our method achieves state-of-the-art FID of 3.78 for CIFAR-10 and 7.83 for ImageNet-64 in the one-model-evaluation setting.
  arXiv  Detail & Related papers  (2022-11-24T07:30:27Z)
• IDM-Follower: A Model-Informed Deep Learning Method for Long-Sequence Car-Following Trajectory Prediction [24.94160059351764]
  Most car-following models are generative and only consider the inputs of the speed, position, and acceleration of the last time step. We implement a novel structure with two independent encoders and a self-attention decoder that could sequentially predict the following trajectories. Numerical experiments with multiple settings on simulation and NGSIM datasets show that the IDM-Follower can improve the prediction performance.
  arXiv  Detail & Related papers  (2022-10-20T02:24:27Z)
• Hierarchical Latent Structure for Multi-Modal Vehicle Trajectory Forecasting [0.0]
  We introduce a hierarchical latent structure into a VAE-based trajectory forecasting model. Our model is capable of generating clear multi-modal trajectory distributions and outperforms the state-of-the-art (SOTA) models in terms of prediction accuracy.
  arXiv  Detail & Related papers  (2022-07-11T04:52:28Z)
• SwinVRNN: A Data-Driven Ensemble Forecasting Model via Learned Distribution Perturbation [16.540748935603723]
  We propose a Swin Transformer-based Variational Recurrent Neural Network (SwinVRNN), which is a weather forecasting model combining a SwinRNN predictor with a perturbation module. SwinVRNN surpasses operational ECMWF Integrated Forecasting System (IFS) on surface variables of 2-m temperature and 6-hourly total precipitation at all lead times up to five days.
  arXiv  Detail & Related papers  (2022-05-26T05:11:58Z)
• Model-based micro-data reinforcement learning: what are the crucial model properties and which model to choose? [0.2836066255205732]
  We contribute to micro-data model-based reinforcement learning (MBRL) by rigorously comparing popular generative models. We find that on an environment that requires multimodal posterior predictives, mixture density nets outperform all other models by a large margin. We also found that deterministic models are on par, in fact they consistently (although non-significantly) outperform their probabilistic counterparts.
  arXiv  Detail & Related papers  (2021-07-24T11:38:25Z)
• ARMS: Antithetic-REINFORCE-Multi-Sample Gradient for Binary Variables [60.799183326613395]
  Antithetic REINFORCE-based Multi-Sample gradient estimator. ARMS uses a copula to generate any number of mutually antithetic samples. We evaluate ARMS on several datasets for training generative models, and our experimental results show that it outperforms competing methods.
  arXiv  Detail & Related papers  (2021-05-28T23:19:54Z)
• Approximated Bilinear Modules for Temporal Modeling [116.6506871576514]
  Two-layers in CNNs can be converted to temporal bilinear modules by adding an auxiliary-branch sampling. Our models can outperform most state-of-the-art methods on SomethingSomething v1 and v2 datasets without pretraining.
  arXiv  Detail & Related papers  (2020-07-25T09:07:35Z)
• An EM Approach to Non-autoregressive Conditional Sequence Generation [49.11858479436565]
  Autoregressive (AR) models have been the dominating approach to conditional sequence generation. Non-autoregressive (NAR) models have been recently proposed to reduce the latency by generating all output tokens in parallel. This paper proposes a new approach that jointly optimize both AR and NAR models in a unified Expectation-Maximization framework.
  arXiv  Detail & Related papers  (2020-06-29T20:58:57Z)
• Dynamic Model Pruning with Feedback [64.019079257231]
  We propose a novel model compression method that generates a sparse trained model without additional overhead. We evaluate our method on CIFAR-10 and ImageNet, and show that the obtained sparse models can reach the state-of-the-art performance of dense models.
  arXiv  Detail & Related papers  (2020-06-12T15:07:08Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.