Visual Autoregressive Generation with Next Token Imagination
Abstract Overview
This paper argues that standard visual autoregressive image generation is overly constrained because training supervises each position only with the ground-truth next token, even though multiple plausible continuations may exist. The authors propose Next Token Imagination (NTI), a training paradigm that allows exploration of plausible non-ground-truth tokens while keeping this exploration regulated by the current context. NTI combines Context-Regulated Imagination (CRI), which assigns adaptive probability mass to non-ground-truth tokens, with Snapshot-based Context Approximation (SCA), which efficiently builds model-generated contexts without expensive autoregressive rollout. Experiments on class-conditional ImageNet generation show that this training change improves raster-order autoregressive generators without altering the tokenizer, generation order, or inference pipeline.
Novelty
The distinctive contribution is a shift from ground-truth-only supervision in visual autoregressive training to context-regulated supervision over plausible non-ground-truth next tokens. Methodologically, the paper introduces the paired CRI and SCA mechanisms to make this broader supervision both context-sensitive and computationally practical.
Results
On ImageNet-1K at 256×256, NTI improves an AR-L backbone from FID 3.02 to 1.67 and from IS 256.20 to 312.72, with consistent gains across model sizes. The method also improves non-standard tokenizer setups, reducing FID from 3.94 to 3.66 on GigaTok and from 1.61 to 1.51 on AliTok, while adding about 10% training overhead and leaving inference unchanged.
Key Points
- NTI reframes visual autoregressive training by allowing plausible non-ground-truth next-token supervision instead of forcing every off-ground-truth context back to the observed sequence.
- The method uses CRI to adapt non-ground-truth token weights to the current context and SCA to approximate generated contexts efficiently in a single-pass training setup.
- Empirically, NTI substantially strengthens standard raster-order autoregressive image generation on ImageNet and generalizes across tokenizer designs with modest extra training cost.