Explorative Modeling: Unlocking a Third Pretraining Axis and End-to-End Generation

• URL: http://arxiv.org/abs/2607.27372v1
• Date: Wed, 29 Jul 2026 18:25:17 GMT
• Title: Explorative Modeling: Unlocking a Third Pretraining Axis and End-to-End Generation
• Abstract summary: We introduce Explorative Modeling, a new paradigm that factors the training loop, exploring K candidate matches between model generations and data, and training on the best.<n>First, increasing exploration adds a third pretraining axis beyond parameters and data for existing generative models.<n>Second, XMs enable end-to-end reconstructive generative modeling, matching diffusion on control tasks with 16-256x fewer inference steps.
• Score: 89.51408170909548
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The deep learning revolution, kicked off by AlexNet, taught us that end-to-end training beats decomposing a problem into hand-designed stages. Generative modeling, however, has remained the exception-despite generative models being remarkably capable, they are still not trained end-to-end. This is because, at its core, generative modeling is about handling distributions with many modes, and existing scalable approaches handle this the same way, by factoring the generation procedure, which prevents end-to-end generation. In this work, we introduce Explorative Modeling, a new paradigm that instead factors the training loop, exploring K candidate matches between model generations and data, and training on the best, so predictions commit to modes rather than blurring them. We find Explorative Models (XMs) useful in two settings. First, increasing exploration adds a third pretraining axis beyond parameters and data for existing generative models-where scaling exploration monotonically improves performance across both continuous and discrete domains (images, video, and language). Notably, gains from exploration increase with scale, climbing from 7% to 36% as data scales and from 13% to 23% as models grow, with efficiency gains more than doubling at 3x the compute. Concretely, exploration improves FLOP efficiency by 4.1x, sample efficiency by 6.2x, parameter efficiency by 47%, lifts the strongest of image-generation recipes to a near-state-of-the-art 1.43 FID on ImageNet without guidance, enables scaling how end-to-end existing models are, and unlocks scaling generalization. Second, XMs enable end-to-end reconstructive generative modeling, matching diffusion on control tasks with 16-256x fewer inference steps. Together, these results establish XMs as both a new pretraining axis for existing generative models and a standalone end-to-end generative modeling paradigm.

Related papers

• Solve the Loop: Attractor Models for Language and Reasoning [4.8720589853137435]
  Looped Transformers offer a promising alternative to purely feed-forward computation.<n>We introduce Attractor Models, in which a backbone module first proposes output embeddings, then an attractor module refines them by solving for the fixed point.<n>We show that Attractor Models outperform existing models across two regimes, large-scale language-model pretraining and reasoning with tiny models.
  arXiv  Detail & Related papers  (2026-05-12T17:51:26Z)
• Coupling Models for One-Step Discrete Generation [50.37847093567075]
  Coupling Models is a one-step discrete generative model that learns a direct coupling between discrete sequences and Gaussian latents.<n>It avoids complex continuous flows over the simplex and hand-specified data-to-noise couplings.<n>It reduces LM1B text-generation perplexity by 33% at its lowest-perplexity operating point.
  arXiv  Detail & Related papers  (2026-05-08T03:40:39Z)
• Disentangled Robot Learning via Separate Forward and Inverse Dynamics Pretraining [28.30092786035367]
  DeFI is a novel framework that Decouples visual Forward and Inverse dynamics pretraining to exploit respective data sources.<n>We introduce the General Forward Dynamics Model (GFDM), pretrained on diverse human and robot videos for future prediction, and the General Inverse Dynamics Model (GIDM), trained via self-supervised learning to infer latent actions from unlabeled video transitions.<n>Experiments on CALVIN ABC-D and SimplerEnv demonstrate state-of-the-art performance, with DeFI achieving an average task length of 4.51 for CALVIN, 51.2% success rate on SimplerEnv-Frac
  arXiv  Detail & Related papers  (2026-03-27T17:20:10Z)
• Seed2Scale: A Self-Evolving Data Engine for Embodied AI via Small to Large Model Synergy and Multimodal Evaluation [18.80050956469094]
  Seed2Scale is a self-evolving data engine that overcomes the data bottleneck.<n>It employs the lightweight Vision-Language-Action model, SuperTiny, as a dedicated collector.<n>A pre-trained Vision-Language Model is integrated as a Verifer to autonomously perform success/failure judgment.
  arXiv  Detail & Related papers  (2026-03-09T11:30:45Z)
• ArtiFixer: Enhancing and Extending 3D Reconstruction with Auto-Regressive Diffusion Models [27.324967736816337]
  Per-scene optimization methods such as 3D Gaussian Splatting provide state-of-the-art novel view synthesis quality but extrapolate poorly to under-observed areas.<n>We propose a two-stage pipeline that leverages two key insights.<n>First, we train a powerful bidirectional generative model with a novel opacity mixing strategy.<n>Second, we distill it into a causal auto-regressive model that generates hundreds of frames in a single pass.
  arXiv  Detail & Related papers  (2026-02-28T06:22:40Z)
• Adversarial Flow Models [26.917627135225118]
  We present adversarial flow models, a class of generative models that unifies adversarial models and flow models.<n>Our method supports native one-step or multi-step generation and is trained using the adversarial objective.<n>We show the possibility of end-to-end training of 56-layer and 112-layer models through depth repetition without any intermediate supervision.
  arXiv  Detail & Related papers  (2025-11-27T14:04:08Z)
• One-Step Diffusion Distillation through Score Implicit Matching [74.91234358410281]
  We present Score Implicit Matching (SIM) a new approach to distilling pre-trained diffusion models into single-step generator models. SIM shows strong empirical performances for one-step generators. By applying SIM to a leading transformer-based diffusion model, we distill a single-step generator for text-to-image generation.
  arXiv  Detail & Related papers  (2024-10-22T08:17:20Z)
• Self-Play Fine-Tuning of Diffusion Models for Text-to-Image Generation [59.184980778643464]
  Fine-tuning Diffusion Models remains an underexplored frontier in generative artificial intelligence (GenAI) In this paper, we introduce an innovative technique called self-play fine-tuning for diffusion models (SPIN-Diffusion) Our approach offers an alternative to conventional supervised fine-tuning and RL strategies, significantly improving both model performance and alignment.
  arXiv  Detail & Related papers  (2024-02-15T18:59:18Z)
• A-SDM: Accelerating Stable Diffusion through Redundancy Removal and Performance Optimization [54.113083217869516]
  In this work, we first explore the computational redundancy part of the network. We then prune the redundancy blocks of the model and maintain the network performance. Thirdly, we propose a global-regional interactive (GRI) attention to speed up the computationally intensive attention part.
  arXiv  Detail & Related papers  (2023-12-24T15:37:47Z)
• Consistency Models [89.68380014789861]
  We propose a new family of models that generate high quality samples by directly mapping noise to data. They support fast one-step generation by design, while still allowing multistep sampling to trade compute for sample quality. They also support zero-shot data editing, such as image inpainting, colorization, and super-resolution, without requiring explicit training.
  arXiv  Detail & Related papers  (2023-03-02T18:30:16Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.