MEPA: Multi-Scale Representation Alignment for Visual Autoregressive Modeling with Mixture of Experts

• URL: http://arxiv.org/abs/2607.00371v1
• Date: Wed, 01 Jul 2026 03:11:21 GMT
• Title: MEPA: Multi-Scale Representation Alignment for Visual Autoregressive Modeling with Mixture of Experts
• Abstract summary: We introduce a scale-aware token-routed Mixture of Experts (MoE) architecture, allowing scale-adaptive expert selection.<n>Unlike naive alignment, we analyse and design a residual feature aggregation scheme tailored to the Visual AutoRegressive modeling paradigm.<n>Our model achieves a superior FID compared to the dense baseline on the ImageNet 256*256 benchmark.
• Score: 41.67455261906954
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Visual AutoRegressive modeling (VAR) has pioneered a coarse-to-fine multi-scale autoregressive generative paradigm, demonstrating strong capabilities in image generation. However, VAR still suffers from inherent deficiencies in multi-scale representation learning. Specifically, lower scales primarily capture global semantics, while higher scales focus on fine-grained details. Employing a shared architecture across scales induces optimization conflicts. Moreover, due to the causal autoregressive process, inaccurate semantics at early scales can propagate and significantly degrade the final output. To address these issues, we introduce a scale-aware token-routed Mixture of Experts (MoE) architecture, allowing scale-adaptive expert selection, thereby facilitating decoupled representation learning across scales. In addition, we enhance semantic modeling at early scales by incorporating external self-supervised features. Unlike naive alignment, we analyse and design a residual feature aggregation scheme tailored to the VAR paradigm. Extensive experiments show that our method significantly improves both training efficiency and generation quality. On the ImageNet 256*256 benchmark, our model achieves a superior FID compared to the dense baseline while requiring only half of the default training epochs and a smaller parameter budget, with a merely marginal increase in training cost. Moreover, the performance gap further widens with larger training epochs.

Related papers

• Divide-and-Conquer Inference for Large-Scale Visual Recognition with Multimodal Large Language Models [12.459927405623624]
  Divide-and-Conquer Inference (DCI) is a novel test-time scaling strategy for visual recognition with MLLMs.<n>DCI decomposes complex global classification tasks into simpler, localized subproblems and employs a dynamic pruning mechanism to compress the search space.<n>As a model-agnostic, plug-and-play paradigm, DCI offers an efficient approach for scaling the inferential precision of MLLMs in large-scale scenarios.
  arXiv  Detail & Related papers  (2026-05-24T01:07:05Z)
• Hierarchical Image Tokenization for Multi-Scale Image Super Resolution [47.66856101347057]
  We introduce a multi-scale Image Super Resolution (ISR) method building on recent advances in Visual Auto-Regressive ( VAR) modeling.<n>We introduce two novel components to the VAR training for ISR, aiming at increasing its flexibility and reducing its complexity.<n>Our proposed HIT acts as a strong inductive bias for the VAR training, resulting in a small model that achieves state-of-the-art results without external training data.
  arXiv  Detail & Related papers  (2026-05-14T14:35:51Z)
• Neural Network Optimization Reimagined: Decoupled Techniques for Scratch and Fine-Tuning [49.751529745537546]
  We propose DualOpt, a novel approach that decouples optimization techniques specifically tailored for training from scratch.<n>For training from scratch, we introduce real-time layer-wise weight decay, designed to enhance both convergence and generalization.<n>We extend the layer-wise weight decay to dynamically adjust the rollback levels across layers, adapting to the varying demands of different downstream tasks.
  arXiv  Detail & Related papers  (2026-04-21T06:27:18Z)
• RAE-AR: Taming Autoregressive Models with Representation Autoencoders [61.73674018219353]
  We introduce token simplification via distribution normalization to ease modeling difficulty and improve convergence.<n>We enhance prediction by incorporating Gaussian noise injection during training to mitigate exposure bias.<n>This work paves the way for a more unified architecture across visual understanding and generative modeling.
  arXiv  Detail & Related papers  (2026-04-02T02:39:28Z)
• DreamVAR: Taming Reinforced Visual Autoregressive Model for High-Fidelity Subject-Driven Image Generation [108.71044040025374]
  We present a novel framework for subject-driven image synthesis built upon a Visual Autoregressive model that employs next-scale prediction.<n>We show that Dreamthe achieves superior appearance preservation compared to leading diffusion-based methods.
  arXiv  Detail & Related papers  (2026-01-30T03:32:29Z)
• Scaling Text-to-Image Diffusion Transformers with Representation Autoencoders [74.72147962028265]
  Representation Autoencoders (RAEs) have shown distinct advantages in diffusion modeling on ImageNet.<n>We investigate whether this framework can scale to large-scale, freeform text-to-image (T2I) generation.
  arXiv  Detail & Related papers  (2026-01-22T18:58:16Z)
• Improving Progressive Generation with Decomposable Flow Matching [50.63174319509629]
  Decomposable Flow Matching (DFM) is a simple and effective framework for the progressive generation of visual media.<n>On Imagenet-1k 512px, DFM achieves 35.2% improvements in FDD scores over the base architecture and 26.4% over the best-performing baseline.
  arXiv  Detail & Related papers  (2025-06-24T17:58:02Z)
• Visual Autoregressive Modeling for Image Super-Resolution [14.935662351654601]
  We propose a novel visual autoregressive modeling for ISR framework with the form of next-scale prediction.<n>We collect large-scale data and design a training process to obtain robust generative priors.
  arXiv  Detail & Related papers  (2025-01-31T09:53:47Z)
• Denoising Autoregressive Representation Learning [13.185567468951628]
  Our method, DARL, employs a decoder-only Transformer to predict image patches autoregressively. We show that the learned representation can be improved by using tailored noise schedules and longer training in larger models.
  arXiv  Detail & Related papers  (2024-03-08T10:19:00Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.