ARM: An AutoRegressive Large Multimodal Model with Unified Discrete Representations

• URL: http://arxiv.org/abs/2606.11188v1
• Date: Tue, 09 Jun 2026 17:59:28 GMT
• Title: ARM: An AutoRegressive Large Multimodal Model with Unified Discrete Representations
• Abstract summary: ARM is a representation-based AutoRegressive Model that unifies image understanding, generation, and editing.<n>Our tokenizer is supervised with multiple objectives that jointly promote semantic discriminability, language alignment, and faithful reconstruction.<n>Finally, ARM applies reinforcement learning to optimize task-level objectives such as visual quality, instruction adherence, and edit consistency.
• Score: 84.45092853572407
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: This paper introduces ARM, a discrete representation-based AutoRegressive Model that unifies image understanding, generation, and editing within a next-token prediction framework. ARM is built on three efforts: first, we train a discrete semantic visual tokenizer that maps images into compact token sequences. Our tokenizer is supervised with multiple objectives that jointly promote semantic discriminability, language alignment and faithful reconstruction, thereby supporting diverse tasks in a shared latent space. With this, we train a 7B autoregressive model over large-scale text and image token sequences, seamlessly developing vision-language perception and generation capabilities. Finally, to further improve preference-aligned behavior for text-to-image generation and instruction-guided editing, ARM applies reinforcement learning (RL) to optimize task-level objectives such as visual quality, instruction adherence, and edit consistency. Surprisingly, the results show that RL not only substantially improves performance on the target tasks (e.g., raising WISE overall from 0.50 to 0.56, GEdit-Bench-EN G_O from 5.75 to 6.68), but also induces cross-task synergy between text-to-image generation and editing. Collectively, these findings highlight autoregressive modeling, when paired with strong representations and preference optimization, as a scalable foundation for multimodal intelligence. Code: https://github.com/wdrink/ARM.

Related papers

• Reinforcing the Generation Order of Multimodal Masked Diffusion Models [6.5976113604335564]
  We introduce a learnable control module trained via Group Relative Policy Optimization (GRPO) to determine the generation order.<n>Our results demonstrate that learning this control block substantially improves both text-to-image alignment and multimodal understanding in Diffusion Language Models.
  arXiv  Detail & Related papers  (2026-07-09T02:07:30Z)
• Unified Multimodal Autoregressive Modeling with Shared Context-Visual Tokenizer is Key to Unification [80.62512020268626]
  UniAR is a unified autoregressive framework where a single visual tokenizer serves as the key bridge between understanding and generation.<n>UniAR adapts a pretrained vision encoder with multi-level feature fusion and a lookup-free bitwise quantization scheme.<n>A diffusion-based visual decoder operates on discrete visual tokens to decode high-fidelity images.
  arXiv  Detail & Related papers  (2026-06-16T17:59:22Z)
• EchoGen: Cycle-Consistent Learning for Unified Layout-Image Generation and Understanding [21.73510227758125]
  EchoGen is a unified framework for layout-to-image generation and image grounding.<n>Image grounding possesses strong text and layout understanding abilities, which can compensate for the limitations in layout-to-image generation.<n>Images generated from layouts exhibit high diversity in content, thereby enhancing the robustness of image grounding.
  arXiv  Detail & Related papers  (2026-03-18T17:59:03Z)
• UniWeTok: An Unified Binary Tokenizer with Codebook Size $\mathit{2^{128}}$ for Unified Multimodal Large Language Model [50.68870074090426]
  We introduce UniWeTok, a unified discrete tokenizer for Unified Multimodal Large Language Models.<n>For training framework, we introduce Pre-Post Distillation and a Generative-Aware Prior to enhance the semantic extraction and generative prior of the discrete tokens.<n>We propose a three-stage training framework designed to enhance UniWeTok's adaptability cross various image resolutions and perception-sensitive scenarios.
  arXiv  Detail & Related papers  (2026-02-15T15:07:19Z)
• IAR2: Improving Autoregressive Visual Generation with Semantic-Detail Associated Token Prediction [77.06211178777939]
  IAR2 is an advanced autoregressive framework that enables a hierarchical semantic-detail synthesis process.<n>We show that IAR2 sets a new state-of-the-art for autoregressive image generation, achieving a FID of 1.50 on ImageNet.
  arXiv  Detail & Related papers  (2025-10-08T12:08:21Z)
• Harmonizing Visual Representations for Unified Multimodal Understanding and Generation [53.01486796503091]
  We present emphHarmon, a unified autoregressive framework that harmonizes understanding and generation tasks with a shared MAR encoder.<n>Harmon achieves state-of-the-art image generation results on the GenEval, MJHQ30K and WISE benchmarks.
  arXiv  Detail & Related papers  (2025-03-27T20:50:38Z)
• Unified Autoregressive Visual Generation and Understanding with Continuous Tokens [52.21981295470491]
  We present UniFluid, a unified autoregressive framework for joint visual generation and understanding.<n>Our unified autoregressive architecture processes multimodal image and text inputs, generating discrete tokens for text and continuous tokens for image.<n>We find though there is an inherent trade-off between the image generation and understanding task, a carefully tuned training recipe enables them to improve each other.
  arXiv  Detail & Related papers  (2025-03-17T17:58:30Z)
• UniCMs: A Unified Consistency Model For Efficient Multimodal Generation and Understanding [12.34529497235534]
  consistency models (CMs) have shown promise in efficient generation of both image and text.<n>Key challenge is establishing a unified denoising perspective for both image and text generation.<n>In text-to-image generation, UniCMs outperform SD3 on GenEval, Image Reward, and CLIP Score metrics.<n>In image-to-text generation, UniCMs surpass Show-o on the MMMU benchmark while being $1.5 times$ faster at long-sequence generating speed.
  arXiv  Detail & Related papers  (2025-02-08T02:52:25Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.