Aria: An Open Multimodal Native Mixture-of-Experts Model

• URL: http://arxiv.org/abs/2410.05993v4
• Date: Fri, 10 Jan 2025 08:35:13 GMT
• Title: Aria: An Open Multimodal Native Mixture-of-Experts Model
• Authors: Dongxu Li, Yudong Liu, Haoning Wu, Yue Wang, Zhiqi Shen, Bowen Qu, Xinyao Niu, Fan Zhou, Chengen Huang, Yanpeng Li, Chongyan Zhu, Xiaoyi Ren, Chao Li, Yifan Ye, Peng Liu, Lihuan Zhang, Hanshu Yan, Guoyin Wang, Bei Chen, Junnan Li,
• Abstract summary: Aria is an open multimodal native model with best-in-class performance across a wide range of multimodal, language, and coding tasks.<n>It outperforms Pixtral-12B and Llama3.2-11B, and is competitive against the best proprietary models on various multimodal tasks.<n>We open-source the model weights along with a pipeline that facilitates easy adoptions and adaptations of Aria in real-world applications.
• Score: 45.32344127542739
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Information comes in diverse modalities. Multimodal native AI models are essential to integrate real-world information and deliver comprehensive understanding. While proprietary multimodal native models exist, their lack of openness imposes obstacles for adoptions, let alone adaptations. To fill this gap, we introduce Aria, an open multimodal native model with best-in-class performance across a wide range of multimodal, language, and coding tasks. Aria is a mixture-of-expert model with 3.9B and 3.5B activated parameters per visual token and text token, respectively. It outperforms Pixtral-12B and Llama3.2-11B, and is competitive against the best proprietary models on various multimodal tasks. We pre-train Aria from scratch following a 4-stage pipeline, which progressively equips the model with strong capabilities in language understanding, multimodal understanding, long context window, and instruction following. We open-source the model weights along with a codebase that facilitates easy adoptions and adaptations of Aria in real-world applications.

Related papers

• Instella: Fully Open Language Models with Stellar Performance [24.52924932842208]
  Instella is a family of fully open three billion parameter language models.<n>Powered by Instinct MI300X, Instella is developed through large-scale pre-training, general-purpose instruction tuning, and alignment with human preferences.<n>We release two specialized variants: Instella-Long, capable of handling context lengths up to 128K tokens, and Instella-Math, a reasoning-focused model enhanced through supervised fine-tuning and reinforcement learning.
  arXiv  Detail & Related papers  (2025-11-13T18:52:46Z)
• Show-o2: Improved Native Unified Multimodal Models [57.34173415412808]
  Show-o2 is a native unified multimodal models that leverage autoregressive modeling and flow matching.<n>Built upon a 3D causal variational autoencoder space, unified visual representations are constructed through a dual-path of spatial (-temporal) fusion.
  arXiv  Detail & Related papers  (2025-06-18T15:39:15Z)
• Emerging Properties in Unified Multimodal Pretraining [32.856334401494145]
  We introduce BAGEL, an open-source foundational model that supports multimodal understanding and generation.<n>BAGEL is a unified, decoder-only model pretrained on trillions of tokens curated from large-scale interleaved text, image, video, and web data.<n>It significantly outperforms open-source unified models in both multimodal generation and understanding across standard benchmarks.
  arXiv  Detail & Related papers  (2025-05-20T17:59:30Z)
• Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling [128.24325909395188]
  We introduce InternVL 2.5, an advanced multimodal large language model (MLLM) series that builds upon InternVL 2.0. InternVL 2.5 exhibits competitive performance, rivaling leading commercial models such as GPT-4o and Claude-3.5-Sonnet. We hope this model contributes to the open-source community by setting new standards for developing and applying multimodal AI systems.
  arXiv  Detail & Related papers  (2024-12-06T18:57:08Z)
• Towards Multi-Modal Mastery: A 4.5B Parameter Truly Multi-Modal Small Language Model [0.0]
  We present a novel 4.5B parameter small language model that can handle multiple input and output modalities. Despite its small size, the model achieves near state-of-the-art performance on a variety of tasks.
  arXiv  Detail & Related papers  (2024-11-08T17:15:17Z)
• ANOLE: An Open, Autoregressive, Native Large Multimodal Models for Interleaved Image-Text Generation [27.773146599559286]
  Anole is an open, autoregressive, native large multimodal model for interleaved image-text generation. We have open-sourced our model, training framework, and instruction tuning data.
  arXiv  Detail & Related papers  (2024-07-08T17:08:02Z)
• U3M: Unbiased Multiscale Modal Fusion Model for Multimodal Semantic Segmentation [63.31007867379312]
  We introduce U3M: An Unbiased Multiscale Modal Fusion Model for Multimodal Semantics. We employ feature fusion at multiple scales to ensure the effective extraction and integration of both global and local features. Experimental results demonstrate that our approach achieves superior performance across multiple datasets.
  arXiv  Detail & Related papers  (2024-05-24T08:58:48Z)
• SEED-X: Multimodal Models with Unified Multi-granularity Comprehension and Generation [61.392147185793476]
  We present a unified and versatile foundation model, namely, SEED-X. SEED-X is able to model multi-granularity visual semantics for comprehension and generation tasks. We hope that our work will inspire future research into what can be achieved by versatile multimodal foundation models in real-world applications.
  arXiv  Detail & Related papers  (2024-04-22T17:56:09Z)
• Octopus v3: Technical Report for On-device Sub-billion Multimodal AI Agent [10.998608318944985]
  A multimodal AI agent is characterized by its ability to process and learn from various types of data. We introduce a multimodal model that incorporates the concept of functional token specifically designed for AI agent applications. We demonstrate that this model is capable of operating efficiently on a wide range of edge devices, including as constrained as a Raspberry Pi.
  arXiv  Detail & Related papers  (2024-04-17T15:07:06Z)
• VL-Mamba: Exploring State Space Models for Multimodal Learning [22.701028299912398]
  In this work, we propose VL-Mamba, a multimodal large language model based on state space models. Specifically, we first replace the transformer-based backbone language model such as LLama or Vicuna with the pre-trained Mamba language model.
  arXiv  Detail & Related papers  (2024-03-20T13:48:50Z)
• Noise-powered Multi-modal Knowledge Graph Representation Framework [52.95468915728721]
  The rise of Multi-modal Pre-training highlights the necessity for a unified Multi-Modal Knowledge Graph representation learning framework. We propose a novel SNAG method that utilizes a Transformer-based architecture equipped with modality-level noise masking. Our approach achieves SOTA performance across a total of ten datasets, demonstrating its versatility.
  arXiv  Detail & Related papers  (2024-03-11T15:48:43Z)
• AnyGPT: Unified Multimodal LLM with Discrete Sequence Modeling [115.89786751297348]
  We introduce AnyGPT, an any-to-any multimodal language model that utilizes discrete representations for the unified processing of various modalities. We build a multimodal text-centric dataset for multimodal alignment pre-training. We show that AnyGPT is capable of facilitating any-to-any multimodal conversation while achieving performance comparable to specialized models across all modalities.
  arXiv  Detail & Related papers  (2024-02-19T15:33:10Z)
• Generative Multimodal Models are In-Context Learners [60.50927925426832]
  We introduce Emu2, a generative multimodal model with 37 billion parameters, trained on large-scale multimodal sequences. Emu2 exhibits strong multimodal in-context learning abilities, even emerging to solve tasks that require on-the-fly reasoning.
  arXiv  Detail & Related papers  (2023-12-20T18:59:58Z)
• eP-ALM: Efficient Perceptual Augmentation of Language Models [70.47962271121389]
  We propose to direct effort to efficient adaptations of existing models, and propose to augment Language Models with perception. Existing approaches for adapting pretrained models for vision-language tasks still rely on several key components that hinder their efficiency. We show that by freezing more than 99% of total parameters, training only one linear projection layer, and prepending only one trainable token, our approach (dubbed eP-ALM) significantly outperforms other baselines on VQA and Captioning.
  arXiv  Detail & Related papers  (2023-03-20T19:20:34Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.