Gemma 4 Technical Report

• URL: http://arxiv.org/abs/2607.02770v1
• Date: Thu, 02 Jul 2026 21:08:53 GMT
• Title: Gemma 4 Technical Report
• Abstract summary: We introduce Gemma 4, a new generation of open-weight, natively multimodal language models in the Gemma model family.<n>The Gemma 4 model suite features dense and Mixture-of-Experts architectures, ranging from 2.3B to 31B parameters.<n>We propose a unified, encoder-free architecture for our 12B model, which ingests raw audio and image patches.
• Score: 199.38174392182012
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We introduce Gemma 4, a new generation of open-weight, natively multimodal language models in the Gemma model family. Designed to advance compute efficiency and reasoning, the Gemma 4 model suite features dense and Mixture-of-Experts architectures, ranging from 2.3B to 31B parameters. Alongside improved vision and audio encoders for all model sizes, we propose a unified, encoder-free architecture for our 12B model, which ingests raw audio and image patches. Furthermore, we integrate a thinking mode, enabling Gemma models to generate reasoning traces prior to responding. We improve inference speed, memory, and compute efficiency, as well as long-context abilities through critical design choices. Gemma 4 establishes a leap in performance across STEM, multimodal, and long-context benchmarks, and rivals larger, frontier open models in human-rated tasks.

Related papers

• M*: A Modular, Extensible, Serving System for Multimodal Models [62.77975969000349]
  We present M*, a universal serving system for efficient serving of composite AI models.<n>We show how it can concisely capture composite models from a broad range of families.<n>M* also outperforms the V-JEPA 2-AC rollout baseline for robotic planning by up to 12.5x.
  arXiv  Detail & Related papers  (2026-06-10T21:22:22Z)
• Gemma 3 Technical Report [198.3299202423321]
  We introduce Gemma 3, a multimodal addition to the Gemma family of lightweight open models.<n>This version introduces vision understanding abilities, a wider coverage of languages and longer context.<n>We also change the architecture of the model to reduce the KV-cache memory that tends to explode with long context.
  arXiv  Detail & Related papers  (2025-03-25T15:52:34Z)
• Aria: An Open Multimodal Native Mixture-of-Experts Model [45.32344127542739]
  Aria is an open multimodal native model with best-in-class performance across a wide range of multimodal, language, and coding tasks.<n>It outperforms Pixtral-12B and Llama3.2-11B, and is competitive against the best proprietary models on various multimodal tasks.<n>We open-source the model weights along with a pipeline that facilitates easy adoptions and adaptations of Aria in real-world applications.
  arXiv  Detail & Related papers  (2024-10-08T12:44:57Z)
• xGen-MM (BLIP-3): A Family of Open Large Multimodal Models [145.5926980973847]
  BLIP-3 is an open framework for developing Large Multimodal Models.<n>We release 4B and 14B models, including both the pre-trained base model and the instruction fine-tuned ones.<n>Our models demonstrate competitive performance among open-source LMMs with similar model sizes.
  arXiv  Detail & Related papers  (2024-08-16T17:57:01Z)
• Gemma 2: Improving Open Language Models at a Practical Size [118.04200128754249]
  We introduce Gemma 2, a new addition to the Gemma family of lightweight, state-of-the-art open models. We apply several known technical modifications to the Transformer architecture, such as interleaving local-global attentions. The resulting models deliver the best performance for their size, and even offer competitive alternatives to models that are 2-3 times bigger.
  arXiv  Detail & Related papers  (2024-07-31T19:13:07Z)
• LLaVA-Gemma: Accelerating Multimodal Foundation Models with a Compact Language Model [4.6373877301731]
  We train a suite of multimodal foundation models (MMFM) using the popular LLaVA framework with the recently released Gemma family of large language models (LLMs) We test the effect of ablating three design features: pretraining the connector, utilizing a more powerful image backbone, and increasing the size of the language backbone. The resulting models, which we call LLaVA-Gemma, exhibit moderate performance on an array of evaluations, but fail to improve past the current comparably sized SOTA models.
  arXiv  Detail & Related papers  (2024-03-29T21:32:50Z)
• Gemma: Open Models Based on Gemini Research and Technology [128.57714343844074]
  This work introduces Gemma, a family of lightweight, state-of-the art open models built from the research and technology used to create Gemini models. Gemma models demonstrate strong performance across academic benchmarks for language understanding, reasoning, and safety.
  arXiv  Detail & Related papers  (2024-03-13T06:59:16Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.