Gemma 3 Technical Report

• URL: http://arxiv.org/abs/2503.19786v1
• Date: Tue, 25 Mar 2025 15:52:34 GMT
• Title: Gemma 3 Technical Report
• Abstract summary: We introduce Gemma 3, a multimodal addition to the Gemma family of lightweight open models.<n>This version introduces vision understanding abilities, a wider coverage of languages and longer context.<n>We also change the architecture of the model to reduce the KV-cache memory that tends to explode with long context.
• Score: 198.3299202423321
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We introduce Gemma 3, a multimodal addition to the Gemma family of lightweight open models, ranging in scale from 1 to 27 billion parameters. This version introduces vision understanding abilities, a wider coverage of languages and longer context - at least 128K tokens. We also change the architecture of the model to reduce the KV-cache memory that tends to explode with long context. This is achieved by increasing the ratio of local to global attention layers, and keeping the span on local attention short. The Gemma 3 models are trained with distillation and achieve superior performance to Gemma 2 for both pre-trained and instruction finetuned versions. In particular, our novel post-training recipe significantly improves the math, chat, instruction-following and multilingual abilities, making Gemma3-4B-IT competitive with Gemma2-27B-IT and Gemma3-27B-IT comparable to Gemini-1.5-Pro across benchmarks. We release all our models to the community.

Related papers

• Gemma 4 Technical Report [199.38174392182012]
  We introduce Gemma 4, a new generation of open-weight, natively multimodal language models in the Gemma model family.<n>The Gemma 4 model suite features dense and Mixture-of-Experts architectures, ranging from 2.3B to 31B parameters.<n>We propose a unified, encoder-free architecture for our 12B model, which ingests raw audio and image patches.
  arXiv  Detail & Related papers  (2026-07-02T21:08:53Z)
• VaultGemma: A Differentially Private Gemma Model [80.35976343610584]
  We introduce VaultGemma 1B, a 1 billion parameter model within the Gemma family, fully trained with differential privacy.<n>We openly release this model to the community.
  arXiv  Detail & Related papers  (2025-10-15T21:59:53Z)
• 2 OLMo 2 Furious [154.15728448754854]
  We present OLMo 2, the next generation of our fully open language models.<n> OLMo 2 includes a family of dense autoregressive language models at 7B, 13B and 32B scales.<n>We describe our modified model architecture and training recipe.
  arXiv  Detail & Related papers  (2024-12-31T21:55:10Z)
• BgGPT 1.0: Extending English-centric LLMs to other languages [12.867025651644692]
  We present BgGPT-Gemma-2-27B-Instruct and BgGPT-Gemma-2-9B-Instruct: continually pretrained and fine-tuned versions of Google's Gemma-2 models. Our models demonstrate strong performance in Bulgarian language tasks, setting a new standard for language-specific AI models.
  arXiv  Detail & Related papers  (2024-12-14T16:49:52Z)
• Gemma Scope: Open Sparse Autoencoders Everywhere All At Once on Gemma 2 [11.169778211035826]
  Sparse autoencoders (SAEs) are an unsupervised method for learning a neural network's latent representations into seemingly interpretable features. In this work, we introduce Gemma Scope, an open suite of JumpReLU SAEs trained on all layers and sub-layers of Gemma 2 2B and 9B. We evaluate the quality of each SAE on standard metrics and release these results.
  arXiv  Detail & Related papers  (2024-08-09T16:06:42Z)
• Gemma 2: Improving Open Language Models at a Practical Size [118.04200128754249]
  We introduce Gemma 2, a new addition to the Gemma family of lightweight, state-of-the-art open models. We apply several known technical modifications to the Transformer architecture, such as interleaving local-global attentions. The resulting models deliver the best performance for their size, and even offer competitive alternatives to models that are 2-3 times bigger.
  arXiv  Detail & Related papers  (2024-07-31T19:13:07Z)
• RecurrentGemma: Moving Past Transformers for Efficient Open Language Models [103.59785165735727]
  We introduce RecurrentGemma, a family of open language models using Google's novel Griffin architecture. Griffin combines linear recurrences with local attention to achieve excellent performance on language. We provide two sizes of models, containing 2B and 9B parameters, and provide pre-trained and instruction tuned variants for both.
  arXiv  Detail & Related papers  (2024-04-11T15:27:22Z)
• LLaVA-Gemma: Accelerating Multimodal Foundation Models with a Compact Language Model [4.6373877301731]
  We train a suite of multimodal foundation models (MMFM) using the popular LLaVA framework with the recently released Gemma family of large language models (LLMs) We test the effect of ablating three design features: pretraining the connector, utilizing a more powerful image backbone, and increasing the size of the language backbone. The resulting models, which we call LLaVA-Gemma, exhibit moderate performance on an array of evaluations, but fail to improve past the current comparably sized SOTA models.
  arXiv  Detail & Related papers  (2024-03-29T21:32:50Z)
• Gemma: Open Models Based on Gemini Research and Technology [128.57714343844074]
  This work introduces Gemma, a family of lightweight, state-of-the art open models built from the research and technology used to create Gemini models. Gemma models demonstrate strong performance across academic benchmarks for language understanding, reasoning, and safety.
  arXiv  Detail & Related papers  (2024-03-13T06:59:16Z)
• Baichuan 2: Open Large-scale Language Models [51.34140526283222]
  We present Baichuan 2, a series of large-scale multilingual language models containing 7 billion and 13 billion parameters, trained from scratch, on 2.6 trillion tokens. Baichuan 2 matches or outperforms other open-source models of similar size on public benchmarks like MMLU, CMMLU, GSM8K, and HumanEval.
  arXiv  Detail & Related papers  (2023-09-19T04:13:22Z)
• GLaM: Efficient Scaling of Language Models with Mixture-of-Experts [84.33607245023049]
  We propose and develop a family of language models named GLaM (Generalist Language Model) GLaM uses a sparsely activated mixture-of-experts architecture to scale the model capacity while also incurring substantially less training cost compared to dense variants. It consumes only 1/3 of the energy used to train GPT-3 and requires half of the flops for inference, while still achieving better overall zero-shot and one-shot performance across 29 NLP tasks.
  arXiv  Detail & Related papers  (2021-12-13T18:58:19Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.