Gemma 2: Improving Open Language Models at a Practical Size

• URL: http://arxiv.org/abs/2408.00118v3
• Date: Wed, 2 Oct 2024 15:22:49 GMT
• Title: Gemma 2: Improving Open Language Models at a Practical Size
• Abstract summary: We introduce Gemma 2, a new addition to the Gemma family of lightweight, state-of-the-art open models. We apply several known technical modifications to the Transformer architecture, such as interleaving local-global attentions. The resulting models deliver the best performance for their size, and even offer competitive alternatives to models that are 2-3 times bigger.
• Score: 118.04200128754249
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: In this work, we introduce Gemma 2, a new addition to the Gemma family of lightweight, state-of-the-art open models, ranging in scale from 2 billion to 27 billion parameters. In this new version, we apply several known technical modifications to the Transformer architecture, such as interleaving local-global attentions (Beltagy et al., 2020a) and group-query attention (Ainslie et al., 2023). We also train the 2B and 9B models with knowledge distillation (Hinton et al., 2015) instead of next token prediction. The resulting models deliver the best performance for their size, and even offer competitive alternatives to models that are 2-3 times bigger. We release all our models to the community.

Related papers

• Gemma 4 Technical Report [199.38174392182012]
  We introduce Gemma 4, a new generation of open-weight, natively multimodal language models in the Gemma model family.<n>The Gemma 4 model suite features dense and Mixture-of-Experts architectures, ranging from 2.3B to 31B parameters.<n>We propose a unified, encoder-free architecture for our 12B model, which ingests raw audio and image patches.
  arXiv  Detail & Related papers  (2026-07-02T21:08:53Z)
• VaultGemma: A Differentially Private Gemma Model [80.35976343610584]
  We introduce VaultGemma 1B, a 1 billion parameter model within the Gemma family, fully trained with differential privacy.<n>We openly release this model to the community.
  arXiv  Detail & Related papers  (2025-10-15T21:59:53Z)
• Exploring Diffusion Transformer Designs via Grafting [82.91123758506876]
  We present grafting, a simple approach for editing pretrained diffusion transformers (DiTs) to materialize new architectures under small compute budgets.<n>We show that new diffusion model designs can be explored by grafting pretrained DiTs, with edits ranging from operator replacement to architecture restructuring.
  arXiv  Detail & Related papers  (2025-06-05T17:59:40Z)
• Gemma 3 Technical Report [198.3299202423321]
  We introduce Gemma 3, a multimodal addition to the Gemma family of lightweight open models. This version introduces vision understanding abilities, a wider coverage of languages and longer context. We also change the architecture of the model to reduce the KV-cache memory that tends to explode with long context.
  arXiv  Detail & Related papers  (2025-03-25T15:52:34Z)
• BgGPT 1.0: Extending English-centric LLMs to other languages [12.867025651644692]
  We present BgGPT-Gemma-2-27B-Instruct and BgGPT-Gemma-2-9B-Instruct: continually pretrained and fine-tuned versions of Google's Gemma-2 models. Our models demonstrate strong performance in Bulgarian language tasks, setting a new standard for language-specific AI models.
  arXiv  Detail & Related papers  (2024-12-14T16:49:52Z)
• PaliGemma 2: A Family of Versatile VLMs for Transfer [48.68777561571185]
  PaliGemma 2 is an upgrade of the PaliGemma open Vision-Language Model (VLM) based on the Gemma 2 family of language models. We combine the SigLIP-So400m vision encoder that was also used by PaliGemma with the whole range of Gemma 2 models, from the 2B one all the way up to the 27B model. We train these models at three resolutions (224px, 448px, and 896px) in multiple stages to equip them with broad knowledge for transfer via fine-tuning.
  arXiv  Detail & Related papers  (2024-12-04T18:50:42Z)
• RecurrentGemma: Moving Past Transformers for Efficient Open Language Models [103.59785165735727]
  We introduce RecurrentGemma, a family of open language models using Google's novel Griffin architecture. Griffin combines linear recurrences with local attention to achieve excellent performance on language. We provide two sizes of models, containing 2B and 9B parameters, and provide pre-trained and instruction tuned variants for both.
  arXiv  Detail & Related papers  (2024-04-11T15:27:22Z)
• LLaVA-Gemma: Accelerating Multimodal Foundation Models with a Compact Language Model [4.6373877301731]
  We train a suite of multimodal foundation models (MMFM) using the popular LLaVA framework with the recently released Gemma family of large language models (LLMs) We test the effect of ablating three design features: pretraining the connector, utilizing a more powerful image backbone, and increasing the size of the language backbone. The resulting models, which we call LLaVA-Gemma, exhibit moderate performance on an array of evaluations, but fail to improve past the current comparably sized SOTA models.
  arXiv  Detail & Related papers  (2024-03-29T21:32:50Z)
• Gemma: Open Models Based on Gemini Research and Technology [128.57714343844074]
  This work introduces Gemma, a family of lightweight, state-of-the art open models built from the research and technology used to create Gemini models. Gemma models demonstrate strong performance across academic benchmarks for language understanding, reasoning, and safety.
  arXiv  Detail & Related papers  (2024-03-13T06:59:16Z)
• H2O-Danube-1.8B Technical Report [2.6856284636402106]
  We present H2O-Danube, a series of small 1.8B language models. H2O-Danube2-1.8B achieves the top ranking on Open LLM Leaderboard for all models below the 2B parameter range.
  arXiv  Detail & Related papers  (2024-01-30T08:45:08Z)
• Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruning [52.29522018586365]
  We study structured pruning as an effective means to develop smaller LLMs from pre-trained, larger models. Our approach employs two key techniques: (1) targeted structured pruning, which prunes a larger model to a specified target shape by removing layers, heads, and intermediate and hidden dimensions in an end-to-end manner, and (2) dynamic batch loading, which dynamically updates the composition of sampled data in each training batch based on varying losses across different domains.
  arXiv  Detail & Related papers  (2023-10-10T15:13:30Z)
• Specializing Smaller Language Models towards Multi-Step Reasoning [56.78474185485288]
  We show that abilities can be distilled down from GPT-3.5 ($ge$ 175B) to T5 variants ($le$ 11B) We propose model specialization, to specialize the model's ability towards a target task.
  arXiv  Detail & Related papers  (2023-01-30T08:51:19Z)
• What Language Model to Train if You Have One Million GPU Hours? [54.32062236748831]
  We study the impact of different modeling practices and their impact on zero-shot generalization. We also study the performance of a multilingual model and how it compares to the English-only one. All our models and code are open-sourced at https://huggingface.co/bigscience.
  arXiv  Detail & Related papers  (2022-10-27T13:43:27Z)
• GPT-NeoX-20B: An Open-Source Autoregressive Language Model [16.27825182552061]
  GPT-NeoX-20B is a 20 billion parameter autoregressive language model trained on the Pile. Weights will be made freely and openly available to the public through a permissive license.
  arXiv  Detail & Related papers  (2022-04-14T04:00:27Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.