EmbeddingGemma: Powerful and Lightweight Text Representations

• URL: http://arxiv.org/abs/2509.20354v3
• Date: Sat, 01 Nov 2025 23:38:27 GMT
• Title: EmbeddingGemma: Powerful and Lightweight Text Representations
• Abstract summary: EmbeddingGemma is a new lightweight, open text embedding model based on the Gemma 3 language family.<n>We improve model robustness and expressiveness with a spread-out regularizer.<n>We release EmbeddingGemma to the community to promote further research.
• Score: 42.34520055523387
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We introduce EmbeddingGemma, a new lightweight, open text embedding model based on the Gemma 3 language model family. Our innovative training recipe strategically captures knowledge from larger models via encoder-decoder initialization and geometric embedding distillation. We improve model robustness and expressiveness with a spread-out regularizer, and ensure generalizability by merging checkpoints from varied, optimized mixtures. Evaluated on the Massive Text Embedding Benchmark (MTEB) across multilingual, English, and code domains, EmbeddingGemma (300M) achieves state-of-the-art results. Notably, it outperforms prior top models, both proprietary and open, with fewer than 500M parameters, and provides performance comparable to models double its size, offering an exceptional performance-to-cost ratio. Remarkably, this lead persists when quantizing model weights or truncating embedding outputs. This makes EmbeddingGemma particularly well-suited for low-latency and high-throughput use cases such as on-device applications. We provide ablation studies exploring our key design choices. We release EmbeddingGemma to the community to promote further research.

Related papers

• jina-embeddings-v5-omni: Geometry-preserving Embeddings via Locked Aligned Towers [4.3450882150663235]
  GELATO is a novel approach to multimodal embedding models.<n>GELATO produces results that are competitive with the state-of-the-art.
  arXiv  Detail & Related papers  (2026-05-08T18:45:15Z)
• TranslateGemma Technical Report [39.30046490326195]
  We present TranslateGemma, a suite of open machine translation models based on the Gemma 3 foundation models.<n>We show consistent and substantial gains over the baseline Gemma 3 models across all sizes.<n>We also show that TranslateGemma models retain strong multimodal capabilities, with enhanced performance on the Vistra image translation benchmark.
  arXiv  Detail & Related papers  (2026-01-13T22:23:24Z)
• ReasonEmbed: Enhanced Text Embeddings for Reasoning-Intensive Document Retrieval [46.111605335278746]
  ReasonEmbed is a novel text embedding model developed for reasoning-intensive document retrieval.<n>ReMixer is a new data synthesis method that overcomes the triviality problem prevalent in previous synthetic datasets.<n>Redapter is a self-adaptive learning algorithm that dynamically adjusts training each sample's weight based on its reasoning intensity.
  arXiv  Detail & Related papers  (2025-10-09T14:10:26Z)
• An Enhanced Model-based Approach for Short Text Clustering [58.60681789677676]
  Short text clustering has become increasingly important with the popularity of social media like Twitter, Google+, and Facebook.<n>Existing methods can be broadly categorized into two paradigms: topic model-based approaches and deep representation learning-based approaches.<n>We propose a collapsed Gibbs Sampling algorithm for the Dirichlet Multinomial Mixture model (GSDMM), which effectively handles the sparsity and high dimensionality of short texts.<n>Based on several aspects of GSDMM that warrant further refinement, we propose an improved approach, GSDMM+, designed to further optimize its performance.
  arXiv  Detail & Related papers  (2025-07-18T10:07:42Z)
• Do We Really Need Specialization? Evaluating Generalist Text Embeddings for Zero-Shot Recommendation and Search [51.30225948318648]
  Generalist Text Embedding Models (GTEs) are pre-trained on large-scale corpora.<n>We show that GTEs can guarantee strong zero-shot performance without specialized adaptation.<n>We also show that compressing embedding dimensions by focusing on the most informative directions effectively reduces noise.
  arXiv  Detail & Related papers  (2025-07-07T13:41:52Z)
• KaLM-Embedding-V2: Superior Training Techniques and Data Inspire A Versatile Embedding Model [46.95431131609286]
  KaLM-Embedding-V2, a versatile and compact embedding model, achieves impressive performance in general-purpose text embedding tasks.<n>Key innovations include: (i) pre-training on large-scale weakly supervised open-source corpora; (ii) fine-tuning on high-quality retrieval and non-retrieval datasets; and (iii) model-soup parameter averaging for robust generalization.
  arXiv  Detail & Related papers  (2025-06-26T01:09:44Z)
• Gated Attention for Large Language Models: Non-linearity, Sparsity, and Attention-Sink-Free [81.65559031466452]
  We conduct experiments to investigate gating-augmented softmax attention variants.<n>We find that a simple modification-applying a head-specific sigmoid gate after the Scaled Dot-Product Attention (SDPA)-consistently improves performance.
  arXiv  Detail & Related papers  (2025-05-10T17:15:49Z)
• Training Sparse Mixture Of Experts Text Embedding Models [0.0]
  Transformer-based text embedding models have improved their performance on benchmarks like MIRACL and BEIR by increasing their parameter counts.<n>This scaling approach introduces significant deployment challenges, including increased inference latency and memory usage.<n>We introduce Nomic Embed v2, the first general purpose MoE text embedding model.
  arXiv  Detail & Related papers  (2025-02-11T21:36:31Z)
• Boosting Few-Shot Detection with Large Language Models and Layout-to-Image Synthesis [1.1633929083694388]
  We propose a framework for enhancing few-shot detection beyond state-of-the-art generative augmentation approaches. We introduce our novel layout-aware CLIP score for sample ranking, enabling tight coupling between generated layouts and images. With our approach, a YOLOX-S baseline is boosted by more than 140%, 50%, 35% in mAP on the COCO 5-,10-, and 30-shot settings.
  arXiv  Detail & Related papers  (2024-10-09T12:57:45Z)
• mGTE: Generalized Long-Context Text Representation and Reranking Models for Multilingual Text Retrieval [67.50604814528553]
  We first introduce a text encoder enhanced with RoPE and unpadding, pre-trained in a native 8192-token context. Then we construct a hybrid TRM and a cross-encoder reranker by contrastive learning.
  arXiv  Detail & Related papers  (2024-07-29T03:12:28Z)
• Arctic-Embed: Scalable, Efficient, and Accurate Text Embedding Models [5.2094499417507105]
  This report describes the training dataset creation and recipe behind the family of textttarctic-embed text embedding models. At the time of their release, each model achieved state-of-the-art retrieval accuracy for models of their size on the MTEB Retrieval leaderboard.
  arXiv  Detail & Related papers  (2024-05-08T19:05:18Z)
• Z-Code++: A Pre-trained Language Model Optimized for Abstractive Summarization [108.09419317477986]
  Z-Code++ is a new pre-trained language model optimized for abstractive text summarization. The model is first pre-trained using text corpora for language understanding, and then is continually pre-trained on summarization corpora for grounded text generation. Our model is parameter-efficient in that it outperforms the 600x larger PaLM-540B on XSum, and the finetuned 200x larger GPT3-175B on SAMSum.
  arXiv  Detail & Related papers  (2022-08-21T01:00:54Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.