LEAF: Knowledge Distillation of Text Embedding Models with Teacher-Aligned Representations

• URL: http://arxiv.org/abs/2509.12539v1
• Date: Tue, 16 Sep 2025 00:41:05 GMT
• Title: LEAF: Knowledge Distillation of Text Embedding Models with Teacher-Aligned Representations
• Authors: Robin Vujanic, Thomas Rueckstiess,
• Abstract summary: We present LEAF ("Lightweight Embedding Alignment Framework"), a knowledge distillation framework for text embedding models.<n>A key distinguishing feature is that our distilled leaf models are aligned to their teacher.<n>We show that leaf models automatically inherit MRL and robustness to output quantization whenever these properties are present in the teacher model.
• Score: 2.191505742658975
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We present LEAF ("Lightweight Embedding Alignment Framework"), a knowledge distillation framework for text embedding models. A key distinguishing feature is that our distilled leaf models are aligned to their teacher. In the context of information retrieval, this allows for flexible asymmetric architectures where documents are encoded with the larger teacher model, while queries can be served with the smaller leaf models. We also show that leaf models automatically inherit MRL and robustness to output quantization whenever these properties are present in the teacher model, without explicitly training for them. To demonstrate the capability of our framework we publish leaf-ir, a 23M parameters information retrieval oriented text embedding model trained using LEAF, which sets a new state-of-the-art (SOTA) on BEIR, ranking #1 on the public leaderboard for this benchmark and for models of its size. When run in asymmetric mode, its retrieval performance is further increased. Our scheme is however not restricted to the information retrieval setting, and we demonstrate its wider applicability by synthesizing the multi-task leaf-mt model. This also sets a new SOTA, ranking #1 on the public MTEB v2 (English) leaderboard for its size. LEAF is applicable to black-box models and in contrast to other embedding model training frameworks, it does not require judgments nor hard negatives, and training can be conducted using small batch sizes. Thus, dataset and training infrastructure requirements for our framework are modest. We make our models publicly available under a permissive Apache 2.0 license.

Related papers

• LimiX: Unleashing Structured-Data Modeling Capability for Generalist Intelligence [61.46575527504109]
  LimiX-16M and LimiX-2M treat structured data as a joint distribution over variables and missingness.<n>We evaluate LimiX models across 11 large structured-data benchmarks with broad regimes of sample size, feature dimensionality, class number, categorical-to-numerical feature ratio, missingness, and sample-to-feature ratios.
  arXiv  Detail & Related papers  (2025-09-03T17:39:08Z)
• POINTS-Reader: Distillation-Free Adaptation of Vision-Language Models for Document Conversion [32.52489423671728]
  High-quality labeled data is essential for training accurate document conversion models.<n>We propose a fully automated framework comprising two stages for constructing high-quality document extraction datasets.<n>We train a public POINTS-1.5 model to obtain POINTS-Reader, which surpasses many existing public and proprietary models of comparable or larger size.
  arXiv  Detail & Related papers  (2025-09-01T07:54:18Z)
• KaLM-Embedding-V2: Superior Training Techniques and Data Inspire A Versatile Embedding Model [63.13906424204078]
  We propose KaLM-Embedding-V2, a series of versatile and compact embedding models.<n>For model architecture, we implement the models on a 0.5B compact size with simple mean-pooling to produce fixed-length embeddings.<n>For training data, we curate over 20 categories for pre-training and 100 categories for fine-tuning and contrastive distillation.
  arXiv  Detail & Related papers  (2025-06-26T01:09:44Z)
• Matryoshka Model Learning for Improved Elastic Student Models [62.154536258259384]
  MatTA is a framework for training multiple accurate Student models using a novel Teacher-TA-Student recipe.<n>We demonstrate our method on GPT-2 Medium, a public model, and achieve relative improvements of over 24% on SAT Math and over 10% on the LAMBADA benchmark.
  arXiv  Detail & Related papers  (2025-05-29T10:54:58Z)
• KaLM-Embedding: Superior Training Data Brings A Stronger Embedding Model [27.25688303240741]
  KaLM-Embedding is a general multilingual embedding model that leverages a large quantity of cleaner, more diverse, and domain-specific training data.<n>Our model has been trained with key techniques proven to enhance performance.
  arXiv  Detail & Related papers  (2025-01-02T03:17:51Z)
• Jasper and Stella: distillation of SOTA embedding models [8.708650717134008]
  We propose a novel multi-stage distillation framework that enables a smaller student embedding model to distill multiple teacher embedding models.<n>We utilize Matryoshka Representation Learning (MRL) to reduce the vector dimensionality of the student embedding model effectively.<n>Our student model named Jasper with 2 billion parameters, built upon the Stella embedding model, obtained the No.3 position on the Massive Text Embedding Benchmark leaderboard.
  arXiv  Detail & Related papers  (2024-12-26T04:05:28Z)
• VLM2Vec: Training Vision-Language Models for Massive Multimodal Embedding Tasks [60.5257456681402]
  We study the potential for building universal embeddings capable of handling a wide range of downstream tasks.<n>We build a series of VLM2Vec models on SoTA VLMs like Phi-3.5-V, LLaVA-1.6 and evaluate them on MMEB's evaluation split.<n>Our results show that VLM2Vec achieves an absolute average improvement of 10% to 20% over existing multimodal embedding models.
  arXiv  Detail & Related papers  (2024-10-07T16:14:05Z)
• Arctic-Embed: Scalable, Efficient, and Accurate Text Embedding Models [5.2094499417507105]
  This report describes the training dataset creation and recipe behind the family of textttarctic-embed text embedding models. At the time of their release, each model achieved state-of-the-art retrieval accuracy for models of their size on the MTEB Retrieval leaderboard.
  arXiv  Detail & Related papers  (2024-05-08T19:05:18Z)
• Pink: Unveiling the Power of Referential Comprehension for Multi-modal LLMs [49.88461345825586]
  This paper proposes a new framework to enhance the fine-grained image understanding abilities of MLLMs. We present a new method for constructing the instruction tuning dataset at a low cost by leveraging annotations in existing datasets. We show that our model exhibits a 5.2% accuracy improvement over Qwen-VL and surpasses the accuracy of Kosmos-2 by 24.7%.
  arXiv  Detail & Related papers  (2023-10-01T05:53:15Z)
• I2MVFormer: Large Language Model Generated Multi-View Document Supervision for Zero-Shot Image Classification [108.83932812826521]
  Large Language Models (LLM) trained on web-scale text show impressive abilities to repurpose their learned knowledge for a multitude of tasks. Our proposed model, I2MVFormer, learns multi-view semantic embeddings for zero-shot image classification with these class views. I2MVFormer establishes a new state-of-the-art on three public benchmark datasets for zero-shot image classification with unsupervised semantic embeddings.
  arXiv  Detail & Related papers  (2022-12-05T14:11:36Z)
• Large Language Models with Controllable Working Memory [64.71038763708161]
  Large language models (LLMs) have led to a series of breakthroughs in natural language processing (NLP) What further sets these models apart is the massive amounts of world knowledge they internalize during pretraining. How the model's world knowledge interacts with the factual information presented in the context remains under explored.
  arXiv  Detail & Related papers  (2022-11-09T18:58:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.