BitNet Text Embeddings
Abstract Overview
BitEmbed is an extreme low-bit framework for LLM-based text embeddings that aims to reduce both encoder inference cost and embedding storage cost. It converts pretrained LLM backbones into BitNet-style embedding encoders with ternary weights, quantized activations, and added sub-layer normalization for training stability. The training pipeline combines continual contrastive pre-training with supervised fine-tuning guided by a full-precision teacher through similarity-distribution and attention-relation distillation. The method also trains embeddings to support multiple output precisions so that one checkpoint can trade off storage and quality across deployment settings.
Novelty
The paper presents what it describes as the first systematic study of extreme low-bit quantization specifically for LLM-based text embedding, rather than general language modeling. Its distinctive contribution is the joint treatment of low-bit backbone quantization and multi-precision output embedding training within a distillation-based embedding framework.
Results
On MMTEB (eng, v2), BitEmbed stays close to full-precision teachers on both tested backbones: 67.60 vs. 67.95 for Qwen3-0.6B and 66.10 vs. 66.71 for Gemma3-270M. It also increases CPU token throughput from 364.36 to 830.50 on Qwen3-0.6B and from 1181.28 to 2055.47 on Gemma3-270M. Multi-precision output embeddings provide a storage-performance trade-off, with 8-bit and 4-bit embeddings reported as nearly lossless relative to the 16-bit branch and 1-bit/2-bit embeddings retaining usable performance.
Key Points
- BitEmbed quantizes LLM embedding backbones to ternary weights with quantized activations and stabilizes training using sub-layer normalization, continual contrastive pre-training, and teacher-guided distillation.
- The method is evaluated on MMTEB (eng, v2) with Qwen3-0.6B and Gemma3-270M, where it remains close to full-precision teacher performance while substantially improving CPU throughput.
- A single trained model supports 1-, 2-, 4-, 8-, and 16-bit output embeddings, enabling explicit trade-offs between vector storage cost and downstream embedding quality.