BitNet Text Embeddings

• URL: http://arxiv.org/abs/2606.25674v1
• Date: Wed, 24 Jun 2026 10:37:01 GMT
• Title: BitNet Text Embeddings
• Abstract summary: We present BITEMBED, an extreme low-bit framework for LLM-based text embedding that jointly targets encoding efficiency and vector storage.<n> BITEMBED converts pretrained LLM backbones into BitNet-style embedding encoders with ternary weights, quantized activations, and lightweight normalization refinement.<n> Experiments on MMTEB (eng) with Qwen3-0.6B and Gemma703-2M show that BITEMBED is largely comparable to full precision teacher embedders.
• Score: 104.29781473344813
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: LLM-based text embedders have substantially improved retrieval and semantic representation quality, but their deployment remains costly: large backbone models slow down embedding inference, while high-dimensional full-precision embeddings impose substantial storage and bandwidth overhead on large-scale indexes. In this paper, we present BITEMBED, an extreme low-bit framework for LLM-based text embedding that jointly targets encoding efficiency and vector storage. BITEMBED converts pretrained LLM backbones into BitNet-style embedding encoders with ternary weights, quantized activations, and lightweight normalization refinement. The converted model is adapted to representation learning through continual contrastive pre-training, followed by supervised contrastive fine-tuning with both similarity-distribution distillation and attention-relation distillation from a full-precision teacher. Beyond quantizing the backbone, BITEMBED further trains output embeddings to support multiple storage precisions meeting different storage needs in various scenarios. Experiments on MMTEB (eng, v2) with Qwen3-0.6B and Gemma3-270M show that BITEMBED is largely comparable to full precision teacher embedders. Moreover, BITEMBED flexibly obtains text embeddings of various precisions, achieving a trade-off between performance and storage cost.

Related papers

• Dual-Grained Agent Memory and Shapley Context Attribution for Multimodal Agentic Learner [55.80248281761638]
  multimodal large language models (MLLMs) deliver impressive perception yet still falter on scientific and mathematical reasoning.<n>We propose DG-Mem, a dual-grained agentic memory framework that augments a frozen MLLM with a non- externally stored memory built once from training-time rollouts and consulted read-only at test time.
  arXiv  Detail & Related papers  (2026-08-24T13:56:01Z)
• Extending LLM Context via Associative Recurrent Memory [31.93460198288273]
  We investigate the Associative Recurrent Memory Transformer (ARMT) as a practical approach for enabling long-context processing in large language models (LLMs)<n>Second, we propose a comprehensive training recipe for ARMT-based context extension, combining continued pre-training, synthetic long-context data generation, curriculum learning, and selective integration of associative memory into chosen model layers.<n>Third, we present an extensive experimental study demonstrating that ARMT-augmented models: (i) process inputs well beyond their original context limits without degrading performance relative to in-limit baselines; (ii) generalize more effectively
  arXiv  Detail & Related papers  (2026-07-13T14:37:24Z)
• SAB-LVLM: Significance-Aware Binarization for Large Vision-Language Models [75.75091459417304]
  Large Vision-Language Models (LVLMs) have achieved remarkable progress in multimodal understanding.<n>LVLMs incur substantial memory and latency overhead, severely limiting real-world deployment on resource-constrained devices.<n>We propose a binarization method to drastically reduce storage and computational costs.
  arXiv  Detail & Related papers  (2026-07-02T08:30:00Z)
• Multi-Bitwidth Quantization for LLMs Using Additive Codebooks [12.237109162791091]
  Large language models (LLMs) are increasingly deployed across heterogeneous hardware with varying resource constraints.<n>We propose Drop-by-Drop, a novel multi-bitwidth post-training quantization framework that enables inference-time precision control over LLM weights from a single trained model.
  arXiv  Detail & Related papers  (2026-06-11T04:06:02Z)
• MemDLM: Memory-Enhanced DLM Training [56.40248490616793]
  Diffusion Language Models (DLMs) offer attractive advantages over Auto-Regressive (AR) models.<n>They suffer from a notable train-inference mismatch.<n>We propose MemDLM (Memory-Enhanced DLM), which embeds a simulated denoising process into training.
  arXiv  Detail & Related papers  (2026-03-23T17:39:56Z)
• Fine-tuning MLLMs Without Forgetting Is Easier Than You Think [72.59321247529975]
  We design a 2x2 experimental framework to assess model performance across in-distribution and out-of-distribution image and text inputs.<n>Our results show that appropriate regularization, such as constraining the number of trainable parameters or adopting a low learning rate, effectively prevents forgetting when dealing with out-of-distribution images.<n>We attribute this forgetting as task-specific overfitting and address this issue by introducing a data-hybrid training strategy.
  arXiv  Detail & Related papers  (2026-03-15T17:16:19Z)
• TABED: Test-Time Adaptive Ensemble Drafting for Robust Speculative Decoding in LVLMs [14.030784220154151]
  We propose Test-time Adaptive Batched Ensemble Drafting (TABED) for Large Vision-Language Models.<n> TABED ensembles multiple drafts obtained via batch inference by leveraging deviations from past ground truths available in the SD setting.<n>It achieves an average robust walltime speedup of 1.74x over autoregressive decoding and a 5% improvement over single drafting methods.
  arXiv  Detail & Related papers  (2026-01-28T08:16:57Z)
• ReMatch: Boosting Representation through Matching for Multimodal Retrieval [29.610030065465793]
  ReMatch is a framework that leverages the generative strength of MLLMs for multimodal retrieval.<n>We train the embedding MLLM end-to-end with a chat-style generative matching stage.<n>Our experiments show particularly strong zero-shot generalization results on five datasets.
  arXiv  Detail & Related papers  (2025-11-24T16:28:49Z)
• Learning Grouped Lattice Vector Quantizers for Low-Bit LLM Compression [57.54335545892155]
  We introduce a Grouped Lattice Vector Quantization (GLVQ) framework that assigns each group of weights a customized lattice codebook.<n>Our approach achieves a better trade-off between model size and accuracy compared to existing post-training quantization baselines.
  arXiv  Detail & Related papers  (2025-10-23T20:19:48Z)
• SitEmb-v1.5: Improved Context-Aware Dense Retrieval for Semantic Association and Long Story Comprehension [77.93156509994994]
  We show how to represent short chunks in a way that is conditioned on a broader context window to enhance retrieval performance.<n>Existing embedding models are not well-equipped to encode such situated context effectively.<n>Our method substantially outperforms state-of-the-art embedding models.
  arXiv  Detail & Related papers  (2025-08-03T23:59:31Z)
• Binary Embedding-based Retrieval at Tencent [30.44247353560061]
  Large-scale embedding-based retrieval (EBR) is the cornerstone of search-related industrial applications. We propose a binary embedding-based retrieval engine equipped with a recurrent binarization algorithm that enables customized bits per dimension. We successfully employed the introduced BEBR to Tencent products, including Sogou, Tencent Video, QQ World, etc.
  arXiv  Detail & Related papers  (2023-02-17T06:10:02Z)
• Adapted Multimodal BERT with Layer-wise Fusion for Sentiment Analysis [84.12658971655253]
  We propose Adapted Multimodal BERT, a BERT-based architecture for multimodal tasks. adapter adjusts the pretrained language model for the task at hand, while the fusion layers perform task-specific, layer-wise fusion of audio-visual information with textual BERT representations. In our ablations we see that this approach leads to efficient models, that can outperform their fine-tuned counterparts and are robust to input noise.
  arXiv  Detail & Related papers  (2022-12-01T17:31:42Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.