BGE M3-Embedding: Multi-Lingual, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillation

• URL: http://arxiv.org/abs/2402.03216v4
• Date: Fri, 28 Jun 2024 09:55:49 GMT
• Title: BGE M3-Embedding: Multi-Lingual, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillation
• Authors: Jianlv Chen, Shitao Xiao, Peitian Zhang, Kun Luo, Defu Lian, Zheng Liu,
• Abstract summary: We present a new embedding model, called M3-Embedding, which is distinguished for its versatility in Multi-Linguality, Multi-Functionality, and Multi-Granularity. It can support more than 100 working languages, leading to new state-of-the-art performances on multi-lingual and cross-lingual retrieval tasks. M3-Embedding is able to process inputs of different granularities, spanning from short sentences to long documents of up to 8192 tokens.
• Score: 26.65107475147534
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: In this paper, we present a new embedding model, called M3-Embedding, which is distinguished for its versatility in Multi-Linguality, Multi-Functionality, and Multi-Granularity. It can support more than 100 working languages, leading to new state-of-the-art performances on multi-lingual and cross-lingual retrieval tasks. It can simultaneously perform the three common retrieval functionalities of embedding model: dense retrieval, multi-vector retrieval, and sparse retrieval, which provides a unified model foundation for real-world IR applications. It is able to process inputs of different granularities, spanning from short sentences to long documents of up to 8192 tokens. The effective training of M3-Embedding involves the following technical contributions. We propose a novel self-knowledge distillation approach, where the relevance scores from different retrieval functionalities can be integrated as the teacher signal to enhance the training quality. We also optimize the batching strategy, enabling a large batch size and high training throughput to ensure the discriminativeness of embeddings. To the best of our knowledge, M3-Embedding is the first embedding model which realizes such a strong versatility. The model and code will be publicly available at https://github.com/FlagOpen/FlagEmbedding.

Related papers

• Qwen3 Embedding: Advancing Text Embedding and Reranking Through Foundation Models [90.54780244175511]
  We introduce the Qwen3 Embedding series, a significant advancement over its predecessor, the GTE-Qwen series.<n>The Qwen3 Embedding series offers a spectrum of model sizes for both embedding and reranking tasks.<n>The Qwen3 Embedding series achieves state-of-the-art results across diverse benchmarks.
  arXiv  Detail & Related papers  (2025-06-05T15:49:48Z)
• Tevatron 2.0: Unified Document Retrieval Toolkit across Scale, Language, and Modality [74.59049806800176]
  This demo paper highlights the Tevatron toolkit's key features, bridging academia and industry.<n>We showcase a unified dense retriever achieving strong multilingual and multimodal effectiveness.<n>We also release OmniEmbed, to the best of our knowledge, the first embedding model that unifies text, image document, video, and audio retrieval.
  arXiv  Detail & Related papers  (2025-05-05T08:52:49Z)
• ARMOR v0.1: Empowering Autoregressive Multimodal Understanding Model with Interleaved Multimodal Generation via Asymmetric Synergy [14.703591553247948]
  ARMOR is a framework that achieves both understanding and generation by fine-tuning existing multimodal large language models. ARMOR extends existing MLLMs from three perspectives: model architecture, training data, and training algorithm. Experimental results demonstrate that ARMOR upgrades existing MLLMs to UniMs with promising image generation capabilities.
  arXiv  Detail & Related papers  (2025-03-09T10:15:39Z)
• LUSIFER: Language Universal Space Integration for Enhanced Multilingual Embeddings with Large Language Models [89.13128402847943]
  We present LUSIFER, a novel zero-shot approach that adapts LLM-based embedding models for multilingual tasks without requiring multilingual supervision. LUSIFER's architecture combines a multilingual encoder, serving as a language-universal learner, with an LLM-based embedding model optimized for embedding-specific tasks. We introduce a new benchmark encompassing 5 primary embedding tasks, 123 diverse datasets, and coverage across 14 languages.
  arXiv  Detail & Related papers  (2025-01-01T15:43:07Z)
• VLM2Vec: Training Vision-Language Models for Massive Multimodal Embedding Tasks [60.5257456681402]
  We build universal embedding models capable of handling a wide range of downstream tasks. Our contributions are twofold: (1) MMEB (Massive Multimodal Embedding Benchmark), which covers 4 meta-tasks (i.e. classification, visual question answering, multimodal retrieval, and visual grounding) and 36 datasets, including 20 training and 16 evaluation datasets, and (2) VLM2Vec (Vision-Language Model -> Vector), a contrastive training framework that converts any state-of-the-art vision-language model into an embedding model via training on MMEB.
  arXiv  Detail & Related papers  (2024-10-07T16:14:05Z)
• NVLM: Open Frontier-Class Multimodal LLMs [64.00053046838225]
  We introduce NVLM 1.0, a family of frontier-class multimodal large language models (LLMs) that achieve state-of-the-art results on vision-language tasks. We propose a novel architecture that enhances both training efficiency and multimodal reasoning capabilities. We develop production-grade multimodality for the NVLM-1.0 models, enabling them to excel in vision-language tasks.
  arXiv  Detail & Related papers  (2024-09-17T17:59:06Z)
• M3: A Multi-Task Mixed-Objective Learning Framework for Open-Domain Multi-Hop Dense Sentence Retrieval [12.277521531556852]
  M3 is a novel Multi-hop dense sentence retrieval system built upon a novel Multi-task Mixed-objective approach for dense text representation learning. Our approach yields state-of-the-art performance on a large-scale open-domain fact verification benchmark dataset, FEVER.
  arXiv  Detail & Related papers  (2024-03-21T01:52:07Z)
• Generative Multi-Modal Knowledge Retrieval with Large Language Models [75.70313858231833]
  We propose an innovative end-to-end generative framework for multi-modal knowledge retrieval. Our framework takes advantage of the fact that large language models (LLMs) can effectively serve as virtual knowledge bases. We demonstrate significant improvements ranging from 3.0% to 14.6% across all evaluation metrics when compared to strong baselines.
  arXiv  Detail & Related papers  (2024-01-16T08:44:29Z)
• UniDoc: A Universal Large Multimodal Model for Simultaneous Text Detection, Recognition, Spotting and Understanding [93.92313947913831]
  We introduce UniDoc, a novel multimodal model equipped with text detection and recognition capabilities. To the best of our knowledge, this is the first large multimodal model capable of simultaneous text detection, recognition, spotting, and understanding.
  arXiv  Detail & Related papers  (2023-08-19T17:32:34Z)
• Towards General Text Embeddings with Multi-stage Contrastive Learning [20.803769345818456]
  GTE is a general-purpose text embedding model trained with multi-stage contrastive learning. We train a unified text embedding model by employing contrastive learning over a diverse mixture of datasets from multiple sources.
  arXiv  Detail & Related papers  (2023-08-07T03:52:59Z)
• Macaw-LLM: Multi-Modal Language Modeling with Image, Audio, Video, and Text Integration [50.94902442781148]
  We propose a novel multi-modal large language model (LLM) that seamlessly integrates visual, audio, and textual information. Macaw-LLM consists of three main components: a modality module for encoding multi-modal data, a cognitive module for harnessing pretrained LLMs, and an alignment module for harmonizing diverse representations. We construct a large-scale multi-modal instruction dataset in terms of multi-turn dialogue, including 69K image instances and 50K video instances.
  arXiv  Detail & Related papers  (2023-06-15T12:45:25Z)
• Unsupervised Multimodal Language Representations using Convolutional Autoencoders [5.464072883537924]
  We propose extracting unsupervised Multimodal Language representations that are universal and can be applied to different tasks. We map the word-level aligned multimodal sequences to 2-D matrices and then use Convolutional Autoencoders to learn embeddings by combining multiple datasets. It is also shown that our method is extremely lightweight and can be easily generalized to other tasks and unseen data with small performance drop and almost the same number of parameters.
  arXiv  Detail & Related papers  (2021-10-06T18:28:07Z)
• M3P: Learning Universal Representations via Multitask Multilingual Multimodal Pre-training [119.16007395162431]
  M3P is a Multilingual Multimodal Pre-trained model that combines multilingual pre-training and multimodal pre-training. We show that M3P can achieve comparable results for English and new state-of-the-art results for non-English languages.
  arXiv  Detail & Related papers  (2020-06-04T03:54:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.