ML-Embed: Inclusive and Efficient Embeddings for a Multilingual World

• URL: http://arxiv.org/abs/2605.15081v1
• Date: Thu, 14 May 2026 17:05:26 GMT
• Title: ML-Embed: Inclusive and Efficient Embeddings for a Multilingual World
• Abstract summary: ML-Embed is a suite of inclusive and efficient models built upon a new framework: 3-Dimensional Matryoshka Learning (3D-ML)<n>To address the linguistic challenge, we curate a massively multilingual dataset and train a suite of models ranging from 140M to 8B parameters.<n>In a direct commitment to transparency, we release all models, data, and code.
• Score: 19.088644745246373
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: The development of high-quality text embeddings is increasingly drifting toward an exclusionary future, defined by three critical barriers: prohibitive computational costs, a narrow linguistic focus that neglects most of the world's languages, and a lack of transparency from closed-source or open-weight models that stifles research. To dismantle these barriers, we introduce ML-Embed, a suite of inclusive and efficient models built upon a new framework: 3-Dimensional Matryoshka Learning (3D-ML). Our framework addresses the computational challenge with comprehensive efficiency across the entire model lifecycle. Beyond the storage benefits of Matryoshka Representation Learning (MRL) and flexible inference-time depth provided by Matryoshka Layer Learning (MLL), we introduce Matryoshka Embedding Learning (MEL) for enhanced parameter efficiency. To address the linguistic challenge, we curate a massively multilingual dataset and train a suite of models ranging from 140M to 8B parameters. In a direct commitment to transparency, we release all models, data, and code. Extensive evaluation on 430 tasks demonstrates that our models set new records on 9 of 17 evaluated MTEB benchmarks, with particularly strong results in low-resource languages, providing a reproducible blueprint for building globally equitable and computationally efficient AI systems.

Related papers

• Annotation-Efficient Vision-Language Model Adaptation to the Polish Language Using the LLaVA Framework [38.98519875112922]
  Vision-language models (VLMs) are trained on English-centric data, limiting their performance in other languages and cultural contexts.<n>We reproduce and adapt the LLaVA-Next methodology to create a set of Polish VLMs.<n>We observe a +9.5% improvement over LLaVA-1.6-una-13B on a Polish-adapted MMBench, along with higher-quality captions in generative evaluations.
  arXiv  Detail & Related papers  (2026-02-15T09:54:40Z)
• Large Language Model Data Generation for Enhanced Intent Recognition in German Speech [14.788624194380825]
  Intent recognition (IR) for speech commands is essential for artificial intelligence (AI) assistant systems.<n>We propose a novel approach that combines an adapted Whisper ASR model, fine-tuned on elderly German speech.<n>We generate synthetic speech with a text-to-speech model and conduct extensive cross-dataset testing.
  arXiv  Detail & Related papers  (2025-08-08T12:54:09Z)
• LESS: Large Language Model Enhanced Semi-Supervised Learning for Speech Foundational Models Using in-the-wild Data [5.021795689551854]
  LESS (Large Language Model Enhanced Semi-supervised Learning) is a versatile framework that uses Large Language Models (LLMs) to correct pseudo-labels generated on in-the-wild data.<n>Across Mandarin ASR and Spanish-to-English AST evaluations, LESS delivers consistent gains.<n>We have released the recipe as open source to facilitate further research in this area.
  arXiv  Detail & Related papers  (2025-06-05T03:00:04Z)
• Enhancing Multilingual LLM Pretraining with Model-Based Data Selection [33.68104398807581]
  We propose a model-based filtering framework for multilingual datasets.<n>Our approach emphasizes transparency, simplicity, and efficiency.<n>We extend our framework to 20 languages for which we release the refined pretraining datasets.
  arXiv  Detail & Related papers  (2025-02-14T18:42:07Z)
• Enhancing Code Generation for Low-Resource Languages: No Silver Bullet [55.39571645315926]
  Large Language Models (LLMs) rely on large and diverse datasets to learn syntax, semantics, and usage patterns of programming languages.<n>For low-resource languages, the limited availability of such data hampers the models' ability to generalize effectively.<n>We present an empirical study investigating the effectiveness of several approaches for boosting LLMs' performance on low-resource languages.
  arXiv  Detail & Related papers  (2025-01-31T12:23:28Z)
• OmniBench: Towards The Future of Universal Omni-Language Models [63.16606414452612]
  We introduce OmniBench, a novel benchmark designed to evaluate models' ability to recognize, interpret, and reason across visual, acoustic, and textual inputs simultaneously.<n>Our evaluation reveals that open-source OLMs show significant limitations in instruction-following and reasoning in tri-modal contexts.<n>We advocate for developing more robust tri-modal integration techniques and training strategies to enhance OLM performance.
  arXiv  Detail & Related papers  (2024-09-23T17:59:05Z)
• ULLME: A Unified Framework for Large Language Model Embeddings with Generation-Augmented Learning [72.90823351726374]
  We introduce the Unified framework for Large Language Model Embedding (ULLME), a flexible, plug-and-play implementation that enables bidirectional attention across various LLMs. We also propose Generation-augmented Representation Learning (GRL), a novel fine-tuning method to boost LLMs for text embedding tasks. To showcase our framework's flexibility and effectiveness, we release three pre-trained models from ULLME with different backbone architectures.
  arXiv  Detail & Related papers  (2024-08-06T18:53:54Z)
• Unlocking the Potential of Model Merging for Low-Resource Languages [66.7716891808697]
  Adapting large language models to new languages typically involves continual pre-training (CT) followed by supervised fine-tuning (SFT) We propose model merging as an alternative for low-resource languages, combining models with distinct capabilities into a single model without additional training. Experiments based on Llama-2-7B demonstrate that model merging effectively endows LLMs for low-resource languages with task-solving abilities, outperforming CT-then-SFT in scenarios with extremely scarce data.
  arXiv  Detail & Related papers  (2024-07-04T15:14:17Z)
• Data-Augmentation-Based Dialectal Adaptation for LLMs [26.72394783468532]
  This report presents GMUNLP's participation to the Dialect-Copa shared task at VarDial 2024. The task focuses on evaluating the commonsense reasoning capabilities of large language models (LLMs) on South Slavic micro-dialects. We propose an approach that combines the strengths of different types of language models and leverages data augmentation techniques to improve task performance.
  arXiv  Detail & Related papers  (2024-04-11T19:15:32Z)
• Supervised Knowledge Makes Large Language Models Better In-context Learners [94.89301696512776]
  Large Language Models (LLMs) exhibit emerging in-context learning abilities through prompt engineering. The challenge of improving the generalizability and factuality of LLMs in natural language understanding and question answering remains under-explored. We propose a framework that enhances the reliability of LLMs as it: 1) generalizes out-of-distribution data, 2) elucidates how LLMs benefit from discriminative models, and 3) minimizes hallucinations in generative tasks.
  arXiv  Detail & Related papers  (2023-12-26T07:24:46Z)
• LAMM: Language-Assisted Multi-Modal Instruction-Tuning Dataset, Framework, and Benchmark [81.42376626294812]
  We present Language-Assisted Multi-Modal instruction tuning dataset, framework, and benchmark. Our aim is to establish LAMM as a growing ecosystem for training and evaluating MLLMs. We present a comprehensive dataset and benchmark, which cover a wide range of vision tasks for 2D and 3D vision.
  arXiv  Detail & Related papers  (2023-06-11T14:01:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.