F2LLM Technical Report: Matching SOTA Embedding Performance with 6 Million Open-Source Data

• URL: http://arxiv.org/abs/2510.02294v1
• Date: Thu, 02 Oct 2025 17:58:49 GMT
• Title: F2LLM Technical Report: Matching SOTA Embedding Performance with 6 Million Open-Source Data
• Abstract summary: We introduce F2LLM, a suite of state-of-the-art embedding models in three sizes: 0.6B, 1.7B, and 4B.<n>F2LLM is directly finetuned from foundation models on 6 million query-document-negatives curated from open-source, non-synthetic datasets.<n>On the MTEB English leaderboard, F2LLM-4B ranks 2nd among models with approximately 4B parameters and 7th overall, while F2LLM-1.7B ranks 1st among models in the 1B-2B size range.
• Score: 19.088644745246373
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: We introduce F2LLM - Foundation to Feature Large Language Models, a suite of state-of-the-art embedding models in three sizes: 0.6B, 1.7B, and 4B. Unlike previous top-ranking embedding models that require massive contrastive pretraining, sophisticated training pipelines, and costly synthetic training data, F2LLM is directly finetuned from foundation models on 6 million query-document-negative tuples curated from open-source, non-synthetic datasets, striking a strong balance between training cost, model size, and embedding performance. On the MTEB English leaderboard, F2LLM-4B ranks 2nd among models with approximately 4B parameters and 7th overall, while F2LLM-1.7B ranks 1st among models in the 1B-2B size range. To facilitate future research in the field, we release the models, training dataset, and code, positioning F2LLM as a strong, reproducible, and budget-friendly baseline for future works.

Related papers

• TabH2O: A Unified Foundation Model for Tabular Prediction [12.882141870947706]
  We present TabH2O, a model that performs classification and regression in a single forward pass via in-context learning.<n>We evaluate TabH2O v1 (29.2M parameters) on the TALENT benchmark, where it achieves an average rank of 2.55 out of 6 evaluated methods.
  arXiv  Detail & Related papers  (2026-05-18T13:27:54Z)
• Chain-of-Models Pre-Training: Rethinking Training Acceleration of Vision Foundation Models [16.42169130504086]
  Chain-of-Models Pre-Training (CoM-PT) is a performance-lossless training acceleration method for vision foundation models.<n>CoM-PT is designed to accelerate the training pipeline at the model family level, scaling efficiently as the model family expands.<n>It is extensively validated across 45 datasets spanning zero-shot and fine-tuning tasks.
  arXiv  Detail & Related papers  (2026-04-14T07:26:23Z)
• F2LLM-v2: Inclusive, Performant, and Efficient Embeddings for a Multilingual World [19.088644745246373]
  F2LLM-v2 is a new family of general-purpose, multilingual embedding models in 8 distinct sizes ranging from 80M to 14B.<n>Trained on a newly curated composite of 60 million publicly available high-quality data samples, F2LLM-v2 supports more than 200 languages.
  arXiv  Detail & Related papers  (2026-03-19T17:59:21Z)
• MobileLLM-R1: Exploring the Limits of Sub-Billion Language Model Reasoners with Open Training Recipes [60.57770396565211]
  We show that strong reasoning abilities can emerge with far less data.<n>MobileLLM-R50M achieves an AIME score of 15.5, compared to just 0.6 for OLMo-2-1.48B and 0.3 for SmolLM-2-1.7B.
  arXiv  Detail & Related papers  (2025-09-29T15:43:59Z)
• LimiX: Unleashing Structured-Data Modeling Capability for Generalist Intelligence [61.46575527504109]
  LimiX-16M and LimiX-2M treat structured data as a joint distribution over variables and missingness.<n>We evaluate LimiX models across 11 large structured-data benchmarks with broad regimes of sample size, feature dimensionality, class number, categorical-to-numerical feature ratio, missingness, and sample-to-feature ratios.
  arXiv  Detail & Related papers  (2025-09-03T17:39:08Z)
• Approximating Language Model Training Data from Weights [70.08614275061689]
  We formalize the problem of data approximation from model weights and propose several baselines and metrics.<n>We develop a gradient-based approach that selects the highest-matching data from a large public text corpus.<n>Even when none of the true training data is known, our method is able to locate a small subset of public Web documents.
  arXiv  Detail & Related papers  (2025-06-18T15:26:43Z)
• 2 OLMo 2 Furious [154.15728448754854]
  We present OLMo 2, the next generation of our fully open language models.<n> OLMo 2 includes a family of dense autoregressive language models at 7B, 13B and 32B scales.<n>We describe our modified model architecture and training recipe.
  arXiv  Detail & Related papers  (2024-12-31T21:55:10Z)
• AquilaMoE: Efficient Training for MoE Models with Scale-Up and Scale-Out Strategies [36.645912291368546]
  We present AquilaMoE, a cutting-edge bilingual 8*16B Mixture of Experts (MoE) language model with 8 experts with 16 billion parameters each. This approach optimize performance while minimizing data requirements through a two-stage process. We successfully trained a 16B model and subsequently the 8*16B AquilaMoE model, demonstrating significant improvements in performance and training efficiency.
  arXiv  Detail & Related papers  (2024-08-13T02:07:00Z)
• Compact Language Models via Pruning and Knowledge Distillation [61.56557874432008]
  Minitron models exhibit up to a 16% improvement in MMLU scores compared to training from scratch. Deriving 8B and 4B models from an already pretrained 15B model using our approach requires up to 40x fewer training tokens per model compared to training from scratch.
  arXiv  Detail & Related papers  (2024-07-19T21:47:57Z)
• A Teacher Is Worth A Million Instructions [4.322454918650575]
  Fine-tuning Mistral 7B and 2x7B with our method surpasses the performance of state-of-the-art language models with more than 7B and 13B parameters.
  arXiv  Detail & Related papers  (2024-06-27T11:48:25Z)
• DataComp-LM: In search of the next generation of training sets for language models [200.5293181577585]
  DataComp for Language Models (DCLM) is a testbed for controlled dataset experiments with the goal of improving language models.<n>We provide a standardized corpus of 240T tokens extracted from Common Crawl, effective pretraining recipes based on the OpenLM framework, and a broad suite of 53 downstream evaluations.<n>Participants in the DCLM benchmark can experiment with data curation strategies such as deduplication, filtering, and data mixing at model scales ranging from 412M to 7B parameters.
  arXiv  Detail & Related papers  (2024-06-17T17:42:57Z)
• Smaller Language Models are capable of selecting Instruction-Tuning Training Data for Larger Language Models [39.65879784788677]
  We introduce a novel training data selection based on the learning percentage of the samples. We assert that current language models possess the capability to autonomously select high-quality training data. Our paper introduces a novel approach to training data selection, showcasing a more efficient alternative.
  arXiv  Detail & Related papers  (2024-02-16T03:39:37Z)
• Skill over Scale: The Case for Medium, Domain-Specific Models for SE [4.2630881518611226]
  We show that modestly sized domain-specific models can outperform much larger ones on code labeling tasks.<n>We train two models: SOBertBase (125M parameters) and SOBertLarge (762M parameters) at a budget of just $374 and $1600 each.<n>Results demonstrate that pre-training both extensively and properly on in-domain data can yield a powerful and affordable alternative to leveraging closed-source general-purpose models.
  arXiv  Detail & Related papers  (2023-06-05T21:38:30Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.