Linq-Embed-Mistral Technical Report

• URL: http://arxiv.org/abs/2412.03223v1
• Date: Wed, 04 Dec 2024 11:18:32 GMT
• Title: Linq-Embed-Mistral Technical Report
• Abstract summary: Linq-Embed-Mistral excels in the MTEB benchmarks (as of May 29, 2024)<n>Linq-Embed-Mistral ranks 1st among all models for retrieval tasks on the MTEB leaderboard with a performance score of 60.2.
• Score: 11.77727725268191
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: This report explores the enhancement of text retrieval performance using advanced data refinement techniques. We develop Linq-Embed-Mistral\footnote{\url{https://huggingface.co/Linq-AI-Research/Linq-Embed-Mistral}} by building on the E5-mistral and Mistral-7B-v0.1 models, focusing on sophisticated data crafting, data filtering, and negative mining methods, which are highly tailored to each task, applied to both existing benchmark dataset and highly tailored synthetic dataset generated via large language models (LLMs). Linq-Embed-Mistral excels in the MTEB benchmarks (as of May 29, 2024), achieving an average score of 68.2 across 56 datasets, and ranks 1st among all models for retrieval tasks on the MTEB leaderboard with a performance score of 60.2. This performance underscores its superior capability in enhancing search precision and reliability. Our contributions include advanced data refinement methods that significantly improve model performance on benchmark and synthetic datasets, techniques for homogeneous task ordering and mixed task fine-tuning to enhance model generalization and stability, and a streamlined evaluation process using 4-bit precision and a light retrieval evaluation set, which accelerates validation without sacrificing accuracy.

Related papers

• MLLM-DataEngine: Closing the Loop of Multimodal Instruction Tuning Data Generation [51.15583711678273]
  We propose a novel closed-loop system that bridges data generation, model training, and evaluation.<n>Within each loop iteration, the MLLM-DataEngine first analyzes the weakness of the model based on the evaluation results.<n>We find MLLM-DataEngine could boost the MLLMs capability in a targeted and automatic manner without human participants.
  arXiv  Detail & Related papers  (2026-07-07T08:35:38Z)
• Synthetic Industrial Object Detection: GenAI vs. Feature-Based Methods [5.278929538141005]
  We benchmark a range of domain randomization (DR) and domain adaptation (DA) techniques, including feature-based methods, generative AI (GenAI) and classical rendering approaches.<n>Our evaluation focuses on the effectiveness and efficiency of low-level and high-level feature alignment, as well as a controlled diffusion-based DA method guided by prompts generated from real-world contexts.<n>Results show that if render-based data with enough variability is available as seed, simpler feature-based methods, such as brightness-based and perceptual hashing filtering, outperform more complex GenAI-based approaches in both accuracy and resource efficiency
  arXiv  Detail & Related papers  (2025-11-28T14:51:08Z)
• FADRM: Fast and Accurate Data Residual Matching for Dataset Distillation [21.910537847630067]
  Residual connection has been extensively studied and widely applied at the model architecture level.<n>We introduce the concept of Data Residual Matching for the first time, leveraging data-level skip connections to facilitate data generation and mitigate data information vanishing.
  arXiv  Detail & Related papers  (2025-06-30T17:59:34Z)
• T-SHIRT: Token-Selective Hierarchical Data Selection for Instruction Tuning [5.963754140027611]
  Token-Selective HIeRarchical Data Selection for Instruction Tuning (T-SHIRT) is a novel data selection framework.<n>We demonstrate that models instruction-tuned on a curated dataset can outperform those trained on the entire large-scale dataset.
  arXiv  Detail & Related papers  (2025-06-02T04:59:17Z)
• MOLE: Metadata Extraction and Validation in Scientific Papers Using LLMs [54.5729817345543]
  MOLE is a framework that automatically extracts metadata attributes from scientific papers covering datasets of languages other than Arabic.<n>Our methodology processes entire documents across multiple input formats and incorporates robust validation mechanisms for consistent output.
  arXiv  Detail & Related papers  (2025-05-26T10:31:26Z)
• MLLM-Selector: Necessity and Diversity-driven High-Value Data Selection for Enhanced Visual Instruction Tuning [69.7347209018861]
  We introduce MLLM-Selector, an automated approach that identifies valuable data for visual instruction tuning. We calculate necessity scores for each sample in the VIT data pool to identify samples pivotal for enhancing model performance. Our findings underscore the importance of mixing necessity and diversity in data choice, leading to the creation of MLLM-Selector.
  arXiv  Detail & Related papers  (2025-03-26T12:42:37Z)
• Star-Agents: Automatic Data Optimization with LLM Agents for Instruction Tuning [71.2981957820888]
  We propose a novel Star-Agents framework, which automates the enhancement of data quality across datasets. The framework initially generates diverse instruction data with multiple LLM agents through a bespoke sampling method. The generated data undergo a rigorous evaluation using a dual-model method that assesses both difficulty and quality.
  arXiv  Detail & Related papers  (2024-11-21T02:30:53Z)
• Crafting Efficient Fine-Tuning Strategies for Large Language Models [2.633490094119608]
  Fine-tuning large language models (LLMs) with as few as 200 samples can improve model accuracy from 70% to 88% in a product attribute extraction task. A bayesian hyperparameter optimization method, which evaluates models at 20% of total training time, correlates strongly with final model performance. This approach led to a 2% improvement in accuracy over baseline models when evaluated on an independent test set.
  arXiv  Detail & Related papers  (2024-07-18T21:36:00Z)
• NV-Embed: Improved Techniques for Training LLMs as Generalist Embedding Models [38.41524186248607]
  We introduce the NV-Embed model with a variety of architectural designs and training procedures. Our model has achieved a record-high score of 69.32, ranking No. 1 on the Massive Text Embedding Benchmark (MTEB) We open-source the model at: https://face.co/EIR/NV-Embed-v1.
  arXiv  Detail & Related papers  (2024-05-27T17:59:45Z)
• An Integrated Data Processing Framework for Pretraining Foundation Models [57.47845148721817]
  Researchers and practitioners often have to manually curate datasets from difference sources. We propose a data processing framework that integrates a Processing Module and an Analyzing Module. The proposed framework is easy to use and highly flexible.
  arXiv  Detail & Related papers  (2024-02-26T07:22:51Z)
• How to Train Data-Efficient LLMs [56.41105687693619]
  We study data-efficient approaches for pre-training language models (LLMs) We find that Ask-LLM and Density sampling are the best methods in their respective categories. In our comparison of 19 samplers, involving hundreds of evaluation tasks and pre-training runs, we find that Ask-LLM and Density are the best methods in their respective categories.
  arXiv  Detail & Related papers  (2024-02-15T02:27:57Z)
• TarGEN: Targeted Data Generation with Large Language Models [51.87504111286201]
  TarGEN is a multi-step prompting strategy for generating high-quality synthetic datasets. We augment TarGEN with a method known as self-correction empowering LLMs to rectify inaccurately labeled instances. A comprehensive analysis of the synthetic dataset compared to the original dataset reveals similar or higher levels of dataset complexity and diversity.
  arXiv  Detail & Related papers  (2023-10-27T03:32:17Z)
• MLLM-DataEngine: An Iterative Refinement Approach for MLLM [62.30753425449056]
  We propose a novel closed-loop system that bridges data generation, model training, and evaluation. Within each loop, the MLLM-DataEngine first analyze the weakness of the model based on the evaluation results. For targeting, we propose an Adaptive Bad-case Sampling module, which adjusts the ratio of different types of data. For quality, we resort to GPT-4 to generate high-quality data with each given data type.
  arXiv  Detail & Related papers  (2023-08-25T01:41:04Z)
• Instruction Mining: Instruction Data Selection for Tuning Large Language Models [18.378654454336136]
  InstructMining is designed for automatically selecting premium instruction-following data for finetuning large language models. We show that InstructMining achieves state-of-the-art performance on two of the most popular benchmarks: LLM-as-a-judge and Huggingface OpenLLM leaderboard.
  arXiv  Detail & Related papers  (2023-07-12T16:37:31Z)
• Exploiting Asymmetry for Synthetic Training Data Generation: SynthIE and the Case of Information Extraction [28.51694365908817]
  This work shows that useful data can be synthetically generated even for tasks that cannot be solved directly by large language models. We synthetically generate a dataset of 1.8M data points, establish its superior quality compared to existing datasets in a human evaluation.
  arXiv  Detail & Related papers  (2023-03-07T18:48:55Z)
• GEMv2: Multilingual NLG Benchmarking in a Single Line of Code [161.1761414080574]
  Generation, Evaluation, and Metrics Benchmark introduces a modular infrastructure for dataset, model, and metric developers. GEMv2 supports 40 documented datasets in 51 languages. Models for all datasets can be evaluated online and our interactive data card creation and rendering tools make it easier to add new datasets to the living benchmark.
  arXiv  Detail & Related papers  (2022-06-22T17:52:30Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.