GATE: General Arabic Text Embedding for Enhanced Semantic Textual Similarity with Matryoshka Representation Learning and Hybrid Loss Training

• URL: http://arxiv.org/abs/2505.24581v1
• Date: Fri, 30 May 2025 13:29:03 GMT
• Title: GATE: General Arabic Text Embedding for Enhanced Semantic Textual Similarity with Matryoshka Representation Learning and Hybrid Loss Training
• Authors: Omer Nacar, Anis Koubaa, Serry Sibaee, Yasser Al-Habashi, Adel Ammar, Wadii Boulila,
• Abstract summary: General Arabic Text Embedding (GATE) models achieve state-of-the-art performance on the Semantic Textual Similarity task within the MTEB benchmark.<n>Gate outperforms larger models, including OpenAI, with a 20-25% performance improvement on STS benchmarks.
• Score: 1.4231093967875448
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Semantic textual similarity (STS) is a critical task in natural language processing (NLP), enabling applications in retrieval, clustering, and understanding semantic relationships between texts. However, research in this area for the Arabic language remains limited due to the lack of high-quality datasets and pre-trained models. This scarcity of resources has restricted the accurate evaluation and advance of semantic similarity in Arabic text. This paper introduces General Arabic Text Embedding (GATE) models that achieve state-of-the-art performance on the Semantic Textual Similarity task within the MTEB benchmark. GATE leverages Matryoshka Representation Learning and a hybrid loss training approach with Arabic triplet datasets for Natural Language Inference, which are essential for enhancing model performance in tasks that demand fine-grained semantic understanding. GATE outperforms larger models, including OpenAI, with a 20-25% performance improvement on STS benchmarks, effectively capturing the unique semantic nuances of Arabic.

Related papers

• Resource-Efficient Adaptation of Large Language Models for Text Embeddings via Prompt Engineering and Contrastive Fine-tuning [6.549601823162279]
  Large Language Models (LLMs) have become a cornerstone in Natural Language Processing (NLP)<n>We explore several adaptation strategies for pre-trained, decoder-only LLMs.
  arXiv  Detail & Related papers  (2025-07-30T14:49:30Z)
• CrosGrpsABS: Cross-Attention over Syntactic and Semantic Graphs for Aspect-Based Sentiment Analysis in a Low-Resource Language [0.5937476291232802]
  Aspect-Based Sentiment Analysis (ABSA) is a fundamental task in natural language processing, offering fine-grained insights into opinions expressed in text.<n>This research propose CrosGrpsABS, a novel hybrid framework that leverages bidirectional cross-attention between syntactic and semantic graphs to enhance aspect-level sentiment classification.<n>We evaluate CrosGrpsABS on four low-resource Bengali ABSA datasets and the high-resource English SemEval 2014 Task 4 dataset.
  arXiv  Detail & Related papers  (2025-05-25T07:42:32Z)
• Understanding In-Context Machine Translation for Low-Resource Languages: A Case Study on Manchu [53.437954702561065]
  In-context machine translation (MT) with large language models (LLMs) is a promising approach for low-resource MT.<n>This study systematically investigates how each type of resource, e.g., dictionary, grammar book, and retrieved parallel examples, affect the translation performance.<n>Our results indicate that high-quality dictionaries and good parallel examples are very helpful, while grammars hardly help.
  arXiv  Detail & Related papers  (2025-02-17T14:53:49Z)
• Exploring Retrieval Augmented Generation in Arabic [0.0]
  Retrieval Augmented Generation (RAG) has emerged as a powerful technique in natural language processing. This paper presents a case study on the implementation and evaluation of RAG for Arabic text.
  arXiv  Detail & Related papers  (2024-08-14T10:03:28Z)
• Enhancing Semantic Similarity Understanding in Arabic NLP with Nested Embedding Learning [0.6752538702870792]
  This work presents a novel framework for training Arabic nested embedding models through Matryoshka Embedding Learning. Our innovative contribution includes the translation of various sentence similarity datasets into Arabic. We trained several embedding models on the Arabic Natural Language Inference triplet dataset and assessed their performance.
  arXiv  Detail & Related papers  (2024-07-30T19:03:03Z)
• CELA: Cost-Efficient Language Model Alignment for CTR Prediction [70.65910069412944]
  Click-Through Rate (CTR) prediction holds a paramount position in recommender systems.<n>Recent efforts have sought to mitigate these challenges by integrating Pre-trained Language Models (PLMs)<n>We propose textbfCost-textbfEfficient textbfLanguage Model textbfAlignment (textbfCELA) for CTR prediction.
  arXiv  Detail & Related papers  (2024-05-17T07:43:25Z)
• How Well Do Text Embedding Models Understand Syntax? [50.440590035493074]
  The ability of text embedding models to generalize across a wide range of syntactic contexts remains under-explored. Our findings reveal that existing text embedding models have not sufficiently addressed these syntactic understanding challenges. We propose strategies to augment the generalization ability of text embedding models in diverse syntactic scenarios.
  arXiv  Detail & Related papers  (2023-11-14T08:51:00Z)
• Beyond Contrastive Learning: A Variational Generative Model for Multilingual Retrieval [109.62363167257664]
  We propose a generative model for learning multilingual text embeddings. Our model operates on parallel data in $N$ languages. We evaluate this method on a suite of tasks including semantic similarity, bitext mining, and cross-lingual question retrieval.
  arXiv  Detail & Related papers  (2022-12-21T02:41:40Z)
• Pre-training Language Model Incorporating Domain-specific Heterogeneous Knowledge into A Unified Representation [49.89831914386982]
  We propose a unified pre-trained language model (PLM) for all forms of text, including unstructured text, semi-structured text, and well-structured text. Our approach outperforms the pre-training of plain text using only 1/4 of the data.
  arXiv  Detail & Related papers  (2021-09-02T16:05:24Z)
• Arabic aspect based sentiment analysis using BERT [0.0]
  This article explores the modeling capabilities of contextual embeddings from pre-trained language models, such as BERT. We are building a simple but effective BERT-based neural baseline to handle this task. Our BERT architecture with a simple linear classification layer surpassed the state-of-the-art works, according to the experimental results.
  arXiv  Detail & Related papers  (2021-07-28T11:34:00Z)
• ERICA: Improving Entity and Relation Understanding for Pre-trained Language Models via Contrastive Learning [97.10875695679499]
  We propose a novel contrastive learning framework named ERICA in pre-training phase to obtain a deeper understanding of the entities and their relations in text. Experimental results demonstrate that our proposed ERICA framework achieves consistent improvements on several document-level language understanding tasks.
  arXiv  Detail & Related papers  (2020-12-30T03:35:22Z)
• Syntax-Enhanced Pre-trained Model [49.1659635460369]
  We study the problem of leveraging the syntactic structure of text to enhance pre-trained models such as BERT and RoBERTa. Existing methods utilize syntax of text either in the pre-training stage or in the fine-tuning stage, so that they suffer from discrepancy between the two stages. We present a model that utilizes the syntax of text in both pre-training and fine-tuning stages.
  arXiv  Detail & Related papers  (2020-12-28T06:48:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.