Forging GEMs: Advancing Greek NLP through Quality-Based Corpus Curation

• URL: http://arxiv.org/abs/2510.20002v2
• Date: Fri, 24 Oct 2025 11:58:25 GMT
• Title: Forging GEMs: Advancing Greek NLP through Quality-Based Corpus Curation
• Authors: Alexandra Apostolopoulou, Konstantinos Kanaris, Athanasios Koursaris, Dimitris Tsakalidis, George Domalis, Ioannis E. Livieris,
• Abstract summary: Greek Embedding Models (GEMs) developed to address limitations through architectural diversity and enhanced data curation.<n>Family of models trained on several large-scale, meticulously curated corpora.<n>First bilingual Greek-English embedding models tailored for cross-lingual legal applications.
• Score: 37.2909720705691
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: The advancement of natural language processing for morphologically rich and moderately-resourced languages like Modern Greek has been hindered by architectural stagnation, data scarcity, and limited context processing capabilities, particularly in specialized domains such as law. In this work, we propose the Greek Embedding Models (GEMs), a new family of transformer-based language models, specifically developed to address these limitations through architectural diversity and enhanced data curation. The proposed family of models are trained on several large-scale, meticulously curated corpora, encompassing both comprehensive general-domain datasets and specialized legal collections, addressing the persistent data scarcity that has impeded Greek language modeling advancement. The proposed quality-based corpus curation methodology incorporates extensive preprocessing pipelines, sophisticated deduplication strategies and targeted repetition of high-quality legal sub-corpora to enhance domain adaptation. The GEMs family comprises both established architectures (RoBERTa and Longformer) and advanced models not previously applied to Greek (ELECTRA, ConvBERT, and ModernBERT), providing comprehensive coverage of modern transformer designs. Additionally, we introduce the first bilingual Greek-English embedding models tailored for cross-lingual legal applications. Comprehensive evaluation across three core natural language understanding benchmarks demonstrates that the proposed GEM-RoBERTa and GEM-ConvBERT achieve statistically significant performance improvements over established state-of-the-art models, with accuracy gains of up to 3.6\% while conducted statistical analysis using Friedman Aligned-Ranks and Finner post-hoc tests confirms the superiority of our approach across multiple evaluation metrics.

Related papers

• Speed Always Wins: A Survey on Efficient Architectures for Large Language Models [51.817121227562964]
  Large Language Models (LLMs) have delivered impressive results in language understanding, generation, reasoning, and pushes the ability boundary of multimodal models.<n> Transformer models, as the foundation of modern LLMs, offer a strong baseline with excellent scaling properties.<n>The traditional transformer architecture requires substantial computations and poses significant obstacles for large-scale training and practical deployment.
  arXiv  Detail & Related papers  (2025-08-13T14:13:46Z)
• Aligning Knowledge Graphs and Language Models for Factual Accuracy [7.205708660952737]
  We introduce ALIGNed-LLM, a simple yet effective approach to improve language models' factuality.<n>We use embeddings from a pre-trained Knowledge Graph Embedding (KGE) model, such as TransE, and a trainable projection layer to align entity and text embeddings.
  arXiv  Detail & Related papers  (2025-07-17T08:15:50Z)
• A Survey of Model Architectures in Information Retrieval [59.61734783818073]
  The period from 2019 to the present has represented one of the biggest paradigm shifts in information retrieval (IR) and natural language processing (NLP)<n>We trace the development from traditional term-based methods to modern neural approaches, particularly highlighting the impact of transformer-based models and subsequent large language models (LLMs)<n>We conclude with a forward-looking discussion of emerging challenges and future directions.
  arXiv  Detail & Related papers  (2025-02-20T18:42:58Z)
• Advancing Retrieval-Augmented Generation for Persian: Development of Language Models, Comprehensive Benchmarks, and Best Practices for Optimization [0.0]
  The research aims to improve retrieval and generation accuracy by introducing Persian-specific models.<n>Three datasets-general knowledge(PQuad), scientifically specialized texts, and organizational reports- were used to assess these models.<n>MatinaSRoberta outperformed previous embeddings, achieving superior contextual relevance and retrieval accuracy across datasets.
  arXiv  Detail & Related papers  (2025-01-08T22:16:40Z)
• DSGram: Dynamic Weighting Sub-Metrics for Grammatical Error Correction in the Era of Large Language Models [39.493913608472404]
  Large language model (LLM)-based Grammatical Error Correction (GEC) models often produce corrections that diverge from provided gold references.<n>This discrepancy undermines the reliability of traditional reference-based evaluation metrics.<n>We propose a novel evaluation framework for GEC models, DSGram, integrating Semantic Coherence, Edit Level, and Fluency, and utilizing a dynamic weighting mechanism.
  arXiv  Detail & Related papers  (2024-12-17T11:54:16Z)
• Prompting Encoder Models for Zero-Shot Classification: A Cross-Domain Study in Italian [75.94354349994576]
  This paper explores the feasibility of employing smaller, domain-specific encoder LMs alongside prompting techniques to enhance performance in specialized contexts. Our study concentrates on the Italian bureaucratic and legal language, experimenting with both general-purpose and further pre-trained encoder-only models. The results indicate that while further pre-trained models may show diminished robustness in general knowledge, they exhibit superior adaptability for domain-specific tasks, even in a zero-shot setting.
  arXiv  Detail & Related papers  (2024-07-30T08:50:16Z)
• Contextualization Distillation from Large Language Model for Knowledge Graph Completion [51.126166442122546]
  We introduce the Contextualization Distillation strategy, a plug-in-and-play approach compatible with both discriminative and generative KGC frameworks. Our method begins by instructing large language models to transform compact, structural triplets into context-rich segments. Comprehensive evaluations across diverse datasets and KGC techniques highlight the efficacy and adaptability of our approach.
  arXiv  Detail & Related papers  (2024-01-28T08:56:49Z)
• GreekT5: A Series of Greek Sequence-to-Sequence Models for News Summarization [0.0]
  This paper proposes a series of novel TS models for Greek news articles. The proposed models were thoroughly evaluated on the same dataset against GreekBART. Our evaluation results reveal that most of the proposed models significantly outperform GreekBART on various evaluation metrics.
  arXiv  Detail & Related papers  (2023-11-13T21:33:12Z)
• Exploring Large Language Models for Classical Philology [17.856304057963776]
  We create four language models for Ancient Greek that vary along two dimensions to study their versatility for tasks of interest for Classical languages. We evaluate all models on morphological and syntactic tasks, including lemmatization. Results show that our models provide significant improvements over the SoTA.
  arXiv  Detail & Related papers  (2023-05-23T05:21:02Z)
• GreekBART: The First Pretrained Greek Sequence-to-Sequence Model [13.429669368275318]
  We introduce GreekBART, the first Seq2Seq model based on BART-base architecture and pretrained on a large-scale Greek corpus. We evaluate and compare GreekBART against BART-random, Greek-BERT, and XLM-R on a variety of discriminative tasks.
  arXiv  Detail & Related papers  (2023-04-03T10:48:51Z)
• Sparse*BERT: Sparse Models Generalize To New tasks and Domains [79.42527716035879]
  This paper studies how models pruned using Gradual Unstructured Magnitude Pruning can transfer between domains and tasks. We demonstrate that our general sparse model Sparse*BERT can become SparseBioBERT simply by pretraining the compressed architecture on unstructured biomedical text.
  arXiv  Detail & Related papers  (2022-05-25T02:51:12Z)
• A Unified Strategy for Multilingual Grammatical Error Correction with Pre-trained Cross-Lingual Language Model [100.67378875773495]
  We propose a generic and language-independent strategy for multilingual Grammatical Error Correction. Our approach creates diverse parallel GEC data without any language-specific operations. It achieves the state-of-the-art results on the NLPCC 2018 Task 2 dataset (Chinese) and obtains competitive performance on Falko-Merlin (German) and RULEC-GEC (Russian)
  arXiv  Detail & Related papers  (2022-01-26T02:10:32Z)
• Multi-granular Legal Topic Classification on Greek Legislation [4.09134848993518]
  We study the task of classifying legal texts written in the Greek language. This is the first time the task of Greek legal text classification is considered in an open research project.
  arXiv  Detail & Related papers  (2021-09-30T17:43:00Z)
• TextFlint: Unified Multilingual Robustness Evaluation Toolkit for Natural Language Processing [73.16475763422446]
  We propose a multilingual robustness evaluation platform for NLP tasks (TextFlint) It incorporates universal text transformation, task-specific transformation, adversarial attack, subpopulation, and their combinations to provide comprehensive robustness analysis. TextFlint generates complete analytical reports as well as targeted augmented data to address the shortcomings of the model's robustness.
  arXiv  Detail & Related papers  (2021-03-21T17:20:38Z)
• InfoBERT: Improving Robustness of Language Models from An Information Theoretic Perspective [84.78604733927887]
  Large-scale language models such as BERT have achieved state-of-the-art performance across a wide range of NLP tasks. Recent studies show that such BERT-based models are vulnerable facing the threats of textual adversarial attacks. We propose InfoBERT, a novel learning framework for robust fine-tuning of pre-trained language models.
  arXiv  Detail & Related papers  (2020-10-05T20:49:26Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.