Lemmatization as a Classification Task: Results from Arabic across Multiple Genres

• URL: http://arxiv.org/abs/2506.18399v1
• Date: Mon, 23 Jun 2025 08:34:33 GMT
• Title: Lemmatization as a Classification Task: Results from Arabic across Multiple Genres
• Abstract summary: Lemmatization is crucial for NLP tasks in morphologically rich languages with ambiguous orthography like Arabic.<n>This paper introduces two novel approaches that frame lemmatization as classification into a Lemma-POS-Gloss (LPG) tagset.<n>We also present a new Arabic lemmatization test set covering diverse genres, standardized alongside existing datasets.
• Score: 9.896789483253189
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Lemmatization is crucial for NLP tasks in morphologically rich languages with ambiguous orthography like Arabic, but existing tools face challenges due to inconsistent standards and limited genre coverage. This paper introduces two novel approaches that frame lemmatization as classification into a Lemma-POS-Gloss (LPG) tagset, leveraging machine translation and semantic clustering. We also present a new Arabic lemmatization test set covering diverse genres, standardized alongside existing datasets. We evaluate character level sequence-to-sequence models, which perform competitively and offer complementary value, but are limited to lemma prediction (not LPG) and prone to hallucinating implausible forms. Our results show that classification and clustering yield more robust, interpretable outputs, setting new benchmarks for Arabic lemmatization.

Related papers

• Arabic Sentence Segmentation Across Genres and Punctuation Conditions [14.737566818645663]
  We introduce AraSEG, a genre-diverse sentence segmentation corpus spanning eight genres and a wide range of punctuation and document structure conditions.<n>We evaluate LLMs, lightweight encoder models, and dependency-based models under increasingly challenging segmentation settings.<n>We show that accurate sentence segmentation substantially improves downstream dependency parsing.
  arXiv  Detail & Related papers  (2026-06-06T07:37:46Z)
• Arabic Morphosyntactic Tagging and Dependency Parsing with Large Language Models [13.143003615122245]
  Large language models (LLMs) perform strongly on many NLP tasks, but their ability to produce explicit linguistic structure remains unclear.<n>We evaluate instruction-tuned LLMs on two structured prediction tasks for Standard Arabic: morphosyntactic tagging and labeled dependency parsing.
  arXiv  Detail & Related papers  (2026-03-17T16:06:29Z)
• Lexicon-Enriched Graph Modeling for Arabic Document Readability Prediction [0.2333246928962179]
  We present a graph-based approach enriched with lexicons to predict document-level readability in Arabic.<n>Our system models each document as a sentence-level graph, where nodes represent sentences and lemmas.<n>For document-level prediction, sentence-level outputs are aggregated using max pooling to reflect the most difficult sentence.
  arXiv  Detail & Related papers  (2025-09-26T19:29:32Z)
• The Arabic Generality Score: Another Dimension of Modeling Arabic Dialectness [10.837144343838945]
  Arabic dialects form a diverse continuum, yet NLP models often treat them as discrete categories.<n>We propose a complementary measure: the Arabic Generality Score (AGS), which quantifies how widely a word is used across dialects.
  arXiv  Detail & Related papers  (2025-08-24T13:06:00Z)
• HeQ: a Large and Diverse Hebrew Reading Comprehension Benchmark [54.73504952691398]
  We set out to deliver a Hebrew Machine Reading dataset as extractive Questioning.<n>The morphologically rich nature of Hebrew poses a challenge to this endeavor.<n>We devise a novel set of guidelines, a controlled crowdsourcing protocol, and revised evaluation metrics.
  arXiv  Detail & Related papers  (2025-08-03T15:53:01Z)
• Vocabulary-free Fine-grained Visual Recognition via Enriched Contextually Grounded Vision-Language Model [52.01031460230826]
  Traditional approaches rely heavily on fixed vocabularies and closed-set classification paradigms.<n>Recent research has demonstrated that combining large language models with vision-language models (VLMs) makes open-set recognition possible.<n>We propose our training-free method, Enriched-FineR, which demonstrates state-of-the-art results in fine-grained visual recognition.
  arXiv  Detail & Related papers  (2025-07-30T20:06:01Z)
• Split Matching for Inductive Zero-shot Semantic Segmentation [52.90218623214213]
  Zero-shot Semantic (ZSS) aims to segment categories that are not annotated during training.<n>We propose Split Matching (SM), a novel assignment strategy that decouples Hungarian matching into two components.<n>SM is the first to introduce decoupled Hungarian matching under the inductive ZSS setting, and achieves state-of-the-art performance on two standard benchmarks.
  arXiv  Detail & Related papers  (2025-05-08T07:56:30Z)
• Refining Sentence Embedding Model through Ranking Sentences Generation with Large Language Models [60.00178316095646]
  Sentence embedding is essential for many NLP tasks, with contrastive learning methods achieving strong performance using datasets like NLI.<n>Recent studies leverage large language models (LLMs) to generate sentence pairs, reducing annotation dependency.<n>We propose a method for controlling the generation direction of LLMs in the latent space. Unlike unconstrained generation, the controlled approach ensures meaningful semantic divergence.<n> Experiments on multiple benchmarks demonstrate that our method achieves new SOTA performance with a modest cost in ranking sentence synthesis.
  arXiv  Detail & Related papers  (2025-02-19T12:07:53Z)
• Leveraging Transformer-Based Models for Predicting Inflection Classes of Words in an Endangered Sami Language [1.788784870849724]
  This paper presents a methodology for training a transformer-based model to classify lexical and morphosyntactic features of Skolt Sami. The motivation behind this work is to support language preservation and revitalization efforts for minority languages like Skolt Sami. Our model achieves an average weighted F1 score of 1.00 for POS classification and 0.81 for inflection class classification.
  arXiv  Detail & Related papers  (2024-11-04T19:41:16Z)
• Dual-level Adaptive Self-Labeling for Novel Class Discovery in Point Cloud Segmentation [15.000460515557211]
  We tackle the novel class discovery in point cloud segmentation, which discovers novel classes based on the semantic knowledge of seen classes. Existing work proposes an online point-wise clustering method with a simplified equal class-size constraint on the novel classes to avoid degenerate solutions. We propose a novel self-labeling strategy that adaptively generates high-quality pseudo-labels for imbalanced classes during model training.
  arXiv  Detail & Related papers  (2024-07-17T11:14:46Z)
• DIALECTBENCH: A NLP Benchmark for Dialects, Varieties, and Closely-Related Languages [49.38663048447942]
  We propose DIALECTBENCH, the first-ever large-scale benchmark for NLP on varieties. This allows for a comprehensive evaluation of NLP system performance on different language varieties. We provide substantial evidence of performance disparities between standard and non-standard language varieties.
  arXiv  Detail & Related papers  (2024-03-16T20:18:36Z)
• IDoFew: Intermediate Training Using Dual-Clustering in Language Models for Few Labels Text Classification [24.11420537250414]
  Bidirectional Representations from Transformers (BERT) have been very effective in various Natural Language Processing (NLP) and text mining tasks including text classification. Some tasks still pose challenges for these models, including text classification with limited labels. We have developed a novel two-stage intermediate clustering with subsequent fine-tuning that models the pseudo-labels reliably.
  arXiv  Detail & Related papers  (2024-01-08T17:07:37Z)
• A Multi-level Supervised Contrastive Learning Framework for Low-Resource Natural Language Inference [54.678516076366506]
  Natural Language Inference (NLI) is a growingly essential task in natural language understanding. Here we propose a multi-level supervised contrastive learning framework named MultiSCL for low-resource natural language inference.
  arXiv  Detail & Related papers  (2022-05-31T05:54:18Z)
• Comparative Error Analysis in Neural and Finite-state Models for Unsupervised Character-level Transduction [34.1177259741046]
  We compare the two model classes side by side and find that they tend to make different types of errors even when achieving comparable performance. We investigate how combining finite-state and sequence-to-sequence models at decoding time affects the output quantitatively and qualitatively.
  arXiv  Detail & Related papers  (2021-06-24T00:09:24Z)
• Detecting Fine-Grained Cross-Lingual Semantic Divergences without Supervision by Learning to Rank [28.910206570036593]
  This work improves the prediction and annotation of fine-grained semantic divergences. We introduce a training strategy for multilingual BERT models by learning to rank synthetic divergent examples of varying granularity. Learning to rank helps detect fine-grained sentence-level divergences more accurately than a strong sentence-level similarity model.
  arXiv  Detail & Related papers  (2020-10-07T21:26:20Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.