SlovKE: A Large-Scale Dataset and LLM Evaluation for Slovak Keyphrase Extraction

• URL: http://arxiv.org/abs/2603.15523v1
• Date: Mon, 16 Mar 2026 16:47:45 GMT
• Title: SlovKE: A Large-Scale Dataset and LLM Evaluation for Slovak Keyphrase Extraction
• Abstract summary: Keyphrase extraction for morphologically rich, low-resource languages remains understudied.<n>We construct a dataset of 227,432 scientific abstracts with author-assigned keyphrases.<n>KeyLLM produces keyphrases closer to the canonical forms assigned by authors.<n>Our analysis identifies morphological mismatch as the dominant failure mode for statistical methods.
• Score: 0.00954904463032233
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Keyphrase extraction for morphologically rich, low-resource languages remains understudied, largely due to the scarcity of suitable evaluation datasets. We address this gap for Slovak by constructing a dataset of 227,432 scientific abstracts with author-assigned keyphrases -- scraped and systematically cleaned from the Slovak Central Register of Theses -- representing a 25-fold increase over the largest prior Slovak resource and approaching the scale of established English benchmarks such as KP20K. Using this dataset, we benchmark three unsupervised baselines (YAKE, TextRank, KeyBERT with SlovakBERT embeddings) and evaluate KeyLLM, an LLM-based extraction method using GPT-3.5-turbo. Unsupervised baselines achieve at most 11.6\% exact-match $F1@6$, with a large gap to partial matching (up to 51.5\%), reflecting the difficulty of matching inflected surface forms to author-assigned keyphrases. KeyLLM narrows this exact--partial gap, producing keyphrases closer to the canonical forms assigned by authors, while manual evaluation on 100 documents ($κ= 0.61$) confirms that KeyLLM captures relevant concepts that automated exact matching underestimates. Our analysis identifies morphological mismatch as the dominant failure mode for statistical methods -- a finding relevant to other inflected languages. The dataset (https://huggingface.co/datasets/NaiveNeuron/SlovKE) and evaluation code (https://github.com/NaiveNeuron/SlovKE) are publicly available.

Related papers

• Check Your LLM's Secret Dictionary! Five Lines of Code Reveal What Your LLM Learned (Including What It Shouldn't Have) [0.0]
  We show that singular value decomposition of the lm_head weight matrix of a transformer-based large language model reveals interpretable semantic subspaces directly from the model weights.<n>Analysing GPT-OSS-120B, Gemma-2-2B, and Qwen2.5-1.5B, we find that singular value spectra and vocabulary cluster structures differ systematically across models.
  arXiv  Detail & Related papers  (2026-05-21T05:02:51Z)
• Bilingual Text-to-Motion Generation: A New Benchmark and Baselines [52.71312720094036]
  We introduce BiHumanML3D, the first bilingual text-to-motion benchmark, constructed via LLM-assisted annotation and rigorous manual correction.<n>We also propose a simple yet effective baseline, Bilingual Motion Diffusion synthesis (BiMD), featuring Cross-Lingual Alignment (CLA)<n>BiMD with CLA achieves an FID of 0.045 vs. 0.169 and R@3 of 82.8% vs. 80.8%, significantly outperforms monolingual diffusion models and translation baselines on BiHumanML3D.
  arXiv  Detail & Related papers  (2026-03-26T08:48:27Z)
• A Hybrid Protocol for Large-Scale Semantic Dataset Generation in Low-Resource Languages: The Turkish Semantic Relations Corpus [0.0]
  We present a hybrid methodology for generating large-scale semantic relationship datasets in low-resource languages.<n>Our approach integrates FastText embeddings with Agglomerative Clustering to identify semantic clusters, Gemini 2.5-Flash for automated semantic relationship classification, and integration with curated dictionary sources.<n>The resulting dataset comprises 843,000 unique Turkish semantic pairs representing a 10x scale increase over existing resources at minimal cost.
  arXiv  Detail & Related papers  (2026-01-19T17:38:52Z)
• COMI-LINGUA: Expert Annotated Large-Scale Dataset for Multitask NLP in Hindi-English Code-Mixing [1.2242530642524063]
  COMI-LINGUA is the largest manually annotated Hindi-English code-mixed dataset.<n>It comprises 125K+ high-quality instances across five core NLP tasks.<n>Each instance is annotated by three bilingual annotators, yielding over 376K expert annotations.
  arXiv  Detail & Related papers  (2025-03-27T16:36:39Z)
• MetaKP: On-Demand Keyphrase Generation [52.48698290354449]
  We introduce on-demand keyphrase generation, a novel paradigm that requires keyphrases that conform to specific high-level goals or intents. We present MetaKP, a large-scale benchmark comprising four datasets, 7500 documents, and 3760 goals across news and biomedical domains with human-annotated keyphrases. We demonstrate the potential of our method to serve as a general NLP infrastructure, exemplified by its application in epidemic event detection from social media.
  arXiv  Detail & Related papers  (2024-06-28T19:02:59Z)
• LexMatcher: Dictionary-centric Data Collection for LLM-based Machine Translation [67.24113079928668]
  We present LexMatcher, a method for data curation driven by the coverage of senses found in bilingual dictionaries. Our approach outperforms the established baselines on the WMT2022 test sets.
  arXiv  Detail & Related papers  (2024-06-03T15:30:36Z)
• Towards Realistic Zero-Shot Classification via Self Structural Semantic Alignment [53.2701026843921]
  Large-scale pre-trained Vision Language Models (VLMs) have proven effective for zero-shot classification. In this paper, we aim at a more challenging setting, Realistic Zero-Shot Classification, which assumes no annotation but instead a broad vocabulary. We propose the Self Structural Semantic Alignment (S3A) framework, which extracts structural semantic information from unlabeled data while simultaneously self-learning.
  arXiv  Detail & Related papers  (2023-08-24T17:56:46Z)
• CompoundPiece: Evaluating and Improving Decompounding Performance of Language Models [77.45934004406283]
  We systematically study decompounding, the task of splitting compound words into their constituents. We introduce a dataset of 255k compound and non-compound words across 56 diverse languages obtained from Wiktionary. We introduce a novel methodology to train dedicated models for decompounding.
  arXiv  Detail & Related papers  (2023-05-23T16:32:27Z)
• Visually Grounded Keyword Detection and Localisation for Low-Resource Languages [0.0]
  The study investigates the use of Visually Grounded Speech (VGS) models for keyword localisation in speech. Four methods for localisation are proposed and evaluated on an English dataset, with the best-performing method achieving an accuracy of 57%. A new dataset containing spoken captions in Yoruba language is also collected and released for cross-lingual keyword localisation.
  arXiv  Detail & Related papers  (2023-02-01T21:32:15Z)
• Ensemble Transfer Learning for Multilingual Coreference Resolution [60.409789753164944]
  A problem that frequently occurs when working with a non-English language is the scarcity of annotated training data. We design a simple but effective ensemble-based framework that combines various transfer learning techniques. We also propose a low-cost TL method that bootstraps coreference resolution models by utilizing Wikipedia anchor texts.
  arXiv  Detail & Related papers  (2023-01-22T18:22:55Z)
• Automatic Language Identification for Celtic Texts [0.0]
  This work addresses the identification of the related low-resource languages on the example of the Celtic language family. We collected a new dataset including Irish, Scottish, Welsh and English records. We tested supervised models such as SVM and neural networks with traditional statistical features alongside the output of clustering, autoencoder, and topic modelling methods.
  arXiv  Detail & Related papers  (2022-03-09T16:04:13Z)
• Clustering Word Embeddings with Self-Organizing Maps. Application on LaRoSeDa -- A Large Romanian Sentiment Data Set [15.877673959068455]
  Romanian is one of the understudied languages in computational linguistics. We introduce LaRoSeDa, a large Romanian sentiment data set. We replace the k-means clustering algorithm with self-organizing maps.
  arXiv  Detail & Related papers  (2021-01-11T21:19:22Z)
• MASKER: Masked Keyword Regularization for Reliable Text Classification [73.90326322794803]
  We propose a fine-tuning method, coined masked keyword regularization (MASKER), that facilitates context-based prediction. MASKER regularizes the model to reconstruct the keywords from the rest of the words and make low-confidence predictions without enough context. We demonstrate that MASKER improves OOD detection and cross-domain generalization without degrading classification accuracy.
  arXiv  Detail & Related papers  (2020-12-17T04:54:16Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.