Adapting Multilingual Embedding Models to Historical Luxembourgish

• URL: http://arxiv.org/abs/2502.07938v2
• Date: Wed, 19 Feb 2025 10:38:40 GMT
• Title: Adapting Multilingual Embedding Models to Historical Luxembourgish
• Authors: Andrianos Michail, Corina Julia Raclé, Juri Opitz, Simon Clematide,
• Abstract summary: Pre-trained multilingual models, typically evaluated on contemporary texts, face challenges with historical digitized content due to OCR noise and outdated spellings. We explore the use of multilingual embeddings for cross-lingual semantic search on historical Luxembourgish.
• Score: 5.474797258314828
• License:
• Abstract: The growing volume of digitized historical texts requires effective semantic search using text embeddings. However, pre-trained multilingual models, typically evaluated on contemporary texts, face challenges with historical digitized content due to OCR noise and outdated spellings. We explore the use of multilingual embeddings for cross-lingual semantic search on historical Luxembourgish, a low-resource language. We collect historical Luxembourgish news articles spanning various time periods and use GPT-4o to segment and translate them into closely related languages, creating 20,000 parallel training sentences per language pair. We further create a historical bitext mining evaluation set and find that these models struggle to perform cross-lingual search on historical Luxembourgish. To address this, we propose a simple adaptation method using in-domain training data, achieving up to 98\% accuracy in cross-lingual evaluations. We release our adapted models and historical Luxembourgish-German/French bitexts to support further research.

Related papers

• Text Generation Models for Luxembourgish with Limited Data: A Balanced Multilingual Strategy [7.59001382786429]
  This paper addresses the challenges in developing language models for less-represented languages, with a focus on Luxembourgish. We propose a novel text generation model based on the T5 architecture, combining limited Luxembourgish data with equal amounts of German and French data. For the evaluation, we introduce LuxGen, a text generation benchmark that is the first of its kind for Luxembourgish.
  arXiv  Detail & Related papers  (2024-12-12T16:23:12Z)
• MENTOR: Multilingual tExt detectioN TOward leaRning by analogy [59.37382045577384]
  We propose a framework to detect and identify both seen and unseen language regions inside scene images. "MENTOR" is the first work to realize a learning strategy between zero-shot learning and few-shot learning for multilingual scene text detection.
  arXiv  Detail & Related papers  (2024-03-12T03:35:17Z)
• Sentence Embedding Models for Ancient Greek Using Multilingual Knowledge Distillation [0.0]
  We use a multilingual knowledge distillation approach to train BERT models to produce sentence embeddings for Ancient Greek text. We build a parallel sentence dataset using a sentence-embedding alignment method to align Ancient Greek documents with English translations. We evaluate our models on translation search, semantic similarity, and semantic retrieval tasks.
  arXiv  Detail & Related papers  (2023-08-24T23:38:44Z)
• hmBERT: Historical Multilingual Language Models for Named Entity Recognition [0.6226609932118123]
  We tackle NER for identifying persons, locations, and organizations in historical texts. In this work, we tackle NER for historical German, English, French, Swedish, and Finnish by training large historical language models.
  arXiv  Detail & Related papers  (2022-05-31T07:30:33Z)
• Models and Datasets for Cross-Lingual Summarisation [78.56238251185214]
  We present a cross-lingual summarisation corpus with long documents in a source language associated with multi-sentence summaries in a target language. The corpus covers twelve language pairs and directions for four European languages, namely Czech, English, French and German. We derive cross-lingual document-summary instances from Wikipedia by combining lead paragraphs and articles' bodies from language aligned Wikipedia titles.
  arXiv  Detail & Related papers  (2022-02-19T11:55:40Z)
• From FreEM to D'AlemBERT: a Large Corpus and a Language Model for Early Modern French [57.886210204774834]
  We present our efforts to develop NLP tools for Early Modern French (historical French from the 16$textth$ to the 18$textth$ centuries). We present the $textFreEM_textmax$ corpus of Early Modern French and D'AlemBERT, a RoBERTa-based language model trained on $textFreEM_textmax$.
  arXiv  Detail & Related papers  (2022-02-18T22:17:22Z)
• Summarising Historical Text in Modern Languages [13.886432536330805]
  We introduce the task of historical text summarisation, where documents in historical forms of a language are summarised in the corresponding modern language. This is a fundamentally important routine to historians and digital humanities researchers but has never been automated. We compile a high-quality gold-standard text summarisation dataset, which consists of historical German and Chinese news from hundreds of years ago summarised in modern German or Chinese.
  arXiv  Detail & Related papers  (2021-01-26T13:00:07Z)
• Unsupervised Domain Adaptation of a Pretrained Cross-Lingual Language Model [58.27176041092891]
  Recent research indicates that pretraining cross-lingual language models on large-scale unlabeled texts yields significant performance improvements. We propose a novel unsupervised feature decomposition method that can automatically extract domain-specific features from the entangled pretrained cross-lingual representations. Our proposed model leverages mutual information estimation to decompose the representations computed by a cross-lingual model into domain-invariant and domain-specific parts.
  arXiv  Detail & Related papers  (2020-11-23T16:00:42Z)
• Learning Contextualised Cross-lingual Word Embeddings and Alignments for Extremely Low-Resource Languages Using Parallel Corpora [63.5286019659504]
  We propose a new approach for learning contextualised cross-lingual word embeddings based on a small parallel corpus. Our method obtains word embeddings via an LSTM encoder-decoder model that simultaneously translates and reconstructs an input sentence.
  arXiv  Detail & Related papers  (2020-10-27T22:24:01Z)
• XCOPA: A Multilingual Dataset for Causal Commonsense Reasoning [68.57658225995966]
  Cross-lingual Choice of Plausible Alternatives (XCOPA) is a typologically diverse multilingual dataset for causal commonsense reasoning in 11 languages. We evaluate a range of state-of-the-art models on this novel dataset, revealing that the performance of current methods falls short compared to translation-based transfer.
  arXiv  Detail & Related papers  (2020-05-01T12:22:33Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.