On Bilingual Lexicon Induction with Large Language Models

• URL: http://arxiv.org/abs/2310.13995v2
• Date: Sun, 25 Feb 2024 22:34:50 GMT
• Title: On Bilingual Lexicon Induction with Large Language Models
• Authors: Yaoyiran Li, Anna Korhonen, Ivan Vuli\'c
• Abstract summary: We examine the potential of the latest generation of Large Language Models for the development of bilingual lexicons. We study 1) zero-shot prompting for unsupervised BLI and 2) few-shot in-context prompting with a set of seed translation pairs. Our work is the first to demonstrate strong BLI capabilities of text-to-text mLLMs.
• Score: 81.6546357879259
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Bilingual Lexicon Induction (BLI) is a core task in multilingual NLP that still, to a large extent, relies on calculating cross-lingual word representations. Inspired by the global paradigm shift in NLP towards Large Language Models (LLMs), we examine the potential of the latest generation of LLMs for the development of bilingual lexicons. We ask the following research question: Is it possible to prompt and fine-tune multilingual LLMs (mLLMs) for BLI, and how does this approach compare against and complement current BLI approaches? To this end, we systematically study 1) zero-shot prompting for unsupervised BLI and 2) few-shot in-context prompting with a set of seed translation pairs, both without any LLM fine-tuning, as well as 3) standard BLI-oriented fine-tuning of smaller LLMs. We experiment with 18 open-source text-to-text mLLMs of different sizes (from 0.3B to 13B parameters) on two standard BLI benchmarks covering a range of typologically diverse languages. Our work is the first to demonstrate strong BLI capabilities of text-to-text mLLMs. The results reveal that few-shot prompting with in-context examples from nearest neighbours achieves the best performance, establishing new state-of-the-art BLI scores for many language pairs. We also conduct a series of in-depth analyses and ablation studies, providing more insights on BLI with (m)LLMs, also along with their limitations.

Related papers

• Think Carefully and Check Again! Meta-Generation Unlocking LLMs for Low-Resource Cross-Lingual Summarization [108.6908427615402]
  Cross-lingual summarization ( CLS) aims to generate a summary for the source text in a different target language. Currently, instruction-tuned large language models (LLMs) excel at various English tasks. Recent studies have shown that LLMs' performance on CLS tasks remains unsatisfactory even with few-shot settings.
  arXiv  Detail & Related papers  (2024-10-26T00:39:44Z)
• Self-Augmented In-Context Learning for Unsupervised Word Translation [23.495503962839337]
  Large language models (LLMs) demonstrate strong word translation or bilingual lexicon induction (BLI) capabilities in few-shot setups. We propose self-augmented in-context learning (SAIL) for unsupervised BLI. Our method shows substantial gains over zero-shot prompting of LLMs on two established BLI benchmarks.
  arXiv  Detail & Related papers  (2024-02-15T15:43:05Z)
• Native Language Identification with Large Language Models [60.80452362519818]
  We show that GPT models are proficient at NLI classification, with GPT-4 setting a new performance record of 91.7% on the benchmark11 test set in a zero-shot setting. We also show that unlike previous fully-supervised settings, LLMs can perform NLI without being limited to a set of known classes.
  arXiv  Detail & Related papers  (2023-12-13T00:52:15Z)
• ProMap: Effective Bilingual Lexicon Induction via Language Model Prompting [22.743097175747575]
  We introduce ProMap, a novel approach for bilingual induction (BLI) ProMap relies on an effective padded prompting of language models with a seed dictionary that achieves good performance when used independently. When evaluated on both rich-resource and low-resource languages, ProMap consistently achieves state-of-the-art results.
  arXiv  Detail & Related papers  (2023-10-28T18:33:24Z)
• When your Cousin has the Right Connections: Unsupervised Bilingual Lexicon Induction for Related Data-Imbalanced Languages [29.346191691508125]
  Unsupervised bilingual lexicon induction is most likely to be useful for low-resource languages, where large datasets are not available. We show that state-of-the-art BLI methods in the literature exhibit near-zero performance for severely data-imbalanced language pairs. We present a new method for unsupervised BLI between a related LRL and HRL that only requires inference on a masked language model of the HRL.
  arXiv  Detail & Related papers  (2023-05-23T12:49:21Z)
• Multilingual Machine Translation with Large Language Models: Empirical Results and Analysis [103.89753784762445]
  Large language models (LLMs) have demonstrated remarkable potential in handling multilingual machine translation (MMT) This paper systematically investigates the advantages and challenges of LLMs for MMT. We thoroughly evaluate eight popular LLMs, including ChatGPT and GPT-4.
  arXiv  Detail & Related papers  (2023-04-10T15:51:30Z)
• Improving Bilingual Lexicon Induction with Cross-Encoder Reranking [31.142790337451366]
  We propose a novel semi-supervised post-hoc reranking method termed BLICEr (BLI with Cross-Encoder Reranking) The key idea is to 'extract' cross-lingual lexical knowledge from mPLMs, and then combine it with the original CLWEs. BLICEr establishes new results on two standard BLI benchmarks spanning a wide spectrum of diverse languages.
  arXiv  Detail & Related papers  (2022-10-30T21:26:07Z)
• Exposing Cross-Lingual Lexical Knowledge from Multilingual Sentence Encoders [85.80950708769923]
  We probe multilingual language models for the amount of cross-lingual lexical knowledge stored in their parameters, and compare them against the original multilingual LMs. We also devise a novel method to expose this knowledge by additionally fine-tuning multilingual models. We report substantial gains on standard benchmarks.
  arXiv  Detail & Related papers  (2022-04-30T13:23:16Z)
• Improving Word Translation via Two-Stage Contrastive Learning [46.71404992627519]
  We propose a robust and effective two-stage contrastive learning framework for the BLI task. Comprehensive experiments on standard BLI datasets for diverse languages show substantial gains achieved by our framework.
  arXiv  Detail & Related papers  (2022-03-15T22:51:22Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.