Challenges in Adapting Multilingual LLMs to Low-Resource Languages using LoRA PEFT Tuning

• URL: http://arxiv.org/abs/2411.18571v1
• Date: Wed, 27 Nov 2024 18:14:38 GMT
• Title: Challenges in Adapting Multilingual LLMs to Low-Resource Languages using LoRA PEFT Tuning
• Authors: Omkar Khade, Shruti Jagdale, Abhishek Phaltankar, Gauri Takalikar, Raviraj Joshi,
• Abstract summary: This study investigates the effects of Low-Rank Adaptation (LoRA) -Efficient Fine-Tuning (PEFT) on multilingual Gemma models for Marathi. Using a translated dataset with 52,000 instruction-response pairs, our findings reveal that while evaluation performance decline post-fine-tuning, manual assessments frequently suggest that the fine-tuned models outperform their original counterparts.
• Score: 0.4194295877935868
• License:
• Abstract: Large Language Models (LLMs) have demonstrated remarkable multilingual capabilities, yet challenges persist in adapting these models for low-resource languages. In this study, we investigate the effects of Low-Rank Adaptation (LoRA) Parameter-Efficient Fine-Tuning (PEFT) on multilingual Gemma models for Marathi, a language with limited resources. Using a translated Alpaca dataset with 52,000 instruction-response pairs, our findings reveal that while evaluation metrics often show a performance decline post-fine-tuning, manual assessments frequently suggest that the fine-tuned models outperform their original counterparts. The observations indicate improvements in target language generation capabilities but a reduction in reasoning abilities following language adaptation. These results underscore the need for improved evaluation methodologies and the creation of high-quality native datasets to accurately assess language-specific model performance in low-resource settings.

Related papers

• Evaluating Standard and Dialectal Frisian ASR: Multilingual Fine-tuning and Language Identification for Improved Low-resource Performance [9.624005980086707]
  State-of-the-art methods deploy self-supervised transfer learning where a model pre-trained on large amounts of data is fine-tuned using little labeled data. We show that Frisian ASR performance can be improved by using multilingual fine-tuning data and an auxiliary language identification task.
  arXiv  Detail & Related papers  (2025-02-07T12:42:46Z)
• Enhancing Code Generation for Low-Resource Languages: No Silver Bullet [55.39571645315926]
  Large Language Models (LLMs) rely on large and diverse datasets to learn syntax, semantics, and usage patterns of programming languages. For low-resource languages, the limited availability of such data hampers the models' ability to generalize effectively. We present an empirical study investigating the effectiveness of several approaches for boosting LLMs' performance on low-resource languages.
  arXiv  Detail & Related papers  (2025-01-31T12:23:28Z)
• QueEn: A Large Language Model for Quechua-English Translation [20.377876059048692]
  We propose QueEn, a novel approach for Quechua-English translation that combines Retrieval-Augmented Generation (RAG) with parameter-efficient fine-tuning techniques. Our approach substantially exceeds baseline models, with a BLEU score of 17.6 compared to 1.5 for standard GPT models.
  arXiv  Detail & Related papers  (2024-12-06T17:04:21Z)
• How to Learn a New Language? An Efficient Solution for Self-Supervised Learning Models Unseen Languages Adaption in Low-Resource Scenario [72.02391485962127]
  Speech Self-Supervised Learning (SSL) models achieve impressive performance on Automatic Speech Recognition (ASR) In low-resource language ASR, they encounter the domain mismatch problem between pre-trained and low-resource languages. We extend a conventional efficient fine-tuning scheme based on the adapter to handle these issues.
  arXiv  Detail & Related papers  (2024-11-27T10:51:00Z)
• On Limitations of LLM as Annotator for Low Resource Languages [0.4194295877935868]
  Low-resource languages face significant challenges due to the lack of sufficient linguistic data, resources, and tools for tasks such as supervised learning, annotation, and classification. This shortage hinders the development of accurate models and datasets, making it difficult to perform critical NLP tasks like sentiment analysis or hate speech detection. To bridge this gap, Large Language Models (LLMs) present an opportunity for potential annotators, capable of generating datasets and resources for these underrepresented languages.
  arXiv  Detail & Related papers  (2024-11-26T17:55:37Z)
• Unlocking the Potential of Model Merging for Low-Resource Languages [66.7716891808697]
  Adapting large language models to new languages typically involves continual pre-training (CT) followed by supervised fine-tuning (SFT) We propose model merging as an alternative for low-resource languages, combining models with distinct capabilities into a single model without additional training. Experiments based on Llama-2-7B demonstrate that model merging effectively endows LLMs for low-resource languages with task-solving abilities, outperforming CT-then-SFT in scenarios with extremely scarce data.
  arXiv  Detail & Related papers  (2024-07-04T15:14:17Z)
• Targeted Multilingual Adaptation for Low-resource Language Families [17.212424929235624]
  We study best practices for adapting a pre-trained model to a language family. Our adapted models significantly outperform mono- and multilingual baselines. Low-resource languages can be aggressively up-sampled during training at little detriment to performance in high-resource languages.
  arXiv  Detail & Related papers  (2024-05-20T23:38:06Z)
• Bridging the Bosphorus: Advancing Turkish Large Language Models through Strategies for Low-Resource Language Adaptation and Benchmarking [1.3716808114696444]
  Large Language Models (LLMs) are becoming crucial across various fields, emphasizing the urgency for high-quality models in underrepresented languages. This study explores the unique challenges faced by low-resource languages, such as data scarcity, model selection, evaluation, and computational limitations.
  arXiv  Detail & Related papers  (2024-05-07T21:58:45Z)
• Democratizing LLMs for Low-Resource Languages by Leveraging their English Dominant Abilities with Linguistically-Diverse Prompts [75.33019401706188]
  Large language models (LLMs) are known to effectively perform tasks by simply observing few exemplars. We propose to assemble synthetic exemplars from a diverse set of high-resource languages to prompt the LLMs to translate from any language into English. Our unsupervised prompting method performs on par with supervised few-shot learning in LLMs of different sizes for translations between English and 13 Indic and 21 African low-resource languages.
  arXiv  Detail & Related papers  (2023-06-20T08:27:47Z)
• mFACE: Multilingual Summarization with Factual Consistency Evaluation [79.60172087719356]
  Abstractive summarization has enjoyed renewed interest in recent years, thanks to pre-trained language models and the availability of large-scale datasets. Despite promising results, current models still suffer from generating factually inconsistent summaries. We leverage factual consistency evaluation models to improve multilingual summarization.
  arXiv  Detail & Related papers  (2022-12-20T19:52:41Z)
• Language Models are Few-shot Multilingual Learners [66.11011385895195]
  We evaluate the multilingual skills of the GPT and T5 models in conducting multi-class classification on non-English languages. We show that, given a few English examples as context, pre-trained language models can predict not only English test samples but also non-English ones.
  arXiv  Detail & Related papers  (2021-09-16T03:08:22Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.