Adapting Large Language Models to a Low-Resource Agglutinative Language: A Comparative Study of LoRA and QLoRA for Bashkir

• URL: http://arxiv.org/abs/2605.04948v1
• Date: Wed, 06 May 2026 14:14:35 GMT
• Title: Adapting Large Language Models to a Low-Resource Agglutinative Language: A Comparative Study of LoRA and QLoRA for Bashkir
• Abstract summary: This paper presents a comparative study of parameter-efficient fine-tuning methods, including LoRA and QLoRA, applied to the task of adapting large language models to the Bashkir language.<n> Experimental evaluation is conducted on a Bashkir text corpus of 71k documents (46.9M tokens) using models of various architectures.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: This paper presents a comparative study of parameter-efficient fine-tuning (PEFT) methods, including LoRA and QLoRA, applied to the task of adapting large language models to the Bashkir language, a low-resource agglutinative language of the Turkic family. Experimental evaluation is conducted on a Bashkir text corpus of 71k documents (46.9M tokens) using models of various architectures: DistilGPT2, GPT-2 (base, medium), Phi-2, Qwen2.5-7B, DeepSeek-7B, and Mistral-7B. To improve the reliability of results, each configuration was trained with three different random seeds. The lowest perplexity on the test set was obtained for GPT-2 medium with full fine-tuning (3.34). Meanwhile, QLoRA applied to Mistral-7B (3.79) and Phi-2 (3.81) achieved comparable quality with over 40 times fewer trainable parameters. However, we also observed cases of significant quality degradation when using PEFT for certain architectures (e.g., DeepSeek-7B with rank 8, perplexity = 129.55), indicating that the outcome depends critically on the choice of the base model and its tokenizer. Additionally, a qualitative analysis of generated texts based on Bashkir prompts revealed that models with the best perplexity do not necessarily produce the most coherent outputs: QLoRA-tuned models generated monolingual Bashkir continuations, whereas the fully fine-tuned model with the lowest perplexity frequently switched to English. The results suggest that QLoRA on 7B-scale models offers an effective compromise between quality and computational cost for Bashkir. To ensure reproducibility, open data, code, and trained adapters will be released upon acceptance.

Related papers

• Small Language Models as Judges for Rubric-Based Reinforcement Learning [57.707881387886516]
  We study whether smaller language models can serve as efficient and reliable rubric-based judges.<n>We compare three ways of extracting criterion-level judgments from small models: Generative verdicts, Yes/No Logprob margins, and Probe judges.<n>Across both datasets, the Qwen3-1.7B Probe judge achieves the strongest criterion-level agreement among these methods.
  arXiv  Detail & Related papers  (2026-08-30T20:00:17Z)
• Off-the-Shelf LLMs as Process Scorers: Training-Free Alternative to PRMs for Mathematical Reasoning [51.88950852117154]
  Chunk-Level Guided Generation is a training-free alternative that uses an off-the-shelf large language model as a process scorer.<n>We show that scoring variable-length reasoning steps with large-model likelihoods is unreliable due to a systematic length bias.<n>Chunk-Level Guided Generation produces substantially shorter reasoning traces than PRM guided search.
  arXiv  Detail & Related papers  (2026-06-01T04:43:36Z)
• Gemma 4, Phi-4, and Qwen3: Accuracy-Efficiency Tradeoffs in Dense and MoE Reasoning Language Models [6.396911723204044]
  Mixture-of-experts (MoE) language models are often expected to offer better quality-efficiency tradeoffs than dense models.<n>We present a benchmark of seven recent reasoning-oriented instruction-tuned models spanning dense and MoE designs.
  arXiv  Detail & Related papers  (2026-04-08T12:50:52Z)
• Benchmarking Linguistic Adaptation in Comparable-Sized LLMs: A Study of Llama-3.1-8B, Mistral-7B-v0.1, and Qwen3-8B on Romanized Nepali [0.0]
  Romanized Nepali, the Nepali language written in the Latin alphabet, is the dominant medium for informal digital communication in Nepal.<n>This study presents a systematic benchmarking of linguistic adaptation across three comparable-sized open-weight models.
  arXiv  Detail & Related papers  (2026-03-25T07:02:51Z)
• Efficient Fine-Tuning Methods for Portuguese Question Answering: A Comparative Study of PEFT on BERTimbau and Exploratory Evaluation of Generative LLMs [1.5049442691806052]
  This work presents a systematic evaluation of BERTimbau for Question Answering on SQuAD-BR, the Brazilian Portuguese translation of SQuAD v1.<n>We show that encoder-based models can be efficiently fine-tuned for extractive Brazilian Portuguese QA with substantially lower computational cost than large generative LLMs.
  arXiv  Detail & Related papers  (2026-03-22T21:56:05Z)
• Evaluating Prompt Engineering Techniques for RAG in Small Language Models: A Multi-Hop QA Approach [9.672512327395435]
  Retrieval Augmented Generation (RAG) is a powerful approach for enhancing the factual grounding of language models by integrating external knowledge.<n>This paper presents a large-scale empirical study to investigate the influence of prompt template design on RAG performance.<n>Our findings, based on a test set of 18720 instances, reveal significant performance gains of up to 83% on Qwen2.5 and 84.5% on Gemma3-4B-It.
  arXiv  Detail & Related papers  (2026-02-14T21:17:44Z)
• Parameter-Efficient Fine-Tuning for Low-Resource Languages: A Comparative Study of LLMs for Bengali Hate Speech Detection [0.0]
  This paper presents the first application of Efficient Fine-Tuning (PEFT) for Bengali hate speech detection using LoRA and QLoRA.<n>Three instruction-tuned large language models were fine-tuned on the BD-SHS dataset of 50,281 annotated comments.<n>Llama-3.2-3B achieved the highest F1-score of 92.23%, followed by Mistral-7B at 88.94% and Gemma-3-4B at 80.25%.
  arXiv  Detail & Related papers  (2025-10-19T20:03:22Z)
• Critique-GRPO: Advancing LLM Reasoning with Natural Language and Numerical Feedback [59.078756231841574]
  Critique-GRPO is an online RL framework that integrates both natural language and numerical feedback for effective policy optimization.<n>We show Critique-GRPO consistently outperforms supervised learning and RL-based fine-tuning methods across eight challenging mathematical, STEM, and general reasoning tasks.
  arXiv  Detail & Related papers  (2025-06-03T17:39:02Z)
• Language Model Preference Evaluation with Multiple Weak Evaluators [89.90733463933431]
  We introduce PGED, a novel approach that leverages multiple model-based evaluators to construct preference graphs, and then ensembles and denoises these graphs for acyclic, non-contradictory evaluation results.<n>We demonstrate PGED's superiority in three applications: 1) model ranking for evaluation, 2) response selection for test-time scaling, and 3) data selection for model fine-tuning.
  arXiv  Detail & Related papers  (2024-10-14T01:57:25Z)
• DataComp-LM: In search of the next generation of training sets for language models [200.5293181577585]
  DataComp for Language Models (DCLM) is a testbed for controlled dataset experiments with the goal of improving language models.<n>We provide a standardized corpus of 240T tokens extracted from Common Crawl, effective pretraining recipes based on the OpenLM framework, and a broad suite of 53 downstream evaluations.<n>Participants in the DCLM benchmark can experiment with data curation strategies such as deduplication, filtering, and data mixing at model scales ranging from 412M to 7B parameters.
  arXiv  Detail & Related papers  (2024-06-17T17:42:57Z)
• Advancing LLM Reasoning Generalists with Preference Trees [119.57169648859707]
  We introduce Eurus, a suite of large language models (LLMs) optimized for reasoning. Eurus models achieve state-of-the-art results among open-source models on a diverse set of benchmarks.
  arXiv  Detail & Related papers  (2024-04-02T16:25:30Z)
• Extreme Compression of Large Language Models via Additive Quantization [59.3122859349777]
  Our algorithm, called AQLM, generalizes the classic Additive Quantization (AQ) approach for information retrieval. We provide fast GPU and CPU implementations of AQLM for token generation, which enable us to match or outperform optimized FP16 implementations for speed.
  arXiv  Detail & Related papers  (2024-01-11T18:54:44Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.