Efficient Fine-Tuning Methods for Portuguese Question Answering: A Comparative Study of PEFT on BERTimbau and Exploratory Evaluation of Generative LLMs

• URL: http://arxiv.org/abs/2603.21418v1
• Date: Sun, 22 Mar 2026 21:56:05 GMT
• Title: Efficient Fine-Tuning Methods for Portuguese Question Answering: A Comparative Study of PEFT on BERTimbau and Exploratory Evaluation of Generative LLMs
• Abstract summary: This work presents a systematic evaluation of BERTimbau for Question Answering on SQuAD-BR, the Brazilian Portuguese translation of SQuAD v1.<n>We show that encoder-based models can be efficiently fine-tuned for extractive Brazilian Portuguese QA with substantially lower computational cost than large generative LLMs.
• Score: 1.5049442691806052
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Although large language models have transformed natural language processing, their computational costs create accessibility barriers for low-resource languages such as Brazilian Portuguese. This work presents a systematic evaluation of Parameter-Efficient Fine-Tuning (PEFT) and quantization techniques applied to BERTimbau for Question Answering on SQuAD-BR, the Brazilian Portuguese translation of SQuAD v1. We evaluate 40 configurations combining four PEFT methods (LoRA, DoRA, QLoRA, QDoRA) across two model sizes (Base: 110M, Large: 335M parameters). Our findings reveal three critical insights: (1) LoRA achieves 95.8\% of baseline performance on BERTimbau-Large while reducing training time by 73.5\% (F1=81.32 vs 84.86); (2) higher learning rates (2e-4) substantially improve PEFT performance, with F1 gains of up to +19.71 points over standard rates; and (3) larger models show twice the quantization resilience (loss of 4.83 vs 9.56 F1 points). These results demonstrate that encoder-based models can be efficiently fine-tuned for extractive Brazilian Portuguese QA with substantially lower computational cost than large generative LLMs, promoting more sustainable approaches aligned with \textit{Green AI} principles. An exploratory evaluation of Tucano and Sabiá on the same extractive QA benchmark shows that while generative models can reach competitive F1 scores with LoRA fine-tuning, they require up to 4.2$\times$ more GPU memory and 3$\times$ more training time than BERTimbau-Base, reinforcing the efficiency advantage of smaller encoder-based architectures for this task.

Related papers

• DistilledGemma: Balanced Efficiency-Accuracy for Person-Place Relation Extraction from Multilingual Historical Articles [3.079885946230076]
  We present DistilledGemma, an efficient and accurate system for the HIPE-2026 shared task on person-place relation extraction.<n>Our approach adopts a three-stage knowledge distillation pipeline designed to balance classification accuracy with computational efficiency.
  arXiv  Detail & Related papers  (2026-06-28T00:38:45Z)
• Adapting Large Language Models to a Low-Resource Agglutinative Language: A Comparative Study of LoRA and QLoRA for Bashkir [0.0]
  This paper presents a comparative study of parameter-efficient fine-tuning methods, including LoRA and QLoRA, applied to the task of adapting large language models to the Bashkir language.<n> Experimental evaluation is conducted on a Bashkir text corpus of 71k documents (46.9M tokens) using models of various architectures.
  arXiv  Detail & Related papers  (2026-05-06T14:14:35Z)
• Bielik-Q2-Sharp: A Comparative Study of Extreme 2-bit Quantization Methods for a Polish 11B Language Model [0.0]
  We present Bielik-Q2-Sharp, the first systematic academic evaluation of extreme 2-bit quantization applied to a Polish large language model.<n>We compare six state-of-the-art post-training quantization methods -- QuIP#, SpinQuant+GPTQ, ButterflyQuant, QTIP, VPTQ, and AQLM.<n>Our best variant (QuIP# E8P12) achieves 71.92% across 22 Polish benchmarks versus 72.07% for the IQ2_XXS baseline.
  arXiv  Detail & Related papers  (2026-03-04T15:19:35Z)
• Scaling Trends for Multi-Hop Contextual Reasoning in Mid-Scale Language Models [0.0]
  We present a controlled study of multi-hop contextual reasoning in large language models.<n>We show that multi-agent systems show the inverse pattern, achieving up to 80% on reasoning tasks where rule-based methods fail.
  arXiv  Detail & Related papers  (2026-01-06T20:18:55Z)
• Efficient Hate Speech Detection: A Three-Layer LoRA-Tuned BERTweet Framework [0.0]
  This paper addresses the challenge of developing computationally efficient hate speech detection systems.<n>We propose a novel three-layer framework that combines rule-based pre-filtering with a parameter-efficient LoRA-tuned BERTweet model.<n>Our approach achieves 94% of the performance of state-of-the-art large language models like SafePhi.
  arXiv  Detail & Related papers  (2025-11-08T15:47:18Z)
• Every Step Evolves: Scaling Reinforcement Learning for Trillion-Scale Thinking Model [100.86587937568832]
  Ring-1T is the first open-source, state-of-the-art thinking model with a trillion-scale parameter.<n>It features 1 trillion total parameters and activates approximately 50 billion per token.
  arXiv  Detail & Related papers  (2025-10-21T17:46:14Z)
• Teaching LLM to Reason: Reinforcement Learning from Algorithmic Problems without Code [76.80306464249217]
  We propose TeaR, which aims at teaching LLMs to reason better.<n>TeaR leverages careful data curation and reinforcement learning to guide models in discovering optimal reasoning paths through code-related tasks.<n>We conduct extensive experiments using two base models and three long-CoT distillation models, with model sizes ranging from 1.5 billion to 32 billion parameters, and across 17 benchmarks spanning Math, Knowledge, Code, and Logical Reasoning.
  arXiv  Detail & Related papers  (2025-07-10T07:34:05Z)
• Hunyuan-TurboS: Advancing Large Language Models through Mamba-Transformer Synergy and Adaptive Chain-of-Thought [196.74837065805488]
  Hunyuan-TurboS is a large hybrid Transformer-Mamba Mixture of Experts model.<n>It balances high performance and efficiency, offering substantial capabilities at lower inference costs.
  arXiv  Detail & Related papers  (2025-05-21T12:11:53Z)
• EfficientLLM: Efficiency in Large Language Models [64.3537131208038]
  Large Language Models (LLMs) have driven significant progress, yet their growing counts and context windows incur prohibitive compute, energy, and monetary costs.<n>We introduce EfficientLLM, a novel benchmark and the first comprehensive empirical study evaluating efficiency techniques for LLMs at scale.
  arXiv  Detail & Related papers  (2025-05-20T02:27:08Z)
• Phi-4-Mini-Reasoning: Exploring the Limits of Small Reasoning Language Models in Math [135.1260782461186]
  Chain-of-Thought (CoT) significantly enhances formal reasoning capabilities in Large Language Models (LLMs)<n>However, improving reasoning in Small Language Models (SLMs) remains challenging due to their limited model capacity.<n>We present a systematic training recipe for SLMs that consists of four steps: (1) large-scale mid-training on diverse distilled long-CoT data, (2) supervised fine-tuning on high-quality long-CoT data, (3) Rollout DPO leveraging a carefully curated preference dataset, and (4) Reinforcement Learning (RL) with Verifiable Reward.
  arXiv  Detail & Related papers  (2025-04-30T00:04:35Z)
• Unlocking Parameter-Efficient Fine-Tuning for Low-Resource Language Translation [3.558269947618352]
  We evaluate the performance of 8 PEFT methods with in total of 15 architectures using the SacreBLEU score. We showed that 6 PEFT architectures outperform the baseline for both in-domain and out-domain tests. The Houlsby+Inversion adapter has the best performance overall, proving the effectiveness of PEFT methods.
  arXiv  Detail & Related papers  (2024-04-05T16:42:28Z)
• Breaking Language Barriers in Multilingual Mathematical Reasoning: Insights and Observations [59.056367787688146]
  This paper pioneers exploring and training powerful Multilingual Math Reasoning (xMR) LLMs. We construct the first multilingual math reasoning instruction dataset, MGSM8KInstruct, encompassing ten distinct languages. By utilizing translation, we construct the first multilingual math reasoning instruction dataset, MGSM8KInstruct, encompassing ten distinct languages.
  arXiv  Detail & Related papers  (2023-10-31T08:09:20Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.