RoQLlama: A Lightweight Romanian Adapted Language Model

• URL: http://arxiv.org/abs/2410.04269v1
• Date: Sat, 5 Oct 2024 19:14:11 GMT
• Title: RoQLlama: A Lightweight Romanian Adapted Language Model
• Abstract summary: We tackle the problem of reduced computing resources by using QLoRA for training. We release RoQLlama-7b, a quantized LLM, which shows equal or improved results compared to its full-sized counterpart. We introduce a novel Romanian dataset, namely RoMedQA, which contains single-choice medical questions in Romanian.
• Score: 2.1892046440619626
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The remarkable achievements obtained by open-source large language models (LLMs) in recent years have predominantly been concentrated on tasks involving the English language. In this paper, we aim to advance the performance of Llama2 models on Romanian tasks. We tackle the problem of reduced computing resources by using QLoRA for training. We release RoQLlama-7b, a quantized LLM, which shows equal or improved results compared to its full-sized counterpart when tested on seven Romanian downstream tasks in the zero-shot setup. Also, it consistently achieves higher average scores across all few-shot prompts. Additionally, we introduce a novel Romanian dataset, namely RoMedQA, which contains single-choice medical questions in Romanian.

Related papers

• Parameter Efficient Multimodal Instruction Tuning for Romanian Vision Language Models [2.1990085634399428]
  We contribute to reducing the multimodal NLP resource gap for Romanian.<n>We translate the widely known Flickr30k dataset into Romanian and further extend it for visual question answering.<n>We demonstrate the usefulness of our datasets by fine-tuning open-source VLMs on Romanian visual question answering.
  arXiv  Detail & Related papers  (2025-12-16T21:36:28Z)
• Fine-Tuning Large Language Models with QLoRA for Offensive Language Detection in Roman Urdu-English Code-Mixed Text [5.908448629364552]
  We propose a QLoRA based fine tuning framework to improve offensive language detection in Roman Urdu-English text.<n>We translate the Roman Urdu-English code mixed dataset into English using Google Translate to leverage English LLMs.<n>We fine tuned several transformers and large language models, including Meta LLaMA 3 8B, Mistral 7B v0.1, LLaMA 2 7B, ModernBERT, and RoBERTa.
  arXiv  Detail & Related papers  (2025-10-04T05:38:46Z)
• MedQARo: A Large-Scale Benchmark for Medical Question Answering in Romanian [50.767415194856135]
  We introduce MedQARo, the first large-scale medical QA benchmark in Romanian.<n>We construct a high-quality and large-scale dataset comprising 102,646 QA pairs related to cancer patients.
  arXiv  Detail & Related papers  (2025-08-22T13:48:37Z)
• Low-Resource Transliteration for Roman-Urdu and Urdu Using Transformer-Based Models [0.6554326244334868]
  Transliteration between Urdu and its Romanized form, Roman Urdu, remains underexplored. We propose a transformer-based approach using the m2m100 multilingual translation model. Our model achieves strong transliteration performance, with Char-BLEU scores of 96.37 for Urdu->Roman-Urdu and 97.44 for Roman-Urdu->Urdu.
  arXiv  Detail & Related papers  (2025-03-27T14:18:50Z)
• Demystifying Multilingual Chain-of-Thought in Process Reward Modeling [71.12193680015622]
  We tackle the challenge of extending process reward models (PRMs) to multilingual settings. We train multilingual PRMs on a dataset spanning seven languages, which is translated from English. Our results highlight the sensitivity of multilingual PRMs to both the number of training languages and the volume of English data.
  arXiv  Detail & Related papers  (2025-02-18T09:11:44Z)
• LLMic: Romanian Foundation Language Model [76.09455151754062]
  We present LLMic, a foundation language model designed specifically for the Romanian Language. We show that fine-tuning LLMic for language translation after the initial pretraining phase outperforms existing solutions in English-to-Romanian translation tasks.
  arXiv  Detail & Related papers  (2025-01-13T22:14:45Z)
• PromptRefine: Enhancing Few-Shot Performance on Low-Resource Indic Languages with Example Selection from Related Example Banks [57.86928556668849]
  Large Language Models (LLMs) have recently demonstrated impressive few-shot learning capabilities through in-context learning (ICL) ICL performance is highly dependent on the choice of few-shot demonstrations, making the selection of the most optimal examples a persistent research challenge. In this work, we propose PromptRefine, a novel Alternating Minimization approach for example selection that improves ICL performance on low-resource Indic languages.
  arXiv  Detail & Related papers  (2024-12-07T17:51:31Z)
• Think Carefully and Check Again! Meta-Generation Unlocking LLMs for Low-Resource Cross-Lingual Summarization [108.6908427615402]
  Cross-lingual summarization ( CLS) aims to generate a summary for the source text in a different target language. Currently, instruction-tuned large language models (LLMs) excel at various English tasks. Recent studies have shown that LLMs' performance on CLS tasks remains unsatisfactory even with few-shot settings.
  arXiv  Detail & Related papers  (2024-10-26T00:39:44Z)
• "Vorbeşti Româneşte?" A Recipe to Train Powerful Romanian LLMs with English Instructions [40.64721381920061]
  We are the first to collect and translate a large collection of texts, instructions, and benchmarks and train, evaluate, and release open-source LLMs tailored for Romanian. We argue for the usefulness and high performance of RoLLMs by obtaining state-of-the-art results across the board.
  arXiv  Detail & Related papers  (2024-06-26T11:39:51Z)
• Language Models can Exploit Cross-Task In-context Learning for Data-Scarce Novel Tasks [22.66167973623777]
  Large Language Models (LLMs) have transformed NLP with their remarkable In-context Learning (ICL) capabilities. This paper investigates whether LLMs can generalize from labeled examples of predefined tasks to novel tasks. We show that cross-task prompting leads to a remarkable performance boost of 107% for LLaMA-2 7B, 18.6% for LLaMA-2 13B, and 3.2% for GPT 3.5 on average over zero-shot prompting.
  arXiv  Detail & Related papers  (2024-05-17T05:20:49Z)
• GeMQuAD : Generating Multilingual Question Answering Datasets from Large Language Models using Few Shot Learning [4.8838210812204235]
  In this paper, we propose GeMQuAD - a semi-supervised learning approach, applied to a dataset generated through ICL with just one example in the target language. We iteratively identify high-quality data to enhance model performance, especially for low-resource multilingual setting. Our framework outperforms the machine translation-augmented model by 0.22/1.68 F1/EM points for Hindi and 0.82/1.37 F1/EM points for Spanish on the MLQA dataset.
  arXiv  Detail & Related papers  (2024-04-14T06:55:42Z)
• RoCode: A Dataset for Measuring Code Intelligence from Problem Definitions in Romanian [10.035193313198207]
  We present RoCode, a competitive programming dataset consisting of 2,642 problems written in Romanian. We argue for the need to develop code models for languages other than English.
  arXiv  Detail & Related papers  (2024-02-20T18:32:47Z)
• Look Before You Leap: A Universal Emergent Decomposition of Retrieval Tasks in Language Models [58.57279229066477]
  We study how language models (LMs) solve retrieval tasks in diverse situations. We introduce ORION, a collection of structured retrieval tasks spanning six domains. We find that LMs internally decompose retrieval tasks in a modular way.
  arXiv  Detail & Related papers  (2023-12-13T18:36:43Z)
• PAXQA: Generating Cross-lingual Question Answering Examples at Training Scale [53.92008514395125]
  PAXQA (Projecting annotations for cross-lingual (x) QA) decomposes cross-lingual QA into two stages. We propose a novel use of lexically-constrained machine translation, in which constrained entities are extracted from the parallel bitexts. We show that models fine-tuned on these datasets outperform prior synthetic data generation models over several extractive QA datasets.
  arXiv  Detail & Related papers  (2023-04-24T15:46:26Z)
• Ensemble Transfer Learning for Multilingual Coreference Resolution [60.409789753164944]
  A problem that frequently occurs when working with a non-English language is the scarcity of annotated training data. We design a simple but effective ensemble-based framework that combines various transfer learning techniques. We also propose a low-cost TL method that bootstraps coreference resolution models by utilizing Wikipedia anchor texts.
  arXiv  Detail & Related papers  (2023-01-22T18:22:55Z)
• RoMQA: A Benchmark for Robust, Multi-evidence, Multi-answer Question Answering [87.18962441714976]
  We introduce RoMQA, the first benchmark for robust, multi-evidence, multi-answer question answering (QA) We evaluate state-of-the-art large language models in zero-shot, few-shot, and fine-tuning settings, and find that RoMQA is challenging. Our results show that RoMQA is a challenging benchmark for large language models, and provides a quantifiable test to build more robust QA methods.
  arXiv  Detail & Related papers  (2022-10-25T21:39:36Z)
• MOROCCO: Model Resource Comparison Framework [61.444083353087294]
  We present MOROCCO, a framework to compare language models compatible with ttjiant environment which supports over 50 NLU tasks. We demonstrate its applicability for two GLUE-like suites in different languages.
  arXiv  Detail & Related papers  (2021-04-29T13:01:27Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.