Quantum-RAG and PunGPT2: Advancing Low-Resource Language Generation and Retrieval for the Punjabi Language

• URL: http://arxiv.org/abs/2508.01918v1
• Date: Sun, 03 Aug 2025 21:03:22 GMT
• Title: Quantum-RAG and PunGPT2: Advancing Low-Resource Language Generation and Retrieval for the Punjabi Language
• Abstract summary: We present PunGPT2, the first fully open-source suite of Punjabi large language models.<n>We also present Pun-RAG, a retrieval-augmented generation framework combining PunGPT2 with a dense FAISS retriever.<n>We propose Quantum-RAG, a novel hybrid retrieval system that fuses sparse (BM25) and dense methods with quantum-inspired semantic matching.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Despite the rapid advancement of large language models (LLMs), low-resource languages remain largely excluded from the NLP landscape. We present PunGPT2, the first fully open-source suite of Punjabi large language models, trained from scratch on a 35GB domain-diverse corpus encompassing literature, religious texts, news, and social discourse. Unlike prior multilingual approaches, PunGPT2 captures rich syntactic and morphological features unique to Punjabi through a tokenizer optimised with byte pair encoding and linguistically aligned pretraining objectives. To improve factual grounding and domain recall, we introduce Pun-RAG, a retrieval-augmented generation framework combining PunGPT2 with a dense FAISS retriever over a curated Punjabi knowledge base. We further develop Pun-Instruct, a parameter-efficient, instruction-tuned variant using QLoRA, enabling robust zero-shot and instruction-following performance with significantly reduced compute needs. As a key innovation, we propose Quantum-RAG, a novel hybrid retrieval system that fuses sparse (BM25) and dense methods with quantum-inspired semantic matching. By encoding queries using amplitude-based embeddings and retrieving via quantum kernel similarity, Quantum-RAG achieves improved contextual relevance with minimal memory overhead marking the first practical integration of quantum representations in low-resource language generation. Our models significantly outperform strong multilingual baselines (mBERT, mT5, MuRIL) in perplexity, factuality, and fluency. This work provides a scalable, reproducible blueprint for extending LLM capabilities to underrepresented languages and pioneers quantum-aware retrieval in low-resource NLP

Related papers

• MameLoshnLM: Yiddish Language Model and Evaluation Benchmark [62.936233688546984]
  We present MameLoshnLM, the first open-source 8B- parameter language model built specifically for Yiddish.<n>Despite Yiddish's rich textual tradition, its limited digital presence and the scarcity of reliable evaluation resources have constrained progress in Yiddish language modeling.<n>Our results provide both a foundation for Yiddish NLP and a practical template for language model development in historically rich but digitally underrepresented languages.
  arXiv  Detail & Related papers  (2026-08-06T10:24:08Z)
• BHARATI: Morphology-Aware Tokenizers for Classical Indian Languages with Subword Fertility Analysis [0.0]
  This paper presents BHARATI, a set of SentencePiece BPE token trained on a balanced 781 MB corpus spanning seven languages.<n>We describe three successive tokenizer versions: v1 (English and Sanskrit only), v2 (four-language support with byte-fallback for Tamil), and v3 (full seven-language native subword coverage)<n>Subword fertility analysis demonstrates that v3 averages 2.6 tokens per Indian Knowledge System (IKS) technical term, compared to 5.25 tokens per term with GPT-2's tokenizer and 3.75 tokens with the multilingual SentencePiece baseline.
  arXiv  Detail & Related papers  (2026-07-25T18:23:06Z)
• Omnilingual SONAR: Cross-Lingual and Cross-Modal Sentence Embeddings Bridging Massively Multilingual Text and Speech [61.759910921200834]
  Cross-lingual sentence encoders typically cover only a few hundred languages.<n>We introduce OmniSONAR, a new family of omnilingual, cross-lingual and cross-language sentence embedding models.
  arXiv  Detail & Related papers  (2026-03-17T14:47:35Z)
• Nwāchā Munā: A Devanagari Speech Corpus and Proximal Transfer Benchmark for Nepal Bhasha ASR [0.9918851323801791]
  We introduce Nwch Mun, a newly curated 5.39-hour manually transcribed Devanagari speech corpus for Nepal Bhasha.<n>We establish the first benchmark using script-preserving annotated acoustic modeling.
  arXiv  Detail & Related papers  (2026-03-08T09:35:05Z)
• Benchmarking BERT-based Models for Sentence-level Topic Classification in Nepali Language [1.6474262142781433]
  This study benchmarks multilingual, Indic, Hindi, and Nepali BERT variants to evaluate their effectiveness in Nepali topic classification.<n>Ten pre-trained models, including mBERT, XLM-R, MuRIL, DevBERT, HindiBERT, IndicBERT, and NepBERTa, were fine-tuned and tested.<n>Indic models, particularly MuRIL-large, achieved the highest F1-score of 90.60%, outperforming multilingual and monolingual models.
  arXiv  Detail & Related papers  (2026-02-27T11:42:38Z)
• ks-lit-3m: A 3.1 million word kashmiri text dataset for large language model pretraining [0.0]
  This paper introduces KS-LIT-3M, a curated corpus of 3.1 million words (16.4 million characters) specifically designed for pretraining language models on Kashmiri.<n>The dataset is released under the CC-BY-4.0 license to facilitate research in Kashmiri natural language processing.
  arXiv  Detail & Related papers  (2026-01-03T06:43:26Z)
• Evaluating Subword Tokenization Techniques for Bengali: A Benchmark Study with BengaliBPE [0.0]
  BengaliBPE is a language-aware subword tokenizer for the Bengali script.<n>It applies Unicode normalization and morphology-aware merge rules to maintain linguistic consistency and preserve subword integrity.<n>It provides the most detailed segmentation and the best morphological interpretability, albeit with slightly higher computational cost.
  arXiv  Detail & Related papers  (2025-11-07T15:23:32Z)
• Leveraging the Cross-Domain & Cross-Linguistic Corpus for Low Resource NMT: A Case Study On Bhili-Hindi-English Parallel Corpus [3.435561406656216]
  The linguistic diversity of India poses significant machine translation challenges, especially for underrepresented tribal languages like Bhili.<n>This paper introduces Bhili-Hindi-English Parallel Corpus (BH EPC), the first and largest parallel corpus worldwide comprising 110,000 meticulously curated sentences across Bhili, Hindi, and English.<n>BH EPC spans critical domains such as education, administration, and news, establishing a valuable benchmark for research in low resource machine translation.
  arXiv  Detail & Related papers  (2025-11-01T10:39:56Z)
• Towards Inclusive NLP: Assessing Compressed Multilingual Transformers across Diverse Language Benchmarks [33.2185998586144]
  This study benchmarks the performance of multilingual and monolingual Large Language Models (LLMs) across Arabic, English, and Indic languages.<n>Findings show significant performance differences driven by linguistic diversity and resource availability.<n> Quantization (4-bit and 8-bit) is effective in maintaining model accuracy while promoting efficiency, but aggressive pruning significantly compromises performance.
  arXiv  Detail & Related papers  (2025-07-25T22:35:10Z)
• NeoBabel: A Multilingual Open Tower for Visual Generation [32.79724699684266]
  We introduce NeoBabel, a novel multilingual image generation framework.<n>It supports six languages: English, Chinese, Dutch, French, Hindi, and Persian.<n>It achieves state-of-the-art multilingual performance while retaining strong English capability.
  arXiv  Detail & Related papers  (2025-07-08T16:19:45Z)
• Leveraging the Potential of Prompt Engineering for Hate Speech Detection in Low-Resource Languages [2.8811725782388686]
  This paper investigates how we can overcome the limitation via prompt engineering on large language models (LLMs) focusing on low-resource Bengali language.<n>We investigate six prompting strategies - zero-shot prompting, refusal suppression, flattering the classifier, multi-shot prompting, role prompting, and finally our innovative metaphor prompting to detect hate speech effectively in low-resource languages.<n>To prove the effectiveness of our metaphor prompting in the low-resource Bengali language, we also evaluate it in another low-resource language Hindi, and two high-resource languages - English and German.
  arXiv  Detail & Related papers  (2025-06-30T14:59:25Z)
• Efficient Generation of Parameterised Quantum Circuits from Large Texts [0.3298092151372303]
  DisCoCirc is capable of directly encoding entire documents as parameterised quantum circuits (PQCs)<n>This paper introduces an efficient methodology for converting large-scale texts into quantum circuits using tree-like representations of pregroup diagrams.
  arXiv  Detail & Related papers  (2025-05-19T14:57:53Z)
• HYPEROFA: Expanding LLM Vocabulary to New Languages via Hypernetwork-Based Embedding Initialization [50.27950279695363]
  Many pre-trained language models (PLMs) exhibit suboptimal performance on mid- and low-resource languages.<n>A common strategy to address this is to introduce new tokens specific to the target languages, initialize their embeddings, and apply continual pre-training on target-language data.<n>We propose HYPEROFA, a hypernetwork-based approach for more adaptive token embedding.
  arXiv  Detail & Related papers  (2025-04-21T19:40:32Z)
• CoSTA: Code-Switched Speech Translation using Aligned Speech-Text Interleaving [61.73180469072787]
  We focus on the problem of spoken translation (ST) of code-switched speech in Indian languages to English text. We present a new end-to-end model architecture COSTA that scaffolds on pretrained automatic speech recognition (ASR) and machine translation (MT) modules. COSTA significantly outperforms many competitive cascaded and end-to-end multimodal baselines by up to 3.5 BLEU points.
  arXiv  Detail & Related papers  (2024-06-16T16:10:51Z)
• BTR: Binary Token Representations for Efficient Retrieval Augmented Language Models [77.0501668780182]
  Retrieval augmentation addresses many critical problems in large language models. Running retrieval-augmented language models (LMs) is slow and difficult to scale due to processing large amounts of retrieved text. We introduce binary token representations (BTR), which use 1-bit vectors to precompute every token in passages.
  arXiv  Detail & Related papers  (2023-10-02T16:48:47Z)
• Baichuan 2: Open Large-scale Language Models [51.34140526283222]
  We present Baichuan 2, a series of large-scale multilingual language models containing 7 billion and 13 billion parameters, trained from scratch, on 2.6 trillion tokens.<n>Baichuan 2 matches or outperforms other open-source models of similar size on public benchmarks like MMLU, CMMLU, GSM8K, and HumanEval.
  arXiv  Detail & Related papers  (2023-09-19T04:13:22Z)
• Toward Quantum Machine Translation of Syntactically Distinct Languages [0.0]
  We explore the feasibility of language translation using quantum natural language processing algorithms on noisy intermediate-scale quantum (NISQ) devices. We employ Shannon entropy to demonstrate the significant role of some appropriate angles of rotation gates in the performance of parametrized quantum circuits.
  arXiv  Detail & Related papers  (2023-07-31T11:24:54Z)
• Soft Language Clustering for Multilingual Model Pre-training [57.18058739931463]
  We propose XLM-P, which contextually retrieves prompts as flexible guidance for encoding instances conditionally. Our XLM-P enables (1) lightweight modeling of language-invariant and language-specific knowledge across languages, and (2) easy integration with other multilingual pre-training methods.
  arXiv  Detail & Related papers  (2023-06-13T08:08:08Z)
• Extrapolating Multilingual Understanding Models as Multilingual Generators [82.1355802012414]
  This paper explores methods to empower multilingual understanding models the generation abilities to get a unified model. We propose a textbfSemantic-textbfGuided textbfAlignment-then-Denoising (SGA) approach to adapt an encoder to a multilingual generator with a small number of new parameters.
  arXiv  Detail & Related papers  (2023-05-22T15:33:21Z)
• Mitigating Data Imbalance and Representation Degeneration in Multilingual Machine Translation [103.90963418039473]
  Bi-ACL is a framework that uses only target-side monolingual data and a bilingual dictionary to improve the performance of the MNMT model. We show that Bi-ACL is more effective both in long-tail languages and in high-resource languages.
  arXiv  Detail & Related papers  (2023-05-22T07:31:08Z)
• Memory-efficient NLLB-200: Language-specific Expert Pruning of a Massively Multilingual Machine Translation Model [92.91310997807936]
  NLLB-200 is a set of multilingual Neural Machine Translation models that cover 202 languages. We propose a pruning method that enables the removal of up to 80% of experts without further finetuning.
  arXiv  Detail & Related papers  (2022-12-19T19:29:40Z)
• LAMASSU: Streaming Language-Agnostic Multilingual Speech Recognition and Translation Using Neural Transducers [71.76680102779765]
  Automatic speech recognition (ASR) and speech translation (ST) can both use neural transducers as the model structure. We propose LAMASSU, a streaming language-agnostic multilingual speech recognition and translation model using neural transducers.
  arXiv  Detail & Related papers  (2022-11-05T04:03:55Z)
• naab: A ready-to-use plug-and-play corpus for Farsi [1.381198851698147]
  naab is the largest publicly available, cleaned, and ready-to-use Farsi textual corpus.<n>Naab consists of 130GB of data, comprising over 250 million paragraphs and 15 billion words.<n>Naab-raw is an unprocessed version of the dataset, along with a pre-processing toolkit.
  arXiv  Detail & Related papers  (2022-08-29T10:40:58Z)
• Evaluation of Transfer Learning for Polish with a Text-to-Text Model [54.81823151748415]
  We introduce a new benchmark for assessing the quality of text-to-text models for Polish. The benchmark consists of diverse tasks and datasets: KLEJ benchmark adapted for text-to-text, en-pl translation, summarization, and question answering. We present plT5 - a general-purpose text-to-text model for Polish that can be fine-tuned on various Natural Language Processing (NLP) tasks with a single training objective.
  arXiv  Detail & Related papers  (2022-05-18T09:17:14Z)
• Continual Mixed-Language Pre-Training for Extremely Low-Resource Neural Machine Translation [53.22775597051498]
  We present a continual pre-training framework on mBART to effectively adapt it to unseen languages. Results show that our method can consistently improve the fine-tuning performance upon the mBART baseline. Our approach also boosts the performance on translation pairs where both languages are seen in the original mBART's pre-training.
  arXiv  Detail & Related papers  (2021-05-09T14:49:07Z)
• UNKs Everywhere: Adapting Multilingual Language Models to New Scripts [103.79021395138423]
  Massively multilingual language models such as multilingual BERT (mBERT) and XLM-R offer state-of-the-art cross-lingual transfer performance on a range of NLP tasks. Due to their limited capacity and large differences in pretraining data, there is a profound performance gap between resource-rich and resource-poor target languages. We propose novel data-efficient methods that enable quick and effective adaptation of pretrained multilingual models to such low-resource languages and unseen scripts.
  arXiv  Detail & Related papers  (2020-12-31T11:37:28Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.