Empowering Persian LLMs for Instruction Following: A Novel Dataset and Training Approach

• URL: http://arxiv.org/abs/2407.11186v4
• Date: Tue, 14 Jan 2025 22:07:53 GMT
• Title: Empowering Persian LLMs for Instruction Following: A Novel Dataset and Training Approach
• Authors: Hojjat Mokhtarabadi, Ziba Zamani, Abbas Maazallahi, Mohammad Hossein Manshaei,
• Abstract summary: We introduce FarsInstruct, a dataset designed to enhance the instruction following ability of large language models. FarsInstruct comprises 197 templates across 21 distinct datasets, and we intend to update it consistently, thus augmenting its applicability.
• Score: 0.0
• License:
• Abstract: Instruction-tuned large language models have demonstrated remarkable capabilities in following human instructions across various domains. However, their proficiency remains notably deficient in many low-resource languages. To address this challenge, we begin by introducing FarsInstruct a comprehensive instruction dataset designed to enhance the instruction following ability of large language models specifically for the Persian language a significant yet underrepresented language globally. FarsInstruct encompasses a wide range of task types and datasets, each containing a mix of straightforward to complex manual written instructions, as well as translations from the Public Pool of Prompts, ensuring a rich linguistic and cultural representation. Furthermore, we introduce Co-CoLA, a framework designed to enhance the multi-task adaptability of LoRA-tuned models. Through extensive experimental analyses, our study showcases the effectiveness of the FarsInstruct dataset coupled with training by the Co-CoLA framework, in improving the performance of large language models within the Persian context. As of the current writing, FarsInstruct comprises 197 templates across 21 distinct datasets, and we intend to update it consistently, thus augmenting its applicability.

Related papers

• LUSIFER: Language Universal Space Integration for Enhanced Multilingual Embeddings with Large Language Models [89.13128402847943]
  We present LUSIFER, a novel zero-shot approach that adapts LLM-based embedding models for multilingual tasks without requiring multilingual supervision. LUSIFER's architecture combines a multilingual encoder, serving as a language-universal learner, with an LLM-based embedding model optimized for embedding-specific tasks. We introduce a new benchmark encompassing 5 primary embedding tasks, 123 diverse datasets, and coverage across 14 languages.
  arXiv  Detail & Related papers  (2025-01-01T15:43:07Z)
• Chunk-Distilled Language Modeling [25.238256586953487]
  Chunk-Distilled Language Modeling (CD-LM) is an approach to text generation that addresses two challenges in current large language models (LLMs) Our method combines deep network-based LLMs with a straightforward retrieval module, which allows the generation of multi-token text chunks at a single decoding step.
  arXiv  Detail & Related papers  (2024-12-31T08:32:15Z)
• Extending LLMs to New Languages: A Case Study of Llama and Persian Adaptation [36.92567530333872]
  We study adding a new language, i.e. Persian, to a large language model (LLMs) We employ a multi-stage approach involving pretraining on monolingual Persian data. We evaluate the model's performance at each stage on generation and classification tasks.
  arXiv  Detail & Related papers  (2024-12-17T23:18:06Z)
• MLAN: Language-Based Instruction Tuning Improves Zero-Shot Generalization of Multimodal Large Language Models [79.0546136194314]
  We present a novel instruction tuning recipe to improve the zero-shot task generalization of multimodal large language models. We evaluate the performance of the proposed approach on 9 unseen datasets across both language and vision modalities.
  arXiv  Detail & Related papers  (2024-11-15T20:09:59Z)
• Investigating Multilingual Instruction-Tuning: Do Polyglot Models Demand for Multilingual Instructions? [42.37657013017192]
  We show that instruction-tuning on parallel instead of monolingual corpora benefits cross-lingual instruction following capabilities by up to 9.9%. We also conduct a human annotation study to understand the alignment between human-based and GPT-4-based evaluation within multilingual chat scenarios.
  arXiv  Detail & Related papers  (2024-02-21T11:07:07Z)
• Synthetic Data (Almost) from Scratch: Generalized Instruction Tuning for Language Models [153.14575887549088]
  We introduce Generalized Instruction Tuning (called GLAN), a general and scalable method for instruction tuning of Large Language Models (LLMs) GLAN exclusively utilizes a pre-curated taxonomy of human knowledge and capabilities as input and generates large-scale synthetic instruction data across all disciplines. With the fine-grained key concepts detailed in every class session of the syllabus, we are able to generate diverse instructions with a broad coverage across the entire spectrum of human knowledge and skills.
  arXiv  Detail & Related papers  (2024-02-20T15:00:35Z)
• TPPoet: Transformer-Based Persian Poem Generation using Minimal Data and Advanced Decoding Techniques [0.9831489366502302]
  We train a Persian classical poetry generation model using a transformer architecture on a specialized dataset with no pretraining. We propose a novel decoding method to enhance coherence and meaningfulness in the generated poetry. The results of our training approach and the proposed decoding method are evaluated through comprehensive set of automatic and human evaluations.
  arXiv  Detail & Related papers  (2023-12-04T18:52:26Z)
• Pre-Training to Learn in Context [138.0745138788142]
  The ability of in-context learning is not fully exploited because language models are not explicitly trained to learn in context. We propose PICL (Pre-training for In-Context Learning), a framework to enhance the language models' in-context learning ability. Our experiments show that PICL is more effective and task-generalizable than a range of baselines, outperforming larger language models with nearly 4x parameters.
  arXiv  Detail & Related papers  (2023-05-16T03:38:06Z)
• Efficiently Aligned Cross-Lingual Transfer Learning for Conversational Tasks using Prompt-Tuning [98.60739735409243]
  Cross-lingual transfer of language models trained on high-resource languages like English has been widely studied for many NLP tasks. We introduce XSGD for cross-lingual alignment pretraining, a parallel and large-scale multilingual conversation dataset. To facilitate aligned cross-lingual representations, we develop an efficient prompt-tuning-based method for learning alignment prompts.
  arXiv  Detail & Related papers  (2023-04-03T18:46:01Z)
• LICHEE: Improving Language Model Pre-training with Multi-grained Tokenization [19.89228774074371]
  We propose a simple yet effective pre-training method named LICHEE to efficiently incorporate multi-grained information of input text. Our method can be applied to various pre-trained language models and improve their representation capability.
  arXiv  Detail & Related papers  (2021-08-02T12:08:19Z)
• UNKs Everywhere: Adapting Multilingual Language Models to New Scripts [103.79021395138423]
  Massively multilingual language models such as multilingual BERT (mBERT) and XLM-R offer state-of-the-art cross-lingual transfer performance on a range of NLP tasks. Due to their limited capacity and large differences in pretraining data, there is a profound performance gap between resource-rich and resource-poor target languages. We propose novel data-efficient methods that enable quick and effective adaptation of pretrained multilingual models to such low-resource languages and unseen scripts.
  arXiv  Detail & Related papers  (2020-12-31T11:37:28Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.