Fine-tuning Large Language Models for Domain-specific Machine Translation

• URL: http://arxiv.org/abs/2402.15061v2
• Date: Tue, 17 Dec 2024 12:45:20 GMT
• Title: Fine-tuning Large Language Models for Domain-specific Machine Translation
• Authors: Jiawei Zheng, Hanghai Hong, Feiyan Liu, Xiaoli Wang, Jingsong Su, Yonggui Liang, Shikai Wu,
• Abstract summary: Large language models (LLMs) have shown great potential in domain-specific machine translation (MT)<n>This paper focuses on enhancing the domain-specific MT capability of LLMs by providing high-quality training datasets and proposing a novel fine-tuning framework denoted by DragFT.<n>The results on three domain-specific datasets show that DragFT achieves a significant performance boost and shows superior performance compared to advanced models such as GPT-3.5 and GPT-4o.
• Score: 7.977136709446714
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Large language models (LLMs) have shown great potential in domain-specific machine translation (MT). However, one major issue is that LLMs pre-trained on general domain corpus might not generalize well to specific domains due to the lack of domain-specific knowledge. To address this issue, this paper focuses on enhancing the domain-specific MT capability of LLMs, by providing high-quality training datasets and proposing a novel fine-tuning framework denoted by DragFT. DragFT augments LLMs via three techniques: (i) Dictionary-enhanced prompting integrates dictionary information into prompts to improve the translation of domain-specific terminology.; (ii) RAG-based few-shot example selection provides high-quality examples that simulate both the domain and style characteristics; (iii) Fine-tuning with few-shot examples further enhances performance when using in-domain examples. We deploy DragFT on three well-known LLM backbones with 13B training parameters to validate its effectiveness. The results on three domain-specific datasets show that DragFT achieves a significant performance boost and shows superior performance compared to advanced models such as GPT-3.5 and GPT-4o. The drastic performance improvement of DragFT over existing LLMs can be attributed to incorporating relevant knowledge while mitigating noise.

Related papers

• Leveraging Domain Knowledge at Inference Time for LLM Translation: Retrieval versus Generation [36.41708236431343]
  Large language models (LLMs) have been increasingly adopted for machine translation (MT) Our work studies domain-adapted MT with LLMs through a careful prompting setup. We find that demonstrations consistently outperform terminology, and retrieval consistently outperforms generation.
  arXiv  Detail & Related papers  (2025-03-06T22:23:07Z)
• Refining Translations with LLMs: A Constraint-Aware Iterative Prompting Approach [7.5069214839655345]
  Large language models (LLMs) have demonstrated remarkable proficiency in machine translation (MT) We propose a multi-step prompt chain that enhances translation faithfulness by prioritizing key terms crucial for semantic accuracy. Experiments using Llama and Qwen as base models on the FLORES-200 and WMT datasets demonstrate significant improvements over baselines.
  arXiv  Detail & Related papers  (2024-11-13T05:40:24Z)
• Large Language Model for Multi-Domain Translation: Benchmarking and Domain CoT Fine-tuning [55.107329995417786]
  Large language models (LLMs) have demonstrated impressive general understanding and generation abilities. We establish a benchmark for multi-domain translation, featuring 25 German$Leftrightarrow$English and 22 Chinese$Leftrightarrow$English test sets. We propose a domain Chain of Thought (CoT) fine-tuning technique that utilizes the intrinsic multi-domain intelligence of LLMs to improve translation performance.
  arXiv  Detail & Related papers  (2024-10-03T16:15:04Z)
• Exploring Language Model Generalization in Low-Resource Extractive QA [57.14068405860034]
  We investigate Extractive Question Answering (EQA) with Large Language Models (LLMs) under domain drift. We devise a series of experiments to explain the performance gap empirically.
  arXiv  Detail & Related papers  (2024-09-27T05:06:43Z)
• Fine-tuning Large Language Models for Entity Matching [3.7277730514654555]
  Generative large language models (LLMs) are a promising alternative to pre-trained language models for entity matching. This paper explores the potential of fine-tuning LLMs for entity matching.
  arXiv  Detail & Related papers  (2024-09-12T16:20:57Z)
• BLADE: Enhancing Black-box Large Language Models with Small Domain-Specific Models [56.89958793648104]
  Large Language Models (LLMs) are versatile and capable of addressing a diverse range of tasks. Previous approaches either conduct continuous pre-training with domain-specific data or employ retrieval augmentation to support general LLMs. We present a novel framework named BLADE, which enhances Black-box LArge language models with small Domain-spEcific models.
  arXiv  Detail & Related papers  (2024-03-27T08:57:21Z)
• PANDA: Preference Adaptation for Enhancing Domain-Specific Abilities of LLMs [49.32067576992511]
  Large language models often fall short of the performance achieved by domain-specific state-of-the-art models. One potential approach to enhance domain-specific capabilities of LLMs involves fine-tuning them using corresponding datasets. We propose Preference Adaptation for Enhancing Domain-specific Abilities of LLMs (PANDA) Our experimental results reveal that PANDA significantly enhances the domain-specific ability of LLMs on text classification and interactive decision tasks.
  arXiv  Detail & Related papers  (2024-02-20T09:02:55Z)
• Improving Machine Translation with Large Language Models: A Preliminary Study with Cooperative Decoding [73.32763904267186]
  Large Language Models (LLMs) present the potential for achieving superior translation quality. We propose Cooperative Decoding (CoDec) which treats NMT systems as a pretranslation model and MT-oriented LLMs as a supplemental solution.
  arXiv  Detail & Related papers  (2023-11-06T03:41:57Z)
• Simultaneous Machine Translation with Large Language Models [51.470478122113356]
  We investigate the possibility of applying Large Language Models to SimulMT tasks. We conducted experiments using the textttLlama2-7b-chat model on nine different languages from the MUST-C dataset. The results show that LLM outperforms dedicated MT models in terms of BLEU and LAAL metrics.
  arXiv  Detail & Related papers  (2023-09-13T04:06:47Z)
• A Paradigm Shift: The Future of Machine Translation Lies with Large Language Models [55.42263732351375]
  Machine Translation has greatly advanced over the years due to the developments in deep neural networks. The emergence of Large Language Models (LLMs) like GPT-4 and ChatGPT is introducing a new phase in the MT domain. We highlight several new MT directions, emphasizing the benefits of LLMs in scenarios such as Long-Document Translation, Stylized Translation, and Interactive Translation.
  arXiv  Detail & Related papers  (2023-05-02T03:27:27Z)
• Dictionary-based Phrase-level Prompting of Large Language Models for Machine Translation [91.57514888410205]
  Large language models (LLMs) demonstrate remarkable machine translation (MT) abilities via prompting. LLMs can struggle to translate inputs with rare words, which are common in low resource or domain transfer scenarios. We show that LLM prompting can provide an effective solution for rare words as well, by using prior knowledge from bilingual dictionaries to provide control hints in the prompts.
  arXiv  Detail & Related papers  (2023-02-15T18:46:42Z)
• Adaptive Machine Translation with Large Language Models [7.803471587734353]
  We investigate how we can utilize in-context learning to improve real-time adaptive machine translation. We conduct experiments across five diverse language pairs, namely English-to-Arabic (EN-AR), English-to-Chinese (EN-ZH), English-to-French (EN-FR), English-to-Kinyarwanda (EN-RW), and English-to-Spanish (EN-ES)
  arXiv  Detail & Related papers  (2023-01-30T21:17:15Z)
• Pre-trained Language Models for Keyphrase Generation: A Thorough Empirical Study [76.52997424694767]
  We present an in-depth empirical study of keyphrase extraction and keyphrase generation using pre-trained language models. We show that PLMs have competitive high-resource performance and state-of-the-art low-resource performance. Further results show that in-domain BERT-like PLMs can be used to build strong and data-efficient keyphrase generation models.
  arXiv  Detail & Related papers  (2022-12-20T13:20:21Z)
• Multi-Stage Pre-training for Low-Resource Domain Adaptation [24.689862495171408]
  Current approaches directly adapt a pre-trained language model (LM) on in-domain text before fine-tuning to downstream tasks. We show that extending the vocabulary of the LM with domain-specific terms leads to further gains. We apply these approaches incrementally on a pre-trained Roberta-large LM and show considerable performance gain on three tasks in the IT domain.
  arXiv  Detail & Related papers  (2020-10-12T17:57:00Z)
• A Simple Baseline to Semi-Supervised Domain Adaptation for Machine Translation [73.3550140511458]
  State-of-the-art neural machine translation (NMT) systems are data-hungry and perform poorly on new domains with no supervised data. We propose a simple but effect approach to the semi-supervised domain adaptation scenario of NMT. This approach iteratively trains a Transformer-based NMT model via three training objectives: language modeling, back-translation, and supervised translation.
  arXiv  Detail & Related papers  (2020-01-22T16:42:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.