Towards Making the Most of ChatGPT for Machine Translation
- URL: http://arxiv.org/abs/2303.13780v4
- Date: Fri, 20 Oct 2023 07:18:15 GMT
- Title: Towards Making the Most of ChatGPT for Machine Translation
- Authors: Keqin Peng, Liang Ding, Qihuang Zhong, Li Shen, Xuebo Liu, Min Zhang,
Yuanxin Ouyang, Dacheng Tao
- Abstract summary: ChatGPT shows remarkable capabilities for machine translation (MT)
Several prior studies have shown that it achieves comparable results to commercial systems for high-resource languages.
- Score: 75.576405098545
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: ChatGPT shows remarkable capabilities for machine translation (MT). Several
prior studies have shown that it achieves comparable results to commercial
systems for high-resource languages, but lags behind in complex tasks, e.g.,
low-resource and distant-language-pairs translation. However, they usually
adopt simple prompts which can not fully elicit the capability of ChatGPT. In
this paper, we aim to further mine ChatGPT's translation ability by revisiting
several aspects: temperature, task information, and domain information, and
correspondingly propose an optimal temperature setting and two (simple but
effective) prompts: Task-Specific Prompts (TSP) and Domain-Specific Prompts
(DSP). We show that: 1) The performance of ChatGPT depends largely on
temperature, and a lower temperature usually can achieve better performance; 2)
Emphasizing the task information can further improve ChatGPT's performance,
particularly in complex MT tasks; 3) Introducing domain information can elicit
ChatGPT's generalization ability and improve its performance in the specific
domain; 4) ChatGPT tends to generate hallucinations for non-English-centric MT
tasks, which can be partially addressed by our proposed prompts but still need
to be highlighted for the MT/NLP community. We also explore the effects of
advanced in-context learning strategies and find a (negative but interesting)
observation: the powerful chain-of-thought prompt leads to word-by-word
translation behavior, thus bringing significant translation degradation.
Related papers
- (Chat)GPT v BERT: Dawn of Justice for Semantic Change Detection [1.9226023650048942]
Transformer-based language models like BERT and (Chat)GPT have emerged as lexical superheroes with great power to solve open research problems.
We evaluate their ability to solve two diachronic extensions of the Word-in-Context (WiC) task: TempoWiC and HistoWiC.
arXiv Detail & Related papers (2024-01-25T09:36:58Z) - Exploring ChatGPT's Capabilities on Vulnerability Management [56.4403395100589]
We explore ChatGPT's capabilities on 6 tasks involving the complete vulnerability management process with a large-scale dataset containing 70,346 samples.
One notable example is ChatGPT's proficiency in tasks like generating titles for software bug reports.
Our findings reveal the difficulties encountered by ChatGPT and shed light on promising future directions.
arXiv Detail & Related papers (2023-11-11T11:01:13Z) - ChatGPT Beyond English: Towards a Comprehensive Evaluation of Large
Language Models in Multilingual Learning [70.57126720079971]
Large language models (LLMs) have emerged as the most important breakthroughs in natural language processing (NLP)
This paper evaluates ChatGPT on 7 different tasks, covering 37 diverse languages with high, medium, low, and extremely low resources.
Compared to the performance of previous models, our extensive experimental results demonstrate a worse performance of ChatGPT for different NLP tasks and languages.
arXiv Detail & Related papers (2023-04-12T05:08:52Z) - How to Design Translation Prompts for ChatGPT: An Empirical Study [18.678893287863033]
ChatGPT has demonstrated surprising abilities in natural language understanding and natural language generation.
We adopt several translation prompts on a wide range of translations.
Our work provides empirical evidence that ChatGPT still has great potential in translations.
arXiv Detail & Related papers (2023-04-05T01:17:59Z) - Can ChatGPT Understand Too? A Comparative Study on ChatGPT and
Fine-tuned BERT [103.57103957631067]
ChatGPT has attracted great attention, as it can generate fluent and high-quality responses to human inquiries.
We evaluate ChatGPT's understanding ability by evaluating it on the most popular GLUE benchmark, and comparing it with 4 representative fine-tuned BERT-style models.
We find that: 1) ChatGPT falls short in handling paraphrase and similarity tasks; 2) ChatGPT outperforms all BERT models on inference tasks by a large margin; 3) ChatGPT achieves comparable performance compared with BERT on sentiment analysis and question answering tasks.
arXiv Detail & Related papers (2023-02-19T12:29:33Z) - Is ChatGPT a General-Purpose Natural Language Processing Task Solver? [113.22611481694825]
Large language models (LLMs) have demonstrated the ability to perform a variety of natural language processing (NLP) tasks zero-shot.
Recently, the debut of ChatGPT has drawn a great deal of attention from the natural language processing (NLP) community.
It is not yet known whether ChatGPT can serve as a generalist model that can perform many NLP tasks zero-shot.
arXiv Detail & Related papers (2023-02-08T09:44:51Z) - Is ChatGPT A Good Translator? Yes With GPT-4 As The Engine [97.8609714773255]
We evaluate ChatGPT for machine translation, including translation prompt, multilingual translation, and translation robustness.
ChatGPT performs competitively with commercial translation products but lags behind significantly on low-resource or distant languages.
With the launch of the GPT-4 engine, the translation performance of ChatGPT is significantly boosted.
arXiv Detail & Related papers (2023-01-20T08:51:36Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.