Gradable ChatGPT Translation Evaluation

• URL: http://arxiv.org/abs/2401.09984v2
• Date: Tue, 4 Jun 2024 06:32:04 GMT
• Title: Gradable ChatGPT Translation Evaluation
• Authors: Hui Jiao, Bei Peng, Lu Zong, Xiaojun Zhang, Xinwei Li,
• Abstract summary: ChatGPT, as a language model based on large-scale pre-training, has a profound influence on the domain of machine translation. The design of the translation prompt emerges as a key aspect that can wield influence over factors such as the style, precision and accuracy of the translation to a certain extent. This paper proposes a generic taxonomy, which defines gradable translation prompts in terms of expression type, translation style, POS information and explicit statement.
• Score: 7.697698018200632
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: ChatGPT, as a language model based on large-scale pre-training, has exerted a profound influence on the domain of machine translation. In ChatGPT, a "Prompt" refers to a segment of text or instruction employed to steer the model towards generating a specific category of response. The design of the translation prompt emerges as a key aspect that can wield influence over factors such as the style, precision and accuracy of the translation to a certain extent. However, there is a lack of a common standard and methodology on how to design and select a translation prompt. Accordingly, this paper proposes a generic taxonomy, which defines gradable translation prompts in terms of expression type, translation style, POS information and explicit statement, thus facilitating the construction of prompts endowed with distinct attributes tailored for various translation tasks. Specific experiments and cases are selected to validate and illustrate the effectiveness of the method.

Related papers

• Prosody in Cascade and Direct Speech-to-Text Translation: a case study on Korean Wh-Phrases [79.07111754406841]
  This work proposes using contrastive evaluation to measure the ability of direct S2TT systems to disambiguate utterances where prosody plays a crucial role. Our results clearly demonstrate the value of direct translation systems over cascade translation models.
  arXiv  Detail & Related papers  (2024-02-01T14:46:35Z)
• Harnessing GPT-3.5-turbo for Rhetorical Role Prediction in Legal Cases [0.16385815610837165]
  We propose a comprehensive study of one-stage elicitation techniques for querying a large pre-trained generative transformer (GPT-3.5-turbo) in the rhetorical role prediction task of legal cases. We show that the number of examples, the definition of labels, the presentation of the (labelled) textual context and specific questions about this context have a positive influence on the performance of the model.
  arXiv  Detail & Related papers  (2023-10-26T14:19:48Z)
• Optimizing Machine Translation through Prompt Engineering: An Investigation into ChatGPT's Customizability [0.0]
  The study reveals that the inclusion of suitable prompts in large-scale language models like ChatGPT can yield flexible translations. The research scrutinizes the changes in translation quality when prompts are used to generate translations that meet specific conditions.
  arXiv  Detail & Related papers  (2023-08-02T19:11:04Z)
• MetricPrompt: Prompting Model as a Relevance Metric for Few-shot Text Classification [65.51149771074944]
  MetricPrompt eases verbalizer design difficulty by reformulating few-shot text classification task into text pair relevance estimation task. We conduct experiments on three widely used text classification datasets across four few-shot settings. Results show that MetricPrompt outperforms manual verbalizer and other automatic verbalizer design methods across all few-shot settings.
  arXiv  Detail & Related papers  (2023-06-15T06:51:35Z)
• TIGTEC : Token Importance Guided TExt Counterfactuals [1.1642121991499805]
  This paper proposes TIGTEC, an efficient and modular method for generating sparse, plausible and diverse counterfactual explanations. A new attention-based local feature importance is proposed. Experiments show the relevance of TIGTEC in terms of success rate, sparsity, diversity and plausibility.
  arXiv  Detail & Related papers  (2023-04-24T20:11:58Z)
• Linguistically Informed ChatGPT Prompts to Enhance Japanese-Chinese Machine Translation: A Case Study on Attributive Clauses [0.0]
  This paper investigates the issue of correctly translating attributive clauses from Japanese to Chinese. A pre-edit scheme is proposed, which aims to enhance the accuracy of translation. A novel two-step prompt strategy is proposed, which has been demonstrated to improve the average translation accuracy score by over 35%.
  arXiv  Detail & Related papers  (2023-03-27T20:33:40Z)
• Prompting Large Language Model for Machine Translation: A Case Study [87.88120385000666]
  We offer a systematic study on prompting strategies for machine translation. We examine factors for prompt template and demonstration example selection. We explore the use of monolingual data and the feasibility of cross-lingual, cross-domain, and sentence-to-document transfer learning.
  arXiv  Detail & Related papers  (2023-01-17T18:32:06Z)
• Multilingual Extraction and Categorization of Lexical Collocations with Graph-aware Transformers [86.64972552583941]
  We put forward a sequence tagging BERT-based model enhanced with a graph-aware transformer architecture, which we evaluate on the task of collocation recognition in context. Our results suggest that explicitly encoding syntactic dependencies in the model architecture is helpful, and provide insights on differences in collocation typification in English, Spanish and French.
  arXiv  Detail & Related papers  (2022-05-23T16:47:37Z)
• Knowledgeable Prompt-tuning: Incorporating Knowledge into Prompt Verbalizer for Text Classification [68.3291372168167]
  We focus on incorporating external knowledge into the verbalizer, forming a knowledgeable prompt-tuning (KPT) We expand the label word space of the verbalizer using external knowledge bases (KBs) and refine the expanded label word space with the PLM itself before predicting with the expanded label word space. Experiments on zero and few-shot text classification tasks demonstrate the effectiveness of knowledgeable prompt-tuning.
  arXiv  Detail & Related papers  (2021-08-04T13:00:16Z)
• Deep Subjecthood: Higher-Order Grammatical Features in Multilingual BERT [7.057643880514415]
  We investigate how Multilingual BERT (mBERT) encodes grammar by examining how the high-order grammatical feature of morphosyntactic alignment is manifested across the embedding spaces of different languages.
  arXiv  Detail & Related papers  (2021-01-26T19:21:59Z)
• On the Importance of Word Order Information in Cross-lingual Sequence Labeling [80.65425412067464]
  Cross-lingual models that fit into the word order of the source language might fail to handle target languages. We investigate whether making models insensitive to the word order of the source language can improve the adaptation performance in target languages.
  arXiv  Detail & Related papers  (2020-01-30T03:35:44Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.