Towards Human Understanding of Paraphrase Types in ChatGPT

• URL: http://arxiv.org/abs/2407.02302v1
• Date: Tue, 2 Jul 2024 14:35:10 GMT
• Title: Towards Human Understanding of Paraphrase Types in ChatGPT
• Authors: Dominik Meier, Jan Philip Wahle, Terry Ruas, Bela Gipp,
• Abstract summary: Atomic paraphrase types (APT) decompose paraphrases into different linguistic changes. We introduce APTY (Atomic Paraphrase TYpes), a dataset of 500 sentence-level and word-level annotations by 15 annotators. Our results reveal that ChatGPT can generate simple APTs, but struggle with complex structures.
• Score: 7.662751948664846
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Paraphrases represent a human's intuitive ability to understand expressions presented in various different ways. Current paraphrase evaluations of language models primarily use binary approaches, offering limited interpretability of specific text changes. Atomic paraphrase types (APT) decompose paraphrases into different linguistic changes and offer a granular view of the flexibility in linguistic expression (e.g., a shift in syntax or vocabulary used). In this study, we assess the human preferences towards ChatGPT in generating English paraphrases with ten APTs and five prompting techniques. We introduce APTY (Atomic Paraphrase TYpes), a dataset of 500 sentence-level and word-level annotations by 15 annotators. The dataset also provides a human preference ranking of paraphrases with different types that can be used to fine-tune models with RLHF and DPO methods. Our results reveal that ChatGPT can generate simple APTs, such as additions and deletions, but struggle with complex structures (e.g., subordination changes). This study contributes to understanding which aspects of paraphrasing language models have already succeeded at understanding and what remains elusive. In addition, our curated datasets can be used to develop language models with specific linguistic capabilities.

Related papers

• Exploring syntactic information in sentence embeddings through multilingual subject-verb agreement [1.4335183427838039]
  We take the approach of developing curated synthetic data on a large scale, with specific properties. We use a new multiple-choice task and datasets, Blackbird Language Matrices, to focus on a specific grammatical structural phenomenon. We show that despite having been trained on multilingual texts in a consistent manner, multilingual pretrained language models have language-specific differences.
  arXiv  Detail & Related papers  (2024-09-10T14:58:55Z)
• Paraphrase Types Elicit Prompt Engineering Capabilities [9.311064293678154]
  This study systematically and empirically evaluates which linguistic features influence models through paraphrase types. We measure behavioral changes for five models across 120 tasks and six families of paraphrases. Our results show a potential for language models to improve tasks when their prompts are adapted in specific paraphrase types.
  arXiv  Detail & Related papers  (2024-06-28T13:06:31Z)
• A Taxonomy of Ambiguity Types for NLP [53.10379645698917]
  We propose a taxonomy of ambiguity types as seen in English to facilitate NLP analysis. Our taxonomy can help make meaningful splits in language ambiguity data, allowing for more fine-grained assessments of both datasets and model performance.
  arXiv  Detail & Related papers  (2024-03-21T01:47:22Z)
• Paraphrase Types for Generation and Detection [7.800428507692341]
  We name these tasks Paraphrase Type Generation and Paraphrase Type Detection. Our results suggest that while current techniques perform well in a binary classification scenario, the inclusion of fine-grained paraphrase types poses a significant challenge. We believe paraphrase types can unlock a new paradigm for developing paraphrase models and solving tasks in the future.
  arXiv  Detail & Related papers  (2023-10-23T12:32:41Z)
• Improving Mandarin Prosodic Structure Prediction with Multi-level Contextual Information [68.89000132126536]
  This work proposes to use inter-utterance linguistic information to improve the performance of prosodic structure prediction (PSP) Our method achieves better F1 scores in predicting prosodic word (PW), prosodic phrase (PPH) and intonational phrase (IPH)
  arXiv  Detail & Related papers  (2023-08-31T09:19:15Z)
• Playing with Words: Comparing the Vocabulary and Lexical Richness of ChatGPT and Humans [3.0059120458540383]
  generative language models such as ChatGPT have triggered a revolution that can transform how text is generated. Will the use of tools such as ChatGPT increase or reduce the vocabulary used or the lexical richness? This has implications for words, as those not included in AI-generated content will tend to be less and less popular and may eventually be lost.
  arXiv  Detail & Related papers  (2023-08-14T21:19:44Z)
• ParaAMR: A Large-Scale Syntactically Diverse Paraphrase Dataset by AMR Back-Translation [59.91139600152296]
  ParaAMR is a large-scale syntactically diverse paraphrase dataset created by abstract meaning representation back-translation. We show that ParaAMR can be used to improve on three NLP tasks: learning sentence embeddings, syntactically controlled paraphrase generation, and data augmentation for few-shot learning.
  arXiv  Detail & Related papers  (2023-05-26T02:27:33Z)
• Prompting Language Models for Linguistic Structure [73.11488464916668]
  We present a structured prompting approach for linguistic structured prediction tasks. We evaluate this approach on part-of-speech tagging, named entity recognition, and sentence chunking. We find that while PLMs contain significant prior knowledge of task labels due to task leakage into the pretraining corpus, structured prompting can also retrieve linguistic structure with arbitrary labels.
  arXiv  Detail & Related papers  (2022-11-15T01:13:39Z)
• Transparency Helps Reveal When Language Models Learn Meaning [71.96920839263457]
  Our systematic experiments with synthetic data reveal that, with languages where all expressions have context-independent denotations, both autoregressive and masked language models learn to emulate semantic relations between expressions. Turning to natural language, our experiments with a specific phenomenon -- referential opacity -- add to the growing body of evidence that current language models do not well-represent natural language semantics.
  arXiv  Detail & Related papers  (2022-10-14T02:35:19Z)
• Polling Latent Opinions: A Method for Computational Sociolinguistics Using Transformer Language Models [4.874780144224057]
  We use the capacity for memorization and extrapolation of Transformer Language Models to learn the linguistic behaviors of a subgroup within larger corpora of Yelp reviews. We show that even in cases where a specific keyphrase is limited or not present at all in the training corpora, the GPT is able to accurately generate large volumes of text that have the correct sentiment.
  arXiv  Detail & Related papers  (2022-04-15T14:33:58Z)
• Phonological Features for 0-shot Multilingual Speech Synthesis [50.591267188664666]
  We show that code-switching is possible for languages unseen during training, even within monolingual models. We generate intelligible, code-switched speech in a new language at test time, including the approximation of sounds never seen in training.
  arXiv  Detail & Related papers  (2020-08-06T18:25:18Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.