ChatGPT to Replace Crowdsourcing of Paraphrases for Intent Classification: Higher Diversity and Comparable Model Robustness

• URL: http://arxiv.org/abs/2305.12947v2
• Date: Thu, 19 Oct 2023 21:11:53 GMT
• Title: ChatGPT to Replace Crowdsourcing of Paraphrases for Intent Classification: Higher Diversity and Comparable Model Robustness
• Authors: Jan Cegin, Jakub Simko and Peter Brusilovsky
• Abstract summary: We show that ChatGPT-created paraphrases are more diverse and lead to at least as robust models. Traditionally, crowdsourcing has been used for acquiring solutions to a wide variety of human-intelligence tasks.
• Score: 3.126776200660494
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The emergence of generative large language models (LLMs) raises the question: what will be its impact on crowdsourcing? Traditionally, crowdsourcing has been used for acquiring solutions to a wide variety of human-intelligence tasks, including ones involving text generation, modification or evaluation. For some of these tasks, models like ChatGPT can potentially substitute human workers. In this study, we investigate whether this is the case for the task of paraphrase generation for intent classification. We apply data collection methodology of an existing crowdsourcing study (similar scale, prompts and seed data) using ChatGPT and Falcon-40B. We show that ChatGPT-created paraphrases are more diverse and lead to at least as robust models.

Related papers

• Human-interpretable clustering of short-text using large language models [0.0]
  We show that large language models can be used to cluster human-generated content. This success is validated by both human reviewers and ChatGPT.
  arXiv  Detail & Related papers  (2024-05-12T12:55:40Z)
• Retrieval is Accurate Generation [99.24267226311157]
  We introduce a novel method that selects context-aware phrases from a collection of supporting documents. Our model achieves the best performance and the lowest latency among several retrieval-augmented baselines.
  arXiv  Detail & Related papers  (2024-02-27T14:16:19Z)
• Paraphrase Types for Generation and Detection [7.800428507692341]
  We name these tasks Paraphrase Type Generation and Paraphrase Type Detection. Our results suggest that while current techniques perform well in a binary classification scenario, the inclusion of fine-grained paraphrase types poses a significant challenge. We believe paraphrase types can unlock a new paradigm for developing paraphrase models and solving tasks in the future.
  arXiv  Detail & Related papers  (2023-10-23T12:32:41Z)
• A Quality-based Syntactic Template Retriever for Syntactically-controlled Paraphrase Generation [67.98367574025797]
  Existing syntactically-controlled paraphrase generation models perform promisingly with human-annotated or well-chosen syntactic templates. The prohibitive cost makes it unfeasible to manually design decent templates for every source sentence. We propose a novel Quality-based Syntactic Template Retriever (QSTR) to retrieve templates based on the quality of the to-be-generated paraphrases.
  arXiv  Detail & Related papers  (2023-10-20T03:55:39Z)
• On the Generalization of Training-based ChatGPT Detection Methods [33.46128880100525]
  ChatGPT is one of the most popular language models which achieve amazing performance on various natural language tasks. There is also an urgent need to detect the texts generated ChatGPT from human written.
  arXiv  Detail & Related papers  (2023-10-02T16:13:08Z)
• ChatGPT vs State-of-the-Art Models: A Benchmarking Study in Keyphrase Generation Task [0.0]
  Transformer-based language models, including ChatGPT, have demonstrated exceptional performance in various natural language generation tasks. This study compares ChatGPT's keyphrase generation performance with state-of-the-art models, while also testing its potential as a solution for two significant challenges in the field.
  arXiv  Detail & Related papers  (2023-04-27T13:25:43Z)
• ChatGPT Beyond English: Towards a Comprehensive Evaluation of Large Language Models in Multilingual Learning [70.57126720079971]
  Large language models (LLMs) have emerged as the most important breakthroughs in natural language processing (NLP) This paper evaluates ChatGPT on 7 different tasks, covering 37 diverse languages with high, medium, low, and extremely low resources. Compared to the performance of previous models, our extensive experimental results demonstrate a worse performance of ChatGPT for different NLP tasks and languages.
  arXiv  Detail & Related papers  (2023-04-12T05:08:52Z)
• Beyond Contrastive Learning: A Variational Generative Model for Multilingual Retrieval [109.62363167257664]
  We propose a generative model for learning multilingual text embeddings. Our model operates on parallel data in $N$ languages. We evaluate this method on a suite of tasks including semantic similarity, bitext mining, and cross-lingual question retrieval.
  arXiv  Detail & Related papers  (2022-12-21T02:41:40Z)
• Elaboration-Generating Commonsense Question Answering at Scale [77.96137534751445]
  In question answering requiring common sense, language models (e.g., GPT-3) have been used to generate text expressing background knowledge. We finetune smaller language models to generate useful intermediate context, referred to here as elaborations. Our framework alternates between updating two language models -- an elaboration generator and an answer predictor -- allowing each to influence the other.
  arXiv  Detail & Related papers  (2022-09-02T18:32:09Z)
• Polling Latent Opinions: A Method for Computational Sociolinguistics Using Transformer Language Models [4.874780144224057]
  We use the capacity for memorization and extrapolation of Transformer Language Models to learn the linguistic behaviors of a subgroup within larger corpora of Yelp reviews. We show that even in cases where a specific keyphrase is limited or not present at all in the training corpora, the GPT is able to accurately generate large volumes of text that have the correct sentiment.
  arXiv  Detail & Related papers  (2022-04-15T14:33:58Z)
• Pushing Paraphrase Away from Original Sentence: A Multi-Round Paraphrase Generation Approach [97.38622477085188]
  We propose BTmPG (Back-Translation guided multi-round Paraphrase Generation) to improve diversity of paraphrase. We evaluate BTmPG on two benchmark datasets.
  arXiv  Detail & Related papers  (2021-09-04T13:12:01Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.