Differentiate ChatGPT-generated and Human-written Medical Texts
- URL: http://arxiv.org/abs/2304.11567v1
- Date: Sun, 23 Apr 2023 07:38:07 GMT
- Title: Differentiate ChatGPT-generated and Human-written Medical Texts
- Authors: Wenxiong Liao, Zhengliang Liu, Haixing Dai, Shaochen Xu, Zihao Wu,
Yiyang Zhang, Xiaoke Huang, Dajiang Zhu, Hongmin Cai, Tianming Liu, Xiang Li
- Abstract summary: This research is among the first studies on responsible and ethical AIGC (Artificial Intelligence Generated Content) in medicine.
We focus on analyzing the differences between medical texts written by human experts and generated by ChatGPT.
In the next step, we analyze the linguistic features of these two types of content and uncover differences in vocabulary, part-of-speech, dependency, sentiment, perplexity, etc.
- Score: 8.53416950968806
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Background: Large language models such as ChatGPT are capable of generating
grammatically perfect and human-like text content, and a large number of
ChatGPT-generated texts have appeared on the Internet. However, medical texts
such as clinical notes and diagnoses require rigorous validation, and erroneous
medical content generated by ChatGPT could potentially lead to disinformation
that poses significant harm to healthcare and the general public.
Objective: This research is among the first studies on responsible and
ethical AIGC (Artificial Intelligence Generated Content) in medicine. We focus
on analyzing the differences between medical texts written by human experts and
generated by ChatGPT, and designing machine learning workflows to effectively
detect and differentiate medical texts generated by ChatGPT.
Methods: We first construct a suite of datasets containing medical texts
written by human experts and generated by ChatGPT. In the next step, we analyze
the linguistic features of these two types of content and uncover differences
in vocabulary, part-of-speech, dependency, sentiment, perplexity, etc. Finally,
we design and implement machine learning methods to detect medical text
generated by ChatGPT.
Results: Medical texts written by humans are more concrete, more diverse, and
typically contain more useful information, while medical texts generated by
ChatGPT pay more attention to fluency and logic, and usually express general
terminologies rather than effective information specific to the context of the
problem. A BERT-based model can effectively detect medical texts generated by
ChatGPT, and the F1 exceeds 95%.
Related papers
- DEMASQ: Unmasking the ChatGPT Wordsmith [63.8746084667206]
We propose an effective ChatGPT detector named DEMASQ, which accurately identifies ChatGPT-generated content.
Our method addresses two critical factors: (i) the distinct biases in text composition observed in human- and machine-generated content and (ii) the alterations made by humans to evade previous detection methods.
arXiv Detail & Related papers (2023-11-08T21:13:05Z) - Harnessing ChatGPT for thematic analysis: Are we ready? [0.0]
ChatGPT is an advanced natural language processing tool with growing applications across various disciplines in medical research.
This viewpoint explores the utilization of ChatGPT in three core phases of thematic analysis within a medical context.
arXiv Detail & Related papers (2023-10-23T03:55:13Z) - Detecting ChatGPT: A Survey of the State of Detecting ChatGPT-Generated
Text [1.9643748953805937]
generative language models can potentially deceive by generating artificial text that appears to be human-generated.
This survey provides an overview of the current approaches employed to differentiate between texts generated by humans and ChatGPT.
arXiv Detail & Related papers (2023-09-14T13:05:20Z) - Playing with Words: Comparing the Vocabulary and Lexical Richness of
ChatGPT and Humans [3.0059120458540383]
generative language models such as ChatGPT have triggered a revolution that can transform how text is generated.
Will the use of tools such as ChatGPT increase or reduce the vocabulary used or the lexical richness?
This has implications for words, as those not included in AI-generated content will tend to be less and less popular and may eventually be lost.
arXiv Detail & Related papers (2023-08-14T21:19:44Z) - Is ChatGPT Involved in Texts? Measure the Polish Ratio to Detect
ChatGPT-Generated Text [48.36706154871577]
We introduce a novel dataset termed HPPT (ChatGPT-polished academic abstracts)
It diverges from extant corpora by comprising pairs of human-written and ChatGPT-polished abstracts instead of purely ChatGPT-generated texts.
We also propose the "Polish Ratio" method, an innovative measure of the degree of modification made by ChatGPT compared to the original human-written text.
arXiv Detail & Related papers (2023-07-21T06:38:37Z) - ChatGPT for Us: Preserving Data Privacy in ChatGPT via Dialogue Text
Ambiguation to Expand Mental Health Care Delivery [52.73936514734762]
ChatGPT has gained popularity for its ability to generate human-like dialogue.
Data-sensitive domains face challenges in using ChatGPT due to privacy and data-ownership concerns.
We propose a text ambiguation framework that preserves user privacy.
arXiv Detail & Related papers (2023-05-19T02:09:52Z) - ChatGPT Beyond English: Towards a Comprehensive Evaluation of Large
Language Models in Multilingual Learning [70.57126720079971]
Large language models (LLMs) have emerged as the most important breakthroughs in natural language processing (NLP)
This paper evaluates ChatGPT on 7 different tasks, covering 37 diverse languages with high, medium, low, and extremely low resources.
Compared to the performance of previous models, our extensive experimental results demonstrate a worse performance of ChatGPT for different NLP tasks and languages.
arXiv Detail & Related papers (2023-04-12T05:08:52Z) - To ChatGPT, or not to ChatGPT: That is the question! [78.407861566006]
This study provides a comprehensive and contemporary assessment of the most recent techniques in ChatGPT detection.
We have curated a benchmark dataset consisting of prompts from ChatGPT and humans, including diverse questions from medical, open Q&A, and finance domains.
Our evaluation results demonstrate that none of the existing methods can effectively detect ChatGPT-generated content.
arXiv Detail & Related papers (2023-04-04T03:04:28Z) - DeID-GPT: Zero-shot Medical Text De-Identification by GPT-4 [80.36535668574804]
We develop a novel GPT4-enabled de-identification framework (DeID-GPT")
Our developed DeID-GPT showed the highest accuracy and remarkable reliability in masking private information from the unstructured medical text.
This study is one of the earliest to utilize ChatGPT and GPT-4 for medical text data processing and de-identification.
arXiv Detail & Related papers (2023-03-20T11:34:37Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.