Paraphrase Detection: Human vs. Machine Content
- URL: http://arxiv.org/abs/2303.13989v1
- Date: Fri, 24 Mar 2023 13:25:46 GMT
- Title: Paraphrase Detection: Human vs. Machine Content
- Authors: Jonas Becker and Jan Philip Wahle and Terry Ruas and Bela Gipp
- Abstract summary: Human-authored paraphrases exceed machine-generated ones in terms of difficulty, diversity, and similarity.
Transformers emerged as the most effective method across datasets with TF-IDF excelling on semantically diverse corpora.
- Score: 3.8768839735240737
- License: http://creativecommons.org/licenses/by-sa/4.0/
- Abstract: The growing prominence of large language models, such as GPT-4 and ChatGPT,
has led to increased concerns over academic integrity due to the potential for
machine-generated content and paraphrasing. Although studies have explored the
detection of human- and machine-paraphrased content, the comparison between
these types of content remains underexplored. In this paper, we conduct a
comprehensive analysis of various datasets commonly employed for paraphrase
detection tasks and evaluate an array of detection methods. Our findings
highlight the strengths and limitations of different detection methods in terms
of performance on individual datasets, revealing a lack of suitable
machine-generated datasets that can be aligned with human expectations. Our
main finding is that human-authored paraphrases exceed machine-generated ones
in terms of difficulty, diversity, and similarity implying that automatically
generated texts are not yet on par with human-level performance. Transformers
emerged as the most effective method across datasets with TF-IDF excelling on
semantically diverse corpora. Additionally, we identify four datasets as the
most diverse and challenging for paraphrase detection.
Related papers
- Detecting AI-Generated Text: Factors Influencing Detectability with Current Methods [13.14749943120523]
Knowing whether a text was produced by human or artificial intelligence (AI) is important to determining its trustworthiness.
State-of-the art approaches to AIGT detection include watermarking, statistical and stylistic analysis, and machine learning classification.
We aim to provide insight into the salient factors that combine to determine how "detectable" AIGT text is under different scenarios.
arXiv Detail & Related papers (2024-06-21T18:31:49Z) - MUGC: Machine Generated versus User Generated Content Detection [1.6602942962521352]
We show that traditional methods demonstrate a high level of accuracy in identifying machine-generated data.
Machine-generated texts tend to be shorter and exhibit less word variety compared to human-generated content.
readability, bias, moral, and affect comparisons reveal a discernible contrast between machine-generated and human generated content.
arXiv Detail & Related papers (2024-03-28T07:33:53Z) - Beyond Turing: A Comparative Analysis of Approaches for Detecting Machine-Generated Text [1.919654267936118]
Traditional shallow learning, Language Model (LM) fine-tuning, and Multilingual Model fine-tuning are evaluated.
Results reveal considerable differences in performance across methods.
This study paves the way for future research aimed at creating robust and highly discriminative models.
arXiv Detail & Related papers (2023-11-21T06:23:38Z) - DEMASQ: Unmasking the ChatGPT Wordsmith [63.8746084667206]
We propose an effective ChatGPT detector named DEMASQ, which accurately identifies ChatGPT-generated content.
Our method addresses two critical factors: (i) the distinct biases in text composition observed in human- and machine-generated content and (ii) the alterations made by humans to evade previous detection methods.
arXiv Detail & Related papers (2023-11-08T21:13:05Z) - The Imitation Game: Detecting Human and AI-Generated Texts in the Era of
ChatGPT and BARD [3.2228025627337864]
We introduce a novel dataset of human-written and AI-generated texts in different genres.
We employ several machine learning models to classify the texts.
Results demonstrate the efficacy of these models in discerning between human and AI-generated text.
arXiv Detail & Related papers (2023-07-22T21:00:14Z) - On the Possibilities of AI-Generated Text Detection [76.55825911221434]
We argue that as machine-generated text approximates human-like quality, the sample size needed for detection bounds increases.
We test various state-of-the-art text generators, including GPT-2, GPT-3.5-Turbo, Llama, Llama-2-13B-Chat-HF, and Llama-2-70B-Chat-HF, against detectors, including oBERTa-Large/Base-Detector, GPTZero.
arXiv Detail & Related papers (2023-04-10T17:47:39Z) - MGTBench: Benchmarking Machine-Generated Text Detection [54.81446366272403]
This paper proposes the first benchmark framework for MGT detection against powerful large language models (LLMs)
We show that a larger number of words in general leads to better performance and most detection methods can achieve similar performance with much fewer training samples.
Our findings indicate that the model-based detection methods still perform well in the text attribution task.
arXiv Detail & Related papers (2023-03-26T21:12:36Z) - MetaGraspNet: A Large-Scale Benchmark Dataset for Vision-driven Robotic
Grasping via Physics-based Metaverse Synthesis [78.26022688167133]
We present a large-scale benchmark dataset for vision-driven robotic grasping via physics-based metaverse synthesis.
The proposed dataset contains 100,000 images and 25 different object types.
We also propose a new layout-weighted performance metric alongside the dataset for evaluating object detection and segmentation performance.
arXiv Detail & Related papers (2021-12-29T17:23:24Z) - Human-in-the-Loop Disinformation Detection: Stance, Sentiment, or
Something Else? [93.91375268580806]
Both politics and pandemics have recently provided ample motivation for the development of machine learning-enabled disinformation (a.k.a. fake news) detection algorithms.
Existing literature has focused primarily on the fully-automated case, but the resulting techniques cannot reliably detect disinformation on the varied topics, sources, and time scales required for military applications.
By leveraging an already-available analyst as a human-in-the-loop, canonical machine learning techniques of sentiment analysis, aspect-based sentiment analysis, and stance detection become plausible methods to use for a partially-automated disinformation detection system.
arXiv Detail & Related papers (2021-11-09T13:30:34Z) - MGD-GAN: Text-to-Pedestrian generation through Multi-Grained
Discrimination [96.91091607251526]
We propose the Multi-Grained Discrimination enhanced Generative Adversarial Network, that capitalizes a human-part-based Discriminator and a self-cross-attended Discriminator.
A fine-grained word-level attention mechanism is employed in the HPD module to enforce diversified appearance and vivid details.
The substantial improvement over the various metrics demonstrates the efficacy of MGD-GAN on the text-to-pedestrian synthesis scenario.
arXiv Detail & Related papers (2020-10-02T12:24:48Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.