PetKaz at SemEval-2024 Task 8: Can Linguistics Capture the Specifics of LLM-generated Text?

• URL: http://arxiv.org/abs/2404.05483v1
• Date: Mon, 8 Apr 2024 13:05:02 GMT
• Title: PetKaz at SemEval-2024 Task 8: Can Linguistics Capture the Specifics of LLM-generated Text?
• Authors: Kseniia Petukhova, Roman Kazakov, Ekaterina Kochmar,
• Abstract summary: We present our submission to the SemEval-2024 Task 8 "Multigenerator, Multidomain, and Black-Box Machine-Generated Text Detection" Our approach relies on combining embeddings from the RoBERTa-base with diversity features and uses a resampled training set. Our results show that our approach is generalizable across unseen models and domains, achieving an accuracy of 0.91.
• Score: 4.463184061618504
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: In this paper, we present our submission to the SemEval-2024 Task 8 "Multigenerator, Multidomain, and Multilingual Black-Box Machine-Generated Text Detection", focusing on the detection of machine-generated texts (MGTs) in English. Specifically, our approach relies on combining embeddings from the RoBERTa-base with diversity features and uses a resampled training set. We score 12th from 124 in the ranking for Subtask A (monolingual track), and our results show that our approach is generalizable across unseen models and domains, achieving an accuracy of 0.91.

Related papers

• Sharif-MGTD at SemEval-2024 Task 8: A Transformer-Based Approach to Detect Machine Generated Text [2.2039952888743253]
  MGT has emerged as a significant area of study within Natural Language Processing. In this research, we explore the effectiveness of fine-tuning a RoBERTa-base transformer, a powerful neural architecture, to address MGT detection. Our proposed system achieves an accuracy of 78.9% on the test dataset, positioning us at 57th among participants.
  arXiv  Detail & Related papers  (2024-07-16T14:33:01Z)
• SemEval-2024 Task 8: Multidomain, Multimodel and Multilingual Machine-Generated Text Detection [68.858931667807]
  Subtask A is a binary classification task determining whether a text is written by a human or generated by a machine. Subtask B is to detect the exact source of a text, discerning whether it is written by a human or generated by a specific LLM. Subtask C aims to identify the changing point within a text, at which the authorship transitions from human to machine.
  arXiv  Detail & Related papers  (2024-04-22T13:56:07Z)
• TrustAI at SemEval-2024 Task 8: A Comprehensive Analysis of Multi-domain Machine Generated Text Detection Techniques [2.149586323955579]
  Large Language Models (LLMs) generate fluent content across a wide spectrum of user queries. This capability has raised concerns regarding misinformation and personal information leakage. We present our methods for the SemEval2024 Task8, aiming to detect machine-generated text across various domains.
  arXiv  Detail & Related papers  (2024-03-25T10:09:03Z)
• KInIT at SemEval-2024 Task 8: Fine-tuned LLMs for Multilingual Machine-Generated Text Detection [0.0]
  SemEval-2024 Task 8 is focused on multigenerator, multidomain, and multilingual black-box machine-generated text detection. Our submitted method achieved competitive results, ranking at the fourth place, just under 1 percentage point behind the winner.
  arXiv  Detail & Related papers  (2024-02-21T10:09:56Z)
• Team QUST at SemEval-2024 Task 8: A Comprehensive Study of Monolingual and Multilingual Approaches for Detecting AI-generated Text [0.1499944454332829]
  This paper presents the participation of team QUST in Task 8 SemEval 2024. We first performed data augmentation and cleaning on the dataset to enhance model training efficiency and accuracy. In the monolingual task, we evaluated traditional deep-learning methods, multiscale positive-unlabeled framework (MPU), fine-tuning, adapters and ensemble methods.
  arXiv  Detail & Related papers  (2024-02-19T08:22:51Z)
• Breaking Language Barriers in Multilingual Mathematical Reasoning: Insights and Observations [90.73517523001149]
  This paper pioneers exploring and training powerful Multilingual Math Reasoning (xMR) LLMs. By utilizing translation, we construct the first multilingual math reasoning instruction dataset, MGSM8KInstruct. We propose different training strategies to build powerful xMR LLMs, named MathOctopus, notably outperform conventional open-source LLMs.
  arXiv  Detail & Related papers  (2023-10-31T08:09:20Z)
• SEAHORSE: A Multilingual, Multifaceted Dataset for Summarization Evaluation [52.186343500576214]
  We introduce SEAHORSE, a dataset for multilingual, multifaceted summarization evaluation. SEAHORSE consists of 96K summaries with human ratings along 6 dimensions of text quality. We show that metrics trained with SEAHORSE achieve strong performance on the out-of-domain meta-evaluation benchmarks TRUE and mFACE.
  arXiv  Detail & Related papers  (2023-05-22T16:25:07Z)
• mCPT at SemEval-2023 Task 3: Multilingual Label-Aware Contrastive Pre-Training of Transformers for Few- and Zero-shot Framing Detection [63.540146992962526]
  This paper presents the winning system for the zero-shot Spanish framing detection task. Our solution employs a pre-training procedure based on multilingual Transformers. In addition to describing the system, we perform an embedding space analysis and ablation study to demonstrate how our pre-training procedure supports framing detection.
  arXiv  Detail & Related papers  (2023-03-17T11:33:06Z)
• Google USM: Scaling Automatic Speech Recognition Beyond 100 Languages [76.95115818308918]
  We introduce the Universal Speech Model (USM), a single large model that performs automatic speech recognition (ASR) across 100+ languages. This is achieved by pre-training the encoder of the model on a large unlabeled multilingual dataset of 12 million (M) hours spanning over 300 languages. We use multilingual pre-training with random-projection quantization and speech-text modality matching to achieve state-of-the-art performance on downstream multilingual ASR and speech-to-text translation tasks.
  arXiv  Detail & Related papers  (2023-03-02T07:47:18Z)
• Pre-training via Paraphrasing [96.79972492585112]
  We introduce MARGE, a pre-trained sequence-to-sequence model learned with an unsupervised multi-lingual paraphrasing objective. We show it is possible to jointly learn to do retrieval and reconstruction, given only a random initialization. For example, with no additional task-specific training we achieve BLEU scores of up to 35.8 for document translation.
  arXiv  Detail & Related papers  (2020-06-26T14:43:43Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.