Transformer and Hybrid Deep Learning Based Models for Machine-Generated Text Detection

• URL: http://arxiv.org/abs/2405.17964v1
• Date: Tue, 28 May 2024 08:48:08 GMT
• Title: Transformer and Hybrid Deep Learning Based Models for Machine-Generated Text Detection
• Authors: Teodor-George Marchitan, Claudiu Creanga, Liviu P. Dinu,
• Abstract summary: This paper describes the approach of the UniBuc - NLP team in tackling the SemEval 2024 Task 8: Multigenerator, Multidomain, and Multilingual Black-Box Machine-Generated Text Detection.
• Score: 4.373647283459287
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: This paper describes the approach of the UniBuc - NLP team in tackling the SemEval 2024 Task 8: Multigenerator, Multidomain, and Multilingual Black-Box Machine-Generated Text Detection. We explored transformer-based and hybrid deep learning architectures. For subtask B, our transformer-based model achieved a strong \textbf{second-place} out of $77$ teams with an accuracy of \textbf{86.95\%}, demonstrating the architecture's suitability for this task. However, our models showed overfitting in subtask A which could potentially be fixed with less fine-tunning and increasing maximum sequence length. For subtask C (token-level classification), our hybrid model overfit during training, hindering its ability to detect transitions between human and machine-generated text.

Related papers

• AISPACE at SemEval-2024 task 8: A Class-balanced Soft-voting System for Detecting Multi-generator Machine-generated Text [0.0]
  SemEval-2024 Task 8 provides a challenge to detect human-written and machine-generated text. This paper proposes a system that mainly deals with Subtask B. It aims to detect if given full text is written by human or is generated by a specific Large Language Model (LLM), which is actually a multi-class text classification task.
  arXiv  Detail & Related papers  (2024-04-01T06:25:47Z)
• AMOM: Adaptive Masking over Masking for Conditional Masked Language Model [81.55294354206923]
  A conditional masked language model (CMLM) is one of the most versatile frameworks. We introduce a simple yet effective adaptive masking over masking strategy to enhance the refinement capability of the decoder. Our proposed model yields state-of-the-art performance on neural machine translation.
  arXiv  Detail & Related papers  (2023-03-13T20:34:56Z)
• Adapted Multimodal BERT with Layer-wise Fusion for Sentiment Analysis [84.12658971655253]
  We propose Adapted Multimodal BERT, a BERT-based architecture for multimodal tasks. adapter adjusts the pretrained language model for the task at hand, while the fusion layers perform task-specific, layer-wise fusion of audio-visual information with textual BERT representations. In our ablations we see that this approach leads to efficient models, that can outperform their fine-tuned counterparts and are robust to input noise.
  arXiv  Detail & Related papers  (2022-12-01T17:31:42Z)
• Learning Multiscale Transformer Models for Sequence Generation [33.73729074207944]
  We build a multiscale Transformer model by establishing relationships among scales based on word-boundary information and phrase-level prior knowledge. Notably, it yielded consistent performance gains over the strong baseline on several test sets without sacrificing the efficiency.
  arXiv  Detail & Related papers  (2022-06-19T07:28:54Z)
• HETFORMER: Heterogeneous Transformer with Sparse Attention for Long-Text Extractive Summarization [57.798070356553936]
  HETFORMER is a Transformer-based pre-trained model with multi-granularity sparse attentions for extractive summarization. Experiments on both single- and multi-document summarization tasks show that HETFORMER achieves state-of-the-art performance in Rouge F1.
  arXiv  Detail & Related papers  (2021-10-12T22:42:31Z)
• Visual Saliency Transformer [127.33678448761599]
  We develop a novel unified model based on a pure transformer, Visual Saliency Transformer (VST), for both RGB and RGB-D salient object detection (SOD) It takes image patches as inputs and leverages the transformer to propagate global contexts among image patches. Experimental results show that our model outperforms existing state-of-the-art results on both RGB and RGB-D SOD benchmark datasets.
  arXiv  Detail & Related papers  (2021-04-25T08:24:06Z)
• Unsupervised Paraphrasing with Pretrained Language Models [85.03373221588707]
  We propose a training pipeline that enables pre-trained language models to generate high-quality paraphrases in an unsupervised setting. Our recipe consists of task-adaptation, self-supervision, and a novel decoding algorithm named Dynamic Blocking. We show with automatic and human evaluations that our approach achieves state-of-the-art performance on both the Quora Question Pair and the ParaNMT datasets.
  arXiv  Detail & Related papers  (2020-10-24T11:55:28Z)
• Deep Transformers with Latent Depth [42.33955275626127]
  The Transformer model has achieved state-of-the-art performance in many sequence modeling tasks. We present a probabilistic framework to automatically learn which layer(s) to use by learning the posterior distributions of layer selection. We propose a novel method to train one shared Transformer network for multilingual machine translation.
  arXiv  Detail & Related papers  (2020-09-28T07:13:23Z)
• Robust Conversational AI with Grounded Text Generation [77.56950706340767]
  GTG is a hybrid model which uses a large-scale Transformer neural network as its backbone. It generates responses grounded in dialog belief state and real-world knowledge for task completion.
  arXiv  Detail & Related papers  (2020-09-07T23:49:28Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.