ReMoDetect: Reward Models Recognize Aligned LLM's Generations

• URL: http://arxiv.org/abs/2405.17382v1
• Date: Mon, 27 May 2024 17:38:33 GMT
• Title: ReMoDetect: Reward Models Recognize Aligned LLM's Generations
• Authors: Hyunseok Lee, Jihoon Tack, Jinwoo Shin,
• Abstract summary: Large language models (LLMs) generate human-preferable texts. We propose two training schemes to further improve the detection ability of the reward model.
• Score: 55.06804460642062
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The remarkable capabilities and easy accessibility of large language models (LLMs) have significantly increased societal risks (e.g., fake news generation), necessitating the development of LLM-generated text (LGT) detection methods for safe usage. However, detecting LGTs is challenging due to the vast number of LLMs, making it impractical to account for each LLM individually; hence, it is crucial to identify the common characteristics shared by these models. In this paper, we draw attention to a common feature of recent powerful LLMs, namely the alignment training, i.e., training LLMs to generate human-preferable texts. Our key finding is that as these aligned LLMs are trained to maximize the human preferences, they generate texts with higher estimated preferences even than human-written texts; thus, such texts are easily detected by using the reward model (i.e., an LLM trained to model human preference distribution). Based on this finding, we propose two training schemes to further improve the detection ability of the reward model, namely (i) continual preference fine-tuning to make the reward model prefer aligned LGTs even further and (ii) reward modeling of Human/LLM mixed texts (a rephrased texts from human-written texts using aligned LLMs), which serves as a median preference text corpus between LGTs and human-written texts to learn the decision boundary better. We provide an extensive evaluation by considering six text domains across twelve aligned LLMs, where our method demonstrates state-of-the-art results. Code is available at https://github.com/hyunseoklee-ai/reward_llm_detect.

Related papers

• Hide and Seek: Fingerprinting Large Language Models with Evolutionary Learning [0.40964539027092917]
  We introduce a novel black-box approach for fingerprinting Large Language Model (LLM) models. We achieve an impressive 72% accuracy in identifying the correct family of models. This research opens new avenues for understanding LLM behavior and has significant implications for model attribution, security, and the broader field of AI transparency.
  arXiv  Detail & Related papers  (2024-08-06T00:13:10Z)
• DALD: Improving Logits-based Detector without Logits from Black-box LLMs [56.234109491884126]
  Large Language Models (LLMs) have revolutionized text generation, producing outputs that closely mimic human writing. We present Distribution-Aligned LLMs Detection (DALD), an innovative framework that redefines the state-of-the-art performance in black-box text detection. DALD is designed to align the surrogate model's distribution with that of unknown target LLMs, ensuring enhanced detection capability and resilience against rapid model iterations.
  arXiv  Detail & Related papers  (2024-06-07T19:38:05Z)
• SPOT: Text Source Prediction from Originality Score Thresholding [6.790905400046194]
  countermeasures aim at detecting misinformation, usually involve domain specific models trained to recognize the relevance of any information. Instead of evaluating the validity of the information, we propose to investigate LLM generated text from the perspective of trust.
  arXiv  Detail & Related papers  (2024-05-30T21:51:01Z)
• Generative Text Steganography with Large Language Model [10.572149957139736]
  Black-box generative text steganographic method based on user interfaces of large language models, which is called LLM-Stega. We first construct a keyword set and design a new encrypted steganographic mapping to embed secret messages. Comprehensive experiments demonstrate that the proposed LLM-Stega outperforms current state-of-the-art methods.
  arXiv  Detail & Related papers  (2024-04-16T02:19:28Z)
• Unsupervised Information Refinement Training of Large Language Models for Retrieval-Augmented Generation [128.01050030936028]
  We propose an information refinement training method named InFO-RAG. InFO-RAG is low-cost and general across various tasks. It improves the performance of LLaMA2 by an average of 9.39% relative points.
  arXiv  Detail & Related papers  (2024-02-28T08:24:38Z)
• LLM-Detector: Improving AI-Generated Chinese Text Detection with Open-Source LLM Instruction Tuning [4.328134379418151]
  Existing AI-generated text detection models are prone to in-domain over-fitting. We propose LLM-Detector, a novel method for both document-level and sentence-level text detection.
  arXiv  Detail & Related papers  (2024-02-02T05:54:12Z)
• A Survey on LLM-Generated Text Detection: Necessity, Methods, and Future Directions [39.36381851190369]
  There is an imperative need to develop detectors that can detect LLM-generated text. This is crucial to mitigate potential misuse of LLMs and safeguard realms like artistic expression and social networks from harmful influence of LLM-generated content. The detector techniques have witnessed notable advancements recently, propelled by innovations in watermarking techniques, statistics-based detectors, neural-base detectors, and human-assisted methods.
  arXiv  Detail & Related papers  (2023-10-23T09:01:13Z)
• Large Language Models can Contrastively Refine their Generation for Better Sentence Representation Learning [57.74233319453229]
  Large language models (LLMs) have emerged as a groundbreaking technology and their unparalleled text generation capabilities have sparked interest in their application to the fundamental sentence representation learning task. We propose MultiCSR, a multi-level contrastive sentence representation learning framework that decomposes the process of prompting LLMs to generate a corpus. Our experiments reveal that MultiCSR enables a less advanced LLM to surpass the performance of ChatGPT, while applying it to ChatGPT achieves better state-of-the-art results.
  arXiv  Detail & Related papers  (2023-10-17T03:21:43Z)
• Red Teaming Language Model Detectors with Language Models [114.36392560711022]
  Large language models (LLMs) present significant safety and ethical risks if exploited by malicious users. Recent works have proposed algorithms to detect LLM-generated text and protect LLMs. We study two types of attack strategies: 1) replacing certain words in an LLM's output with their synonyms given the context; 2) automatically searching for an instructional prompt to alter the writing style of the generation.
  arXiv  Detail & Related papers  (2023-05-31T10:08:37Z)
• On Learning to Summarize with Large Language Models as References [101.79795027550959]
  Large language models (LLMs) are favored by human annotators over the original reference summaries in commonly used summarization datasets. We study an LLM-as-reference learning setting for smaller text summarization models to investigate whether their performance can be substantially improved.
  arXiv  Detail & Related papers  (2023-05-23T16:56:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.