Can Large Language Models Identify Authorship?
- URL: http://arxiv.org/abs/2403.08213v1
- Date: Wed, 13 Mar 2024 03:22:02 GMT
- Title: Can Large Language Models Identify Authorship?
- Authors: Baixiang Huang, Canyu Chen, Kai Shu
- Abstract summary: Large Language Models (LLMs) have demonstrated exceptional capacity for reasoning and problem-solving.
This paper conducts a comprehensive evaluation of LLMs in authorship analysis.
- Score: 18.378744138365537
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: The ability to accurately identify authorship is crucial for verifying
content authenticity and mitigating misinformation. Large Language Models
(LLMs) have demonstrated exceptional capacity for reasoning and
problem-solving. However, their potential in authorship analysis, encompassing
authorship verification and attribution, remains underexplored. This paper
conducts a comprehensive evaluation of LLMs in these critical tasks.
Traditional studies have depended on hand-crafted stylistic features, whereas
state-of-the-art approaches leverage text embeddings from pre-trained language
models. These methods, which typically require fine-tuning on labeled data,
often suffer from performance degradation in cross-domain applications and
provide limited explainability. This work seeks to address three research
questions: (1) Can LLMs perform zero-shot, end-to-end authorship verification
effectively? (2) Are LLMs capable of accurately attributing authorship among
multiple candidates authors (e.g., 10 and 20)? (3) How can LLMs provide
explainability in authorship analysis, particularly through the role of
linguistic features? Moreover, we investigate the integration of explicit
linguistic features to guide LLMs in their reasoning processes. Our extensive
assessment demonstrates LLMs' proficiency in both tasks without the need for
domain-specific fine-tuning, providing insights into their decision-making via
a detailed analysis of linguistic features. This establishes a new benchmark
for future research on LLM-based authorship analysis. The code and data are
available at https://github.com/baixianghuang/authorship-llm.
Related papers
- LLMs Assist NLP Researchers: Critique Paper (Meta-)Reviewing [106.45895712717612]
Large language models (LLMs) have shown remarkable versatility in various generative tasks.
This study focuses on the topic of LLMs assist NLP Researchers.
To our knowledge, this is the first work to provide such a comprehensive analysis.
arXiv Detail & Related papers (2024-06-24T01:30:22Z) - Aligning Language Models to Explicitly Handle Ambiguity [22.078095273053506]
We propose Alignment with Perceived Ambiguity (APA), a novel pipeline that aligns language models to deal with ambiguous queries.
We show that APA empowers LLMs to explicitly detect and manage ambiguous queries while retaining the ability to answer clear questions.
arXiv Detail & Related papers (2024-04-18T07:59:53Z) - FAC$^2$E: Better Understanding Large Language Model Capabilities by
Dissociating Language and Cognition [57.747888532651]
Large language models (LLMs) are primarily evaluated by overall performance on various text understanding and generation tasks.
We present FAC$2$E, a framework for Fine-grAined and Cognition-grounded LLMs' Capability Evaluation.
arXiv Detail & Related papers (2024-02-29T21:05:37Z) - CLOMO: Counterfactual Logical Modification with Large Language Models [109.60793869938534]
We introduce a novel task, Counterfactual Logical Modification (CLOMO), and a high-quality human-annotated benchmark.
In this task, LLMs must adeptly alter a given argumentative text to uphold a predetermined logical relationship.
We propose an innovative evaluation metric, the Self-Evaluation Score (SES), to directly evaluate the natural language output of LLMs.
arXiv Detail & Related papers (2023-11-29T08:29:54Z) - Learning To Teach Large Language Models Logical Reasoning [33.88499005859982]
Large language models (LLMs) have gained enormous attention from both academia and industry.
However, current LLMs still output unreliable content in practical reasoning tasks due to their inherent issues.
arXiv Detail & Related papers (2023-10-13T14:53:06Z) - Towards LogiGLUE: A Brief Survey and A Benchmark for Analyzing Logical Reasoning Capabilities of Language Models [56.34029644009297]
Large language models (LLMs) have demonstrated the ability to overcome various limitations of formal Knowledge Representation (KR) systems.
LLMs excel most in abductive reasoning, followed by deductive reasoning, while they are least effective at inductive reasoning.
We study single-task training, multi-task training, and "chain-of-thought" knowledge distillation fine-tuning technique to assess the performance of model.
arXiv Detail & Related papers (2023-10-02T01:00:50Z) - Towards LLM-based Autograding for Short Textual Answers [4.853810201626855]
This manuscript is an evaluation of a large language model for the purpose of autograding.
Our findings suggest that while "out-of-the-box" LLMs provide a valuable tool, their readiness for independent automated grading remains a work in progress.
arXiv Detail & Related papers (2023-09-09T22:25:56Z) - Improving Open Information Extraction with Large Language Models: A
Study on Demonstration Uncertainty [52.72790059506241]
Open Information Extraction (OIE) task aims at extracting structured facts from unstructured text.
Despite the potential of large language models (LLMs) like ChatGPT as a general task solver, they lag behind state-of-the-art (supervised) methods in OIE tasks.
arXiv Detail & Related papers (2023-09-07T01:35:24Z) - Sentiment Analysis in the Era of Large Language Models: A Reality Check [69.97942065617664]
This paper investigates the capabilities of large language models (LLMs) in performing various sentiment analysis tasks.
We evaluate performance across 13 tasks on 26 datasets and compare the results against small language models (SLMs) trained on domain-specific datasets.
arXiv Detail & Related papers (2023-05-24T10:45:25Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.