KatFishNet: Detecting LLM-Generated Korean Text through Linguistic Feature Analysis

• URL: http://arxiv.org/abs/2503.00032v4
• Date: Wed, 02 Jul 2025 03:55:41 GMT
• Title: KatFishNet: Detecting LLM-Generated Korean Text through Linguistic Feature Analysis
• Abstract summary: We introduce KatFish, the first benchmark dataset for detecting LLM-generated Korean text.<n>By examining spacing patterns, part-of-speech diversity, and comma usage, we illuminate the linguistic differences between human-written and LLM-generated Korean text.<n>KatFishNet achieves an average of 19.78% higher AUROC compared to the best-performing existing detection method.
• Score: 3.224880576815583
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: The rapid advancement of large language models (LLMs) increases the difficulty of distinguishing between human-written and LLM-generated text. Detecting LLM-generated text is crucial for upholding academic integrity, preventing plagiarism, protecting copyrights, and ensuring ethical research practices. Most prior studies on detecting LLM-generated text focus primarily on English text. However, languages with distinct morphological and syntactic characteristics require specialized detection approaches. Their unique structures and usage patterns can hinder the direct application of methods primarily designed for English. Among such languages, we focus on Korean, which has relatively flexible spacing rules, a rich morphological system, and less frequent comma usage compared to English. We introduce KatFish, the first benchmark dataset for detecting LLM-generated Korean text. The dataset consists of text written by humans and generated by four LLMs across three genres. By examining spacing patterns, part-of-speech diversity, and comma usage, we illuminate the linguistic differences between human-written and LLM-generated Korean text. Building on these observations, we propose KatFishNet, a detection method specifically designed for the Korean language. KatFishNet achieves an average of 19.78% higher AUROC compared to the best-performing existing detection method. Our code and data are available at https://github.com/Shinwoo-Park/detecting_llm_generated_korean_text_through_linguistic_analysis.

Related papers

• DetectRL-X: Towards Reliable Multilingual and Real-World LLM-Generated Text Detection [68.92080038934164]
  This study introduces DetectRL-X, a benchmark designed to evaluate advanced detectors across 8 dimensions.<n>The benchmark encompasses 8 languages commonly used in commercial contexts and collects human-written texts from 6 domains highly susceptible to misuse.
  arXiv  Detail & Related papers  (2026-05-15T01:29:26Z)
• Interpretable Stylistic Variation in Human and LLM Writing Across Genres, Models, and Decoding Strategies [0.34202935599316525]
  Large Language Models (LLMs) are now capable of generating highly fluent, human-like text.<n>LLMs enable many applications, but also raise concerns such as large scale spam, phishing, or academic misuse.<n>We perform a large scale analysis of stylistic variation across human-written text and outputs from 11 LLMs.
  arXiv  Detail & Related papers  (2026-04-15T17:31:21Z)
• EnsemJudge: Enhancing Reliability in Chinese LLM-Generated Text Detection through Diverse Model Ensembles [16.37055946884028]
  EnsemJudge is a robust framework for detecting Chinese LLM-generated text.<n>We trained and evaluated our system on a carefully constructed Chinese dataset provided by NLPCC2025 Shared Task 1.
  arXiv  Detail & Related papers  (2026-03-30T01:50:46Z)
• Linguistically Informed Graph Model and Semantic Contrastive Learning for Korean Short Text Classification [2.4071330817126477]
  We propose LIGRAM, a hierarchical heterogeneous graph model for Korean short-text classification.<n>The proposed model constructs sub-graphs at the morpheme, part-of-speech, and named-entity levels and hierarchically integrates them to compensate for the limited contextual information in short texts.<n>We evaluate LIGRAM on four Korean short-text datasets, where it consistently outperforms existing baseline models.
  arXiv  Detail & Related papers  (2026-03-04T02:17:13Z)
• From Intuition to Expertise: Rubric-Based Cognitive Calibration for Human Detection of LLM-Generated Korean Text [7.067247930901556]
  We introduce LREAD, a rubric derived from national Korean writing standards.<n>Phase 1 measures intuition-only detection, Phase 2 enforces criterion-level scoring, and Phase 3 evaluates domain-focused mastery.<n>Across phases, majority-vote accuracy increases from 60% to 100%, accompanied by stronger inter-annotator agreement.
  arXiv  Detail & Related papers  (2026-01-06T10:51:39Z)
• "I know myself better, but not really greatly": Using LLMs to Detect and Explain LLM-Generated Texts [10.454446545249096]
  Large language models (LLMs) have demonstrated impressive capabilities in generating human-like texts.<n>This paper explores the detection and explanation capabilities of LLM-based detectors of human-generated texts.
  arXiv  Detail & Related papers  (2025-02-18T11:00:28Z)
• Idiosyncrasies in Large Language Models [54.26923012617675]
  We unveil and study idiosyncrasies in Large Language Models (LLMs)<n>We find that fine-tuning existing text embedding models on LLM-generated texts yields excellent classification accuracy.<n>We leverage LLM as judges to generate detailed, open-ended descriptions of each model's idiosyncrasies.
  arXiv  Detail & Related papers  (2025-02-17T18:59:02Z)
• GigaCheck: Detecting LLM-generated Content [72.27323884094953]
  In this work, we investigate the task of generated text detection by proposing the GigaCheck. Our research explores two approaches: (i) distinguishing human-written texts from LLM-generated ones, and (ii) detecting LLM-generated intervals in Human-Machine collaborative texts. Specifically, we use a fine-tuned general-purpose LLM in conjunction with a DETR-like detection model, adapted from computer vision, to localize AI-generated intervals within text.
  arXiv  Detail & Related papers  (2024-10-31T08:30:55Z)
• Linguistics Theory Meets LLM: Code-Switched Text Generation via Equivalence Constrained Large Language Models [16.82812708514889]
  Code-switching, the phenomenon of alternating between two or more languages in a single conversation, presents unique challenges for Natural Language Processing (NLP) Most existing research focuses on either syntactic constraints or neural generation, with few efforts to integrate linguistic theory with large language models (LLMs) for generating natural code-switched text. We introduce EZSwitch, a novel framework that combines Equivalence Constraint Theory (ECT) with LLMs to produce linguistically valid and fluent code-switched text.
  arXiv  Detail & Related papers  (2024-10-30T03:03:32Z)
• Unveiling Large Language Models Generated Texts: A Multi-Level Fine-Grained Detection Framework [9.976099891796784]
  Large language models (LLMs) have transformed human writing by enhancing grammar correction, content expansion, and stylistic refinement. Existing detection methods, which mainly rely on single-feature analysis and binary classification, often fail to effectively identify LLM-generated text in academic contexts. We propose a novel Multi-level Fine-grained Detection framework that detects LLM-generated text by integrating low-level structural, high-level semantic, and deep-level linguistic features.
  arXiv  Detail & Related papers  (2024-10-18T07:25:00Z)
• Understanding and Mitigating Language Confusion in LLMs [76.96033035093204]
  We evaluate 15 typologically diverse languages with existing and newly-created English and multilingual prompts.<n>We find that Llama Instruct and Mistral models exhibit high degrees of language confusion.<n>We find that language confusion can be partially mitigated via few-shot prompting, multilingual SFT and preference tuning.
  arXiv  Detail & Related papers  (2024-06-28T17:03:51Z)
• MultiSocial: Multilingual Benchmark of Machine-Generated Text Detection of Social-Media Texts [0.6053347262128919]
  MultiSocial dataset contains 472,097 texts, of which about 58k are human-written. We use this benchmark to compare existing detection methods in zero-shot as well as fine-tuned form. Our results indicate that the fine-tuned detectors have no problem to be trained on social-media texts.
  arXiv  Detail & Related papers  (2024-06-18T12:26:09Z)
• Towards Reliable Detection of LLM-Generated Texts: A Comprehensive Evaluation Framework with CUDRT [9.682499180341273]
  Large language models (LLMs) have significantly advanced text generation, but the human-like quality of their outputs presents major challenges.<n>We propose CUDRT, a comprehensive evaluation framework and bilingual benchmark in Chinese and English.<n>This framework supports scalable, reproducible experiments and enables analysis of how operational diversity, multilingual training sets, and LLM architectures influence detection performance.
  arXiv  Detail & Related papers  (2024-06-13T12:43:40Z)
• ReMoDetect: Reward Models Recognize Aligned LLM's Generations [55.06804460642062]
  Large language models (LLMs) generate human-preferable texts. In this paper, we identify the common characteristics shared by these models. We propose two training schemes to further improve the detection ability of the reward model.
  arXiv  Detail & Related papers  (2024-05-27T17:38:33Z)
• Who Wrote This? The Key to Zero-Shot LLM-Generated Text Detection Is GECScore [51.65730053591696]
  We propose a simple yet effective black-box zero-shot detection approach based on the observation that human-written texts typically contain more grammatical errors than LLM-generated texts. Experimental results show that our method outperforms current state-of-the-art (SOTA) zero-shot and supervised methods.
  arXiv  Detail & Related papers  (2024-05-07T12:57:01Z)
• Decomposed Prompting: Probing Multilingual Linguistic Structure Knowledge in Large Language Models [54.58989938395976]
  We introduce a decomposed prompting approach for sequence labeling tasks.<n>We test our method on the Universal Dependencies part-of-speech tagging dataset for 38 languages.
  arXiv  Detail & Related papers  (2024-02-28T15:15:39Z)
• LLM-Detector: Improving AI-Generated Chinese Text Detection with Open-Source LLM Instruction Tuning [4.328134379418151]
  Existing AI-generated text detection models are prone to in-domain over-fitting. We propose LLM-Detector, a novel method for both document-level and sentence-level text detection.
  arXiv  Detail & Related papers  (2024-02-02T05:54:12Z)
• SeqXGPT: Sentence-Level AI-Generated Text Detection [62.3792779440284]
  We introduce a sentence-level detection challenge by synthesizing documents polished with large language models (LLMs) We then propose textbfSequence textbfX (Check) textbfGPT, a novel method that utilizes log probability lists from white-box LLMs as features for sentence-level AIGT detection.
  arXiv  Detail & Related papers  (2023-10-13T07:18:53Z)
• Towards Codable Watermarking for Injecting Multi-bits Information to LLMs [86.86436777626959]
  Large language models (LLMs) generate texts with increasing fluency and realism. Existing watermarking methods are encoding-inefficient and cannot flexibly meet the diverse information encoding needs. We propose Codable Text Watermarking for LLMs (CTWL) that allows text watermarks to carry multi-bit customizable information.
  arXiv  Detail & Related papers  (2023-07-29T14:11:15Z)
• DPIC: Decoupling Prompt and Intrinsic Characteristics for LLM Generated Text Detection [56.513637720967566]
  Large language models (LLMs) can generate texts that pose risks of misuse, such as plagiarism, planting fake reviews on e-commerce platforms, or creating inflammatory false tweets. Existing high-quality detection methods usually require access to the interior of the model to extract the intrinsic characteristics. We propose to extract deep intrinsic characteristics of the black-box model generated texts.
  arXiv  Detail & Related papers  (2023-05-21T17:26:16Z)
• KLUE: Korean Language Understanding Evaluation [43.94952771238633]
  We introduce Korean Language Understanding Evaluation (KLUE) benchmark. KLUE is a collection of 8 Korean natural language understanding (NLU) tasks. We build all of the tasks from scratch from diverse source corpora while respecting copyrights.
  arXiv  Detail & Related papers  (2021-05-20T11:40:30Z)
• FILTER: An Enhanced Fusion Method for Cross-lingual Language Understanding [85.29270319872597]
  We propose an enhanced fusion method that takes cross-lingual data as input for XLM finetuning. During inference, the model makes predictions based on the text input in the target language and its translation in the source language. To tackle this issue, we propose an additional KL-divergence self-teaching loss for model training, based on auto-generated soft pseudo-labels for translated text in the target language.
  arXiv  Detail & Related papers  (2020-09-10T22:42:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.