From Intuition to Expertise: Rubric-Based Cognitive Calibration for Human Detection of LLM-Generated Korean Text

• URL: http://arxiv.org/abs/2601.19913v1
• Date: Tue, 06 Jan 2026 10:51:39 GMT
• Title: From Intuition to Expertise: Rubric-Based Cognitive Calibration for Human Detection of LLM-Generated Korean Text
• Abstract summary: We introduce LREAD, a rubric derived from national Korean writing standards.<n>Phase 1 measures intuition-only detection, Phase 2 enforces criterion-level scoring, and Phase 3 evaluates domain-focused mastery.<n>Across phases, majority-vote accuracy increases from 60% to 100%, accompanied by stronger inter-annotator agreement.
• Score: 7.067247930901556
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Distinguishing human-written Korean text from fluent LLM outputs remains difficult even for linguistically trained readers, who can over-trust surface well-formedness. We study whether expert detection can be treated as a learnable skill and improved through structured calibration. We introduce LREAD, a rubric derived from national Korean writing standards and adapted to target micro-level artifacts (e.g., punctuation optionality, spacing behavior, and register shifts). In a three-phase longitudinal blind protocol with Korean linguistics majors, Phase 1 measures intuition-only detection, Phase 2 enforces criterion-level scoring with explicit justifications, and Phase 3 evaluates domain-focused mastery on held-out elementary essays. Across phases, majority-vote accuracy increases from 60% to 100%, accompanied by stronger inter-annotator agreement (Fleiss' kappa: -0.09 --> 0.82). Compared to state-of-the-art LLM detectors, calibrated humans rely more on language-specific micro-diagnostics that are not well captured by coarse discourse priors. Our findings suggest that rubric-scaffolded expert judgment can serve as an interpretable complement to automated detectors for non-English settings, and we release the full rubric and a taxonomy of calibrated detection signatures.

Related papers

• Accurate but Natural? Diagnosing Grammatical and Idiomatic Gaps in Japanese EFL Writing [0.0]
  This study introduces a layered LLM-correction pipeline that isolates structural errors from unnaturalness.<n>Applying the overused-based CEFR-J grammar extractor, we quantify two diagnostic measures: accuracy gaps and idiomatic gaps.<n>Results reveal distinct patterns: definite articles, third-person singular -s, and modals (would, could) exhibit significant accuracy difficulties.<n>A two-dimensional instructional typology maps error rates against idiomatic gaps, distinguishing accurate but grammar items from error-prone or avoided complex forms.
  arXiv  Detail & Related papers  (2026-08-10T08:42:06Z)
• The Impact of Editorial Intervention on Detecting Native Language Traces [1.6210510806456278]
  Native Language Identification (NLI) is the task of determining an author's native language (L1) from their non-native writings.<n>With the advent of human-AI co-authorship, non-native texts are routinely corrected and rewritten by large language models.<n>We investigate the robustness of L1 traces across increasing degrees of editorial intervention.
  arXiv  Detail & Related papers  (2026-05-11T08:58:08Z)
• DETECT: Determining Ease and Textual Clarity of German Text Simplifications [4.005744004522348]
  DETECT is the first German-specific metric that holistically evaluates ATS quality across all three dimensions of simplicity, meaning preservation, and fluency.<n>We construct the largest German human evaluation dataset for text simplification to validate our metric directly.
  arXiv  Detail & Related papers  (2025-10-25T08:20:18Z)
• Beyond the Final Layer: Intermediate Representations for Better Multilingual Calibration in Large Language Models [50.34755385896279]
  Confidence calibration is crucial for the reliable deployment of Large Language Models (LLMs)<n>We conduct the first large-scale, systematic studies of multilingual calibration across six model families and over 100 languages.<n>We find that non-English languages suffer from systematically worse calibration.
  arXiv  Detail & Related papers  (2025-10-03T16:07:15Z)
• SiLVERScore: Semantically-Aware Embeddings for Sign Language Generation Evaluation [29.960223851833785]
  We propose SiLVERScore, a semantically-aware embedding-based evaluation metric for sign language generation.<n>On PHOENIX-14T and CSL-Daily datasets, SiLVERScore achieves near-perfect discrimination between correct and random pairs.
  arXiv  Detail & Related papers  (2025-09-04T00:58:43Z)
• Avaliação de eficiência na leitura: uma abordagem baseada em PLN [0.0]
  This study proposes an automated evaluation model for the cloze test in Brazilian Portuguese.<n>The integrated method demonstrated its effectiveness, achieving a high correlation with human evaluation.
  arXiv  Detail & Related papers  (2025-08-18T02:21:12Z)
• LLM-Assisted Cheating Detection in Korean Language via Keystrokes [1.9344365651682767]
  This paper presents a keystroke-based framework for detecting LLM-assisted cheating in Korean.<n>Our dataset includes 69 participants who completed writing tasks under three conditions: Bona fide writing, paraphrasing ChatGPT responses, and transcribing ChatGPT responses.
  arXiv  Detail & Related papers  (2025-07-29T20:59:03Z)
• Your Language Model Can Secretly Write Like Humans: Contrastive Paraphrase Attacks on LLM-Generated Text Detectors [77.82885394684202]
  We propose textbfContrastive textbfParaphrase textbfAttack (CoPA), a training-free method that effectively deceives text detectors.<n>CoPA constructs an auxiliary machine-like word distribution as a contrast to the human-like distribution generated by large language models.<n>Our theoretical analysis suggests the superiority of the proposed attack.
  arXiv  Detail & Related papers  (2025-05-21T10:08:39Z)
• KatFishNet: Detecting LLM-Generated Korean Text through Linguistic Feature Analysis [7.681048890143491]
  We introduce KatFish, the first benchmark dataset for detecting LLM-generated Korean text.<n>By examining spacing patterns, part-of-speech diversity, and comma usage, we illuminate the linguistic differences between human-written and LLM-generated Korean text.<n>KatFishNet achieves an average of 19.78% higher AUROC compared to the best-performing existing detection method.
  arXiv  Detail & Related papers  (2025-02-25T00:59:27Z)
• Prosody in Cascade and Direct Speech-to-Text Translation: a case study on Korean Wh-Phrases [79.07111754406841]
  This work proposes using contrastive evaluation to measure the ability of direct S2TT systems to disambiguate utterances where prosody plays a crucial role. Our results clearly demonstrate the value of direct translation systems over cascade translation models.
  arXiv  Detail & Related papers  (2024-02-01T14:46:35Z)
• Few-Shot Cross-lingual Transfer for Coarse-grained De-identification of Code-Mixed Clinical Texts [56.72488923420374]
  Pre-trained language models (LMs) have shown great potential for cross-lingual transfer in low-resource settings. We show the few-shot cross-lingual transfer property of LMs for named recognition (NER) and apply it to solve a low-resource and real-world challenge of code-mixed (Spanish-Catalan) clinical notes de-identification in the stroke.
  arXiv  Detail & Related papers  (2022-04-10T21:46:52Z)
• On Cross-Lingual Retrieval with Multilingual Text Encoders [51.60862829942932]
  We study the suitability of state-of-the-art multilingual encoders for cross-lingual document and sentence retrieval tasks. We benchmark their performance in unsupervised ad-hoc sentence- and document-level CLIR experiments. We evaluate multilingual encoders fine-tuned in a supervised fashion (i.e., we learn to rank) on English relevance data in a series of zero-shot language and domain transfer CLIR experiments.
  arXiv  Detail & Related papers  (2021-12-21T08:10:27Z)
• Evaluating Multilingual Text Encoders for Unsupervised Cross-Lingual Retrieval [51.60862829942932]
  We present a systematic empirical study focused on the suitability of the state-of-the-art multilingual encoders for cross-lingual document and sentence retrieval tasks. For sentence-level CLIR, we demonstrate that state-of-the-art performance can be achieved. However, the peak performance is not met using the general-purpose multilingual text encoders off-the-shelf', but rather relying on their variants that have been further specialized for sentence understanding tasks.
  arXiv  Detail & Related papers  (2021-01-21T00:15:38Z)
• Inducing Language-Agnostic Multilingual Representations [61.97381112847459]
  Cross-lingual representations have the potential to make NLP techniques available to the vast majority of languages in the world. We examine three approaches for this: (i) re-aligning the vector spaces of target languages to a pivot source language; (ii) removing language-specific means and variances, which yields better discriminativeness of embeddings as a by-product; and (iii) increasing input similarity across languages by removing morphological contractions and sentence reordering.
  arXiv  Detail & Related papers  (2020-08-20T17:58:56Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.