Inter-Rater Reliability of LLM and Rule-Based Annotation for Inferential Narrative Features: Three Studies on a Turkish Corpus

• URL: http://arxiv.org/abs/2609.13936v1
• Date: Sat, 12 Sep 2026 13:31:30 GMT
• Title: Inter-Rater Reliability of LLM and Rule-Based Annotation for Inferential Narrative Features: Three Studies on a Turkish Corpus
• Abstract summary: This report answers the question of whether a human would agree with labels in a dataset with automatically generated feature annotations.<n>Three studies are reported. Five machine labellers returned positive rates of $0$, $1$, $40$, $72$ and $78$ out of $100$ scenes, against a human count of $9$.<n>Two readings survive: the feature is genuinely inferential and beyond current automatic detection, or the rule's definition is not yet operational enough for any rater to apply consistently.
• Score: 0.0
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Datasets that ship automatically generated feature annotations invite a question rarely asked of them: would a human agree with those labels? This report answers that for the Objective Projection corpus, a Turkish narrative dataset whose scenes carry a per-scene applied_rules field from a rule-based detector over six craft features -- two prohibitions (emotion labelling, simile) and four positive techniques (materialized metaphor, micro-focus, temporal anchor, atmosphere contradiction). Three studies are reported. Study 1 ($n = 120$) scores the detector against blind labels from the scheme's own author. Study 2 ($n = 100$, a disjoint scene set) scores the detector plus Gemini 2.5 Flash and Grok against an independent non-expert rater whose labels were locked before any machine ran. Study 2b re-runs the identical protocol with Claude Fable 5 (High) and ChatGPT 5.5. The central result concerns one rule. On materialized metaphor -- closest to the methodology's theoretical core -- the five machine labellers returned positive rates of $0$, $1$, $40$, $72$ and $78$ out of $100$ scenes, against a human count of $9$. Cohen's $κ$ was at or indistinguishable from chance for five of six labellers, across both human references and both scene sets: $0.004$, $0.015$, $0.000$, $0.019$, $0.027$. Raw agreement ranged from $74.7\%$ to $84.5\%$, an artefact of class imbalance rather than a sign of competence. We deliberately do not resolve this into a single story. Two readings survive: the feature is genuinely inferential and beyond current automatic detection, or the rule's definition is not yet operational enough for any rater to apply consistently -- including the human. Distinguishing them needs a second independent human rater, which this report does not have and therefore does not claim.

Related papers

• The Constitutional Coverage Trilemma in AI Governance [0.0]
  AI systems function as emphconstitutional institutions: each deployed model encodes an implicit ranking among safety, helpfulness, honesty, autonomy, and equity.<n>We ask whether the supply of frontier constitutional types covers human demand.
  arXiv  Detail & Related papers  (2026-09-01T14:08:56Z)
• VERDICT: Agreement Beats Pixel-Space Verification in Real-Document OCSR [6.923778751564602]
  VERDICT is an image-based interface for searching and retrieving molecular records.<n>It enables validated labels for multimodal molecular databases linking structure images, machine-readable representations, and source-publication information.
  arXiv  Detail & Related papers  (2026-08-23T02:41:06Z)
• When AI Rewrites, Classifiers Relax: Uncertainty-Aware Sentiment Analysis on Sarcastic and AI-Paraphrased Social Text [0.0]
  Confidence scores on sarcastic text are significantly lower than on non-sarcastic text.<n>Our results motivate a shift from confident single-label prediction to uncertainty-aware abstention in high-stakes sentiment applications.
  arXiv  Detail & Related papers  (2026-08-15T17:41:45Z)
• Grounding Spatial Relations in a Compact World Model: Instruction Leakage and a Goal-Free Dynamics Fix [46.122203287541005]
  We show that a goal-conditioned predictor reaches a striking $0.90$ relation-readout accuracy, yet this is emphinstruction transcription, not perception.<n>Withholding the goal collapses it to chance ($0.90!to!0.27$, three seeds) and a counterfactual instruction makes the predicted anchors follow the emphfalse instruction $94.5%$ of the time.<n>The diagnosis prescribes the fix: keep the goal out of the dynamics (it belongs to the planner's cost) and supervise the emphread path
  arXiv  Detail & Related papers  (2026-07-08T02:38:43Z)
• What Do EEG Foundation Models Capture from Human Brain Signals? [64.48249643001402]
  Modern EEG foundation models learn directly from raw signals via self-supervised pretraining.<n>We decompose into three sub-questions: emphwhat does the model learn, emphwhat does the model use, and emphhow much can be explained<n>The audit covers three foundation models (CSBrain, CBraMod, LaBraM), five clinical tasks (MDD, Stress, ISRUC-Sleep, TUSL, Siena), and a 6-family 63-feature lexicon.
  arXiv  Detail & Related papers  (2026-05-12T01:57:53Z)
• The Geometry of Forgetting: Temporal Knowledge Drift as an Independent Axis in LLM Representations [50.43168858368539]
  Large language models confidently produce outdated answers, and no existing method can detect them.<n>We show this is not an engineering failure but a structural one: temporal drift is encoded as a direction in the residual stream geometrically to both correctness and uncertainty.
  arXiv  Detail & Related papers  (2026-05-09T22:27:31Z)
• Beating the Style Detector: Three Hours of Agentic Research on the AI-Text Arms Race [3.9508043303559828]
  Reproducing an empirical NLP study used to take weeks.<n>All code, $648$ mimic drafts, trained detectors, diagnostics, and adversarial trajectories are released.
  arXiv  Detail & Related papers  (2026-05-04T14:10:41Z)
• Do We Still Need Humans in the Loop? Comparing Human and LLM Annotation in Active Learning for Hostility Detection [68.37351671559675]
  Active learning can annotate thousands of instances from a short prompt at negligible cost.<n>This raises two questions for active learning (AL): can LLM labels replace human labels within the AL loop, and does AL remain necessary when entire corpora can be labelled at once?<n>We investigate both questions on a new dataset of 277,902 German political TikTok comments.
  arXiv  Detail & Related papers  (2026-04-15T14:10:58Z)
• Can Humans Tell? A Dual-Axis Study of Human Perception of LLM-Generated News [47.03825808787752]
  Can humans tell whether a news article was written by a person or a large language model (LLM)?<n>We investigate this question using JudgeGPT, a study platform that measures source attribution (human vs. machine) and authenticity judgment (legitimate vs. fake) on continuous scales.<n>We report five findings: (1) participants cannot reliably distinguish machine-generated from human-written text; (2) this inability holds across all tested models, including open-weight models with as few as 7B parameters; and (3) accuracy degrades after approximately 30 sequential evaluations due to cognitive fatigue.
  arXiv  Detail & Related papers  (2026-04-04T15:03:42Z)
• Criterion-referenceability determines LLM-as-a-judge validity across physics assessment formats [0.01116979912801043]
  We compare GPT-5.2, Grok 4.1, Claude Opus 4.5, DeepSeek-V3.2, Gemini Pro 3, and committee aggregations against human markers under blind, solution-provided, false-solution, and exemplar-anchored conditions.<n>For $n=771$ blind university exam questions, models achieve fractional mean absolute errors (fMAE) $approx 0.22$ with robust discriminative validity.<n>Across $n=55$ scripts, blind AI marking is harsher and more variable than human marking, with discriminative validity already poor.
  arXiv  Detail & Related papers  (2026-03-16T02:09:06Z)
• Eroding the Truth-Default: A Causal Analysis of Human Susceptibility to Foundation Model Hallucinations and Disinformation in the Wild [47.03825808787752]
  "Fake news familiarity" emerges as a candidate mediator, suggesting that exposure may function as adversarial training for human discriminators.<n>These findings suggest that "pre-bunking" interventions should target cognitive source monitoring rather than demographic segmentation.
  arXiv  Detail & Related papers  (2026-01-30T11:49:58Z)
• UFO$^2$: A Unified Framework towards Omni-supervised Object Detection [175.9693271392461]
  We present UFO$2$, a unified object detection framework that can handle different forms of supervision simultaneously. Specifically, UFO$2$ incorporates strong supervision (e.g., boxes), various forms of partial supervision (e.g., class tags, points, and scribbles) and unlabeled data.
  arXiv  Detail & Related papers  (2020-10-21T07:46:30Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.