GPT4o-Receipt: A Dataset and Human Study for AI-Generated Document Forensics

• URL: http://arxiv.org/abs/2603.11442v1
• Date: Thu, 12 Mar 2026 02:05:27 GMT
• Title: GPT4o-Receipt: A Dataset and Human Study for AI-Generated Document Forensics
• Abstract summary: We present GPT4o-Receipt, a benchmark of 1,235 receipt images pairing GPT-4o-generated receipts with authentic ones from established datasets.<n>Human annotators exhibit the largest visual discrimination gap of any evaluator, yet their binary detection F1 falls well below Claude Sonnet 4 and below Gemini 2.5 Flash.<n>Our five-model evaluation reveals dramatic performance disparities and calibration differences that render simple accuracy metrics insufficient for detector selection.
• Score: 12.448893104517808
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Can humans detect AI-generated financial documents better than machines? We present GPT4o-Receipt, a benchmark of 1,235 receipt images pairing GPT-4o-generated receipts with authentic ones from established datasets, evaluated by five state-of-the-art multimodal LLMs and a 30-annotator crowdsourced perceptual study. Our findings reveal a striking paradox: humans are better at seeing AI artifacts, yet worse at detecting AI documents. Human annotators exhibit the largest visual discrimination gap of any evaluator, yet their binary detection F1 falls well below Claude Sonnet 4 and below Gemini 2.5 Flash. This paradox resolves once the mechanism is understood: the dominant forensic signals in AI-generated receipts are arithmetic errors -- invisible to visual inspection but systematically verifiable by LLMs. Humans cannot perceive that a subtotal is incorrect; LLMs verify it in milliseconds. Beyond the human--LLM comparison, our five-model evaluation reveals dramatic performance disparities and calibration differences that render simple accuracy metrics insufficient for detector selection. GPT4o-Receipt, the evaluation framework, and all results are released publicly to support future research in AI document forensics.

Related papers

• NovGauge: A Fine-Grained Benchmark for Diagnosing LLMs' Capability in Paper Novelty Assessment [54.4265627247104]
  Large language models (LLMs) are increasingly used in peer review at major AI conferences.<n>Existing benchmarks assess novelty as a single holistic score.<n>We present NovGauge, a human-anchored benchmark for fine-grained novelty assessment diagnosis.
  arXiv  Detail & Related papers  (2026-09-10T08:34:56Z)
• Operation-Guided Progressive Human-to-AI Text Transformation Benchmark for Multi-Granularity AI-Text Detection [49.41415417307568]
  OpAI-Bench is an operation-guided benchmark for studying progressive human-to-AI text transformation.<n>We show that AI-text detectability is governed not only by the proportion of AI-edited content, but also by edit operation, domain, and cumulative revision history.
  arXiv  Detail & Related papers  (2026-06-04T17:58:05Z)
• Sem-Detect: Semantic Level Detection of AI Generated Peer-Reviews [11.21278193949403]
  We propose Sem-Detect, an authorship detection method for peer reviews.<n>It compares a target review against multiple AI-generated reviews of the same paper.<n>It is able to distinguish fully AI reviews from authentic human-written ones.
  arXiv  Detail & Related papers  (2026-05-20T20:19:16Z)
• AIForge-Doc: A Benchmark for Detecting AI-Forged Tampering in Financial and Form Documents [7.014776899553499]
  We present AIForge-Doc, the first dedicated benchmark targeting exclusively diffusion-model-based inpainting in financial and form documents with pixel-level annotation.<n>We benchmark three representative detectors -- TruFor, DocTamper, and a zero-shot GPT-4o judge -- and find that all existing methods degrade substantially.
  arXiv  Detail & Related papers  (2026-02-24T05:37:35Z)
• Fact or Fake? Assessing the Role of Deepfake Detectors in Multimodal Misinformation Detection [24.64785245180038]
  We provide the first systematic analysis of deepfake detectors in the context of multimodal misinformation detection.<n>Deepfake detectors offer limited standalone value, achieving F1 scores in the range of 0.26-0.53 on MMFakeBench and 0.33-0.49 on DGM4.<n>The evidence-centric fact-checking system achieves the highest performance, reaching F1 scores of approximately 0.81 on MMFakeBench and 0.55 on DGM4.
  arXiv  Detail & Related papers  (2026-02-02T09:28:16Z)
• Can AI Recognize Its Own Reflection? Self-Detection Performance of LLMs in Computing Education [0.10195618602298682]
  This paper evaluates the capacity of three prominent Large Language Models to identify AI-generated text in computing-specific contexts.<n>We test their performance under both standard and 'deceptive' prompt conditions, where the models were instructed to evade detection.<n>Our findings reveal a significant instability: while default AI-generated text was easily identified, all models struggled to correctly classify human-written work.
  arXiv  Detail & Related papers  (2025-12-29T16:35:52Z)
• Skyra: AI-Generated Video Detection via Grounded Artifact Reasoning [66.51617619673587]
  We present Skyra, a specialized large language model (MLLM) that identifies human-perceivable visual artifacts in AI-generated videos.<n>To support this objective, we construct ViF-CoT-4K for Supervised Fine-Tuning (SFT), which represents the first large-scale AI-generated video dataset with fine-grained human annotations.<n>We then develop a two-stage training strategy that systematically enhances our model's artifact's-temporal perception, explanation capability, and detection accuracy.
  arXiv  Detail & Related papers  (2025-12-17T18:48:26Z)
• Mind Reading or Misreading? LLMs on the Big Five Personality Test [1.3649494534428745]
  We evaluate large language models (LLMs) for automatic personality prediction from text under the binary Five Factor Model (BIG5).<n>Open-source models sometimes approach GPT-4 and prior benchmarks, but no configuration yields consistently reliable predictions in zero-shot binary settings.<n>These findings show that current out-of-the-box LLMs are not yet suitable for APPT, and that careful coordination of prompt design, trait framing, and evaluation metrics is essential for interpretable results.
  arXiv  Detail & Related papers  (2025-11-28T11:40:30Z)
• Can Multi-modal (reasoning) LLMs detect document manipulation? [7.547605888349287]
  Document fraud poses a significant threat to industries reliant on secure and verifiable documentation.<n>This study investigates the efficacy of state-of-the-art multi-modal large language models (LLMs)<n>We benchmark these models against each other and prior work on document fraud detection techniques.
  arXiv  Detail & Related papers  (2025-08-14T18:57:07Z)
• Could AI Trace and Explain the Origins of AI-Generated Images and Text? [53.11173194293537]
  AI-generated content is increasingly prevalent in the real world.<n> adversaries might exploit large multimodal models to create images that violate ethical or legal standards.<n>Paper reviewers may misuse large language models to generate reviews without genuine intellectual effort.
  arXiv  Detail & Related papers  (2025-04-05T20:51:54Z)
• Group-Adaptive Threshold Optimization for Robust AI-Generated Text Detection [58.419940585826744]
  We introduce FairOPT, an algorithm for group-specific threshold optimization for probabilistic AI-text detectors.<n>We partitioned data into subgroups based on attributes (e.g., text length and writing style) and implemented FairOPT to learn decision thresholds for each group to reduce discrepancy.<n>Our framework paves the way for more robust classification in AI-generated content detection via post-processing.
  arXiv  Detail & Related papers  (2025-02-06T21:58:48Z)
• A New Benchmark and Reverse Validation Method for Passage-level Hallucination Detection [63.56136319976554]
  Large Language Models (LLMs) generate hallucinations, which can cause significant damage when deployed for mission-critical tasks. We propose a self-check approach based on reverse validation to detect factual errors automatically in a zero-resource fashion. We empirically evaluate our method and existing zero-resource detection methods on two datasets.
  arXiv  Detail & Related papers  (2023-10-10T10:14:59Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.