Beyond Document Grounding: Span-Level Hallucination Detection over Code, Tool Output, and Documents

• URL: http://arxiv.org/abs/2607.00895v1
• Date: Wed, 01 Jul 2026 13:01:42 GMT
• Title: Beyond Document Grounding: Span-Level Hallucination Detection over Code, Tool Output, and Documents
• Abstract summary: Hallucination detection for retrieval-augmented generation (RAG) is usually evaluated on natural-language document evidence.<n>We introduce a unified benchmark for span-level hallucination detection over code, tool output, structured documents, and existing natural-language RAG datasets.
• Score: 10.262113115662334
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Hallucination detection for retrieval-augmented generation (RAG) is usually evaluated on natural-language document evidence. However, grounded generation systems increasingly rely on structured inputs: source code, developer-tool output, markdown documents, tables, and repository metadata. We introduce a unified benchmark for span-level hallucination detection over code, tool output, structured documents, and existing natural-language RAG datasets. The benchmark is built by starting from grounded correct answers, injecting localized hallucinations with exact character labels, and validating the code test split with evidence-based review. Our fine-tuned Qwen3.5-2B detector reaches 0.689 span-F1 on the unified test set and 0.60 on the code-agent source, where it substantially outperforms LettuceDetect-large (0.17) and the strongest zero-shot LLM judges we evaluated (at most 0.22). The same model remains competitive on established natural-language benchmarks, with 81.8 RAGTruth example-F1 and 0.724 English PsiloQA IoU.

Related papers

• Meddies-PII: A Multilingual Framework for Personally Identifiable Information Extraction in Clinical De-identification [0.649970685896541]
  We introduce Meddies-PII-Dataset, a corpus of one million synthetic clinical documents spanning seventeen languages and nine PII labels.<n>The documents are generated using attribute-conditioned prompts and validated through thirteen deterministic gates.<n>Meddies-PII-Model achieves the highest performance among the evaluated systems on all reported benchmarks.
  arXiv  Detail & Related papers  (2026-09-11T07:53:52Z)
• VisDocAgentBench: Benchmarking Agents for Visually Rich Document Retrieval [65.89247522243959]
  VisDocAgentBench is a closed-corpus benchmark comparing static and agentic retrieval under a ranked-output contract.<n>It contains 2,375 pages from 100 documents and 120 unique-target queries balanced across direct, one-bridge, and two-bridge evidence structures.<n>A strong late-interaction visual retriever reaches 97.50% Recall@1 on direct items but 2.50% on two-bridge items, exposing the limits of query--target matching when relevance depends on corpus context.
  arXiv  Detail & Related papers  (2026-08-18T15:23:06Z)
• MonkeyOCRv2: A Visual-Text Foundation Model for Document AI [71.06391669976786]
  MonkeyOCRv2 is a visual-text pretrained model for document AI.<n> MonkeyDoc v2 is the largest document-image pretraining corpus, comprising 113 million images spanning 17 languages.
  arXiv  Detail & Related papers  (2026-07-13T13:43:39Z)
• SPECTRA: Synthetic IR Test Collections with Relevance Oracles and Controlled Distractor Diagnostics [0.8557392136621891]
  This paper introduces SPECTRA, a reproducible framework for generating synthetic text corpora and retrieval test collections.<n>A single-process Python prototype generated corpora up to 60,000 documents and 9.61 million tokens while preserving controllable long-tail vocabulary growth.<n>Results show that lightweight synthetic corpora can expose retrieval-system scaling and failure modes before costly collection construction begins.
  arXiv  Detail & Related papers  (2026-05-29T17:44:15Z)
• DocAtlas: Multilingual Document Understanding Across 80+ Languages [58.715440331861295]
  We introduce DocAtlas, a framework that constructs high-fidelity OCR datasets and benchmarks covering 82 languages.<n>Our dual pipelines, differential rendering of native DOCX documents and synthetic-based generation for right-to-left scripts produce precise structural annotations.
  arXiv  Detail & Related papers  (2026-05-12T18:09:38Z)
• Whitespaces Don't Lie: Feature-Driven and Embedding-Based Approaches for Detecting Machine-Generated Code [0.2624902795082451]
  Large language models (LLMs) have made it remarkably easy to synthesize plausible source code from natural language prompts.<n>This paper investigates the problem of distinguishing human-written from machine-generated code by comparing two complementary approaches.
  arXiv  Detail & Related papers  (2026-01-27T06:43:51Z)
• A document processing pipeline for the construction of a dataset for topic modeling based on the judgments of the Italian Supreme Court [5.612141846711729]
  We develop a document processing pipeline that produces an anonymized dataset optimized for topic modeling.<n>The pipeline integrates document layout analysis (YOLOv8x), optical character recognition, and text anonymization.<n>Compared to OCR-only methods, our dataset improved topic modeling with a diversity score of 0.6198 and a coherence score of 0.6638.
  arXiv  Detail & Related papers  (2025-05-13T11:06:24Z)
• C-FAITH: A Chinese Fine-Grained Benchmark for Automated Hallucination Evaluation [58.40263551616771]
  We introduce HaluAgent, an agentic framework that automatically constructs fine-grained QA dataset based on some knowledge documents.<n>Our experiments demonstrate that the manually designed rules and prompt optimization can improve the quality of generated data.
  arXiv  Detail & Related papers  (2025-04-14T12:21:55Z)
• BRIGHT: A Realistic and Challenging Benchmark for Reasoning-Intensive Retrieval [54.54576644403115]
  We introduce BRIGHT, the first text retrieval benchmark that requires intensive reasoning to retrieve relevant documents.<n>Our dataset consists of 1,384 real-world queries spanning diverse domains, such as economics, psychology, mathematics, and coding.<n>We show that incorporating explicit reasoning about the query improves retrieval performance by up to 12.2 points.
  arXiv  Detail & Related papers  (2024-07-16T17:58:27Z)
• Factcheck-Bench: Fine-Grained Evaluation Benchmark for Automatic Fact-checkers [121.53749383203792]
  We present a holistic end-to-end solution for annotating the factuality of large language models (LLMs)-generated responses. We construct an open-domain document-level factuality benchmark in three-level granularity: claim, sentence and document. Preliminary experiments show that FacTool, FactScore and Perplexity are struggling to identify false claims.
  arXiv  Detail & Related papers  (2023-11-15T14:41:57Z)
• Precise Zero-Shot Dense Retrieval without Relevance Labels [60.457378374671656]
  Hypothetical Document Embeddings(HyDE) is a zero-shot dense retrieval system. We show that HyDE significantly outperforms the state-of-the-art unsupervised dense retriever Contriever.
  arXiv  Detail & Related papers  (2022-12-20T18:09:52Z)
• Few-Shot Cross-lingual Transfer for Coarse-grained De-identification of Code-Mixed Clinical Texts [56.72488923420374]
  Pre-trained language models (LMs) have shown great potential for cross-lingual transfer in low-resource settings. We show the few-shot cross-lingual transfer property of LMs for named recognition (NER) and apply it to solve a low-resource and real-world challenge of code-mixed (Spanish-Catalan) clinical notes de-identification in the stroke.
  arXiv  Detail & Related papers  (2022-04-10T21:46:52Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.