OpenHalDet: A Unified Benchmark for Hallucination Detection across Diverse Generation Scenarios

• URL: http://arxiv.org/abs/2606.06959v1
• Date: Fri, 05 Jun 2026 06:38:40 GMT
• Title: OpenHalDet: A Unified Benchmark for Hallucination Detection across Diverse Generation Scenarios
• Abstract summary: Hallucination detection is essential for the reliable deployment of large language models.<n>We introduce OpenHalDet, a unified benchmark for hallucination detection across diverse generation scenarios.
• Score: 34.58119255209895
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Hallucination detection is essential for the reliable deployment of large language models (LLMs). However, existing evaluations face two core challenges: inconsistent inference configuration and evaluation, and limited coverage of downstream domains and tasks. Consequently, reported detector performance is often difficult to compare, reproduce, and generalize beyond specific experimental settings. We introduce OpenHalDet, a unified benchmark for hallucination detection across diverse generation scenarios. OpenHalDet standardizes the evaluation pipeline, from prompt construction and response generation to truthfulness annotation, detector scoring, and metric computation. It supports heterogeneous detector families under different access settings, including black-box methods that use only generated outputs, gray-box methods that rely on probability-based signals, and white-box methods that exploit internal model signals. By bringing diverse tasks, models, and detectors into a shared framework, OpenHalDet enables controlled comparison and provides a systematic view of how different detection paradigms behave in LLM applications. We release OpenHalDet as an open and extensible codebase to facilitate reproducible evaluation and future development of hallucination detection methods. The code and datasets are available at https://github.com/Nellie179/Hallucination-Detection.

Related papers

• Mixture-of-Expert Blocks Contain Strong Hallucination Detection Signals [2.2302915692528367]
  We introduce InnerExpert, the first method to leverage MoE-specific signals for per-token hallucination detection.<n>Our results show that InnerExpert outperforms existing methods across five datasets and two MoE architectures.
  arXiv  Detail & Related papers  (2026-08-18T12:00:20Z)
• UniProbe: A Learnable Token-Level Hallucination Detector for Large VLMs using Multi-Structural Internal Representations [61.85916280854257]
  Large Vision-Language Models (LVLMs) achieve impressive visual reasoning and dialogue capabilities, yet frequently hallucinate content unsupported by the visual input.<n>We introduce textbfUniProbe, a lightweight, unified, learnable detector that models a frozen LVLM's heterogeneous computational trace from a single forward pass.
  arXiv  Detail & Related papers  (2026-08-11T12:01:59Z)
• EnsemHalDet: Robust VLM Hallucination Detection via Ensemble of Internal State Detectors [1.4401311275746886]
  Vision-Language Models (VLMs) excel at multimodal tasks, but they remain vulnerable to hallucinations that are factually incorrect or ungrounded in the input image.<n>Recent work suggests that hallucination detection using internal representations is more efficient and accurate than approaches that rely solely on model outputs.<n>We propose EnsemHalDet, an ensemble-based hallucination detection framework.
  arXiv  Detail & Related papers  (2026-04-03T06:48:27Z)
• Zero-shot HOI Detection with MLLM-based Detector-agnostic Interaction Recognition [71.5328300638085]
  Zero-shot Human-object interaction (HOI) detection aims to locate humans and objects in images and recognize their interactions.<n>Existing methods, including two-stage methods, tightly couple interaction recognition with a specific detector.<n>We propose a decoupled framework that separates object detection from IR and leverages multi-modal large language models (MLLMs) for zero-shot IR.
  arXiv  Detail & Related papers  (2026-02-16T19:01:31Z)
• VIGIL: Tackling Hallucination Detection in Image Recontextualization [0.0]
  We introduce VIGIL, the first benchmark dataset and framework providing a fine-grained categorization of hallucinations.<n>Our work addresses a significant gap in multimodal evaluation by decomposing hallucinations into five categories.<n>Our architecture processes recontextualized images through a series of specialized steps targeting object-level fidelity, background consistency, and omission detection.
  arXiv  Detail & Related papers  (2026-02-16T10:47:10Z)
• Osiris: A Lightweight Open-Source Hallucination Detection System [30.63248848082757]
  hallucinations prevent RAG systems from being deployed in production environments.<n>We introduce a multi-hop QA dataset with induced hallucinations.<n>We achieve better recall with a 7B model than GPT-4o on the RAGTruth hallucination detection benchmark.
  arXiv  Detail & Related papers  (2025-05-07T22:45:59Z)
• HalluCounter: Reference-free LLM Hallucination Detection in the Wild! [6.5037356041929675]
  HalluCounter is a reference-free hallucination detection method that utilizes both response-response and query-response consistency and alignment patterns.<n>Our method outperforms state-of-the-art approaches by a significant margin, achieving over 90% average confidence in hallucination detection across datasets.
  arXiv  Detail & Related papers  (2025-03-06T16:59:18Z)
• A Comprehensive Library for Benchmarking Multi-class Visual Anomaly Detection [89.92916473403108]
  This paper proposes a comprehensive visual anomaly detection benchmark, ADer, which is a modular framework for new methods.<n>The benchmark includes multiple datasets from industrial and medical domains, implementing fifteen state-of-the-art methods and nine comprehensive metrics.<n>We objectively reveal the strengths and weaknesses of different methods and provide insights into the challenges and future directions of multi-class visual anomaly detection.
  arXiv  Detail & Related papers  (2024-06-05T13:40:07Z)
• A New Benchmark and Reverse Validation Method for Passage-level Hallucination Detection [63.56136319976554]
  Large Language Models (LLMs) generate hallucinations, which can cause significant damage when deployed for mission-critical tasks. We propose a self-check approach based on reverse validation to detect factual errors automatically in a zero-resource fashion. We empirically evaluate our method and existing zero-resource detection methods on two datasets.
  arXiv  Detail & Related papers  (2023-10-10T10:14:59Z)
• AutoHall: Automated Hallucination Dataset Generation for Large Language Models [56.92068213969036]
  This paper introduces a method for automatically constructing model-specific hallucination datasets based on existing fact-checking datasets called AutoHall. We also propose a zero-resource and black-box hallucination detection method based on self-contradiction.
  arXiv  Detail & Related papers  (2023-09-30T05:20:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.