Hallucination Inspector: A Fact-Checking Judge for API Migration

• URL: http://arxiv.org/abs/2604.20202v1
• Date: Wed, 22 Apr 2026 05:37:29 GMT
• Title: Hallucination Inspector: A Fact-Checking Judge for API Migration
• Abstract summary: We call this issue Scaffolding Hallucination, a failure mode where models generate incorrect calling contexts.<n>In this paper, we show that standard metrics cannot be relied upon to detect these instances of hallucination.<n>We propose Hallucination Inspector, a static analysis tool to detect Scaffolding Hallucination in LLM-generated code.
• Score: 4.682243274694852
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large Language Models (LLMs) are increasingly deployed in automated software engineering for tasks such as API migration. While LLMs are able to identify migration patterns, they often make mistakes and fail to produce correct glue code to invoke the new API in place of the old one. We call this issue Scaffolding Hallucination, a failure mode where models generate incorrect calling contexts by inventing Phantom Symbols -- such as imaginary imports, constructors, and constants -- that do not exist in the API specification. In this paper, we show that standard metrics cannot be relied upon to detect these instances of hallucination. We propose Hallucination Inspector, a static analysis tool to detect Scaffolding Hallucination in LLM-generated code. Our approach includes a lightweight evaluation framework that verifies symbols extracted from the abstract syntax tree against a knowledge base derived directly from software documentation for the API. A preliminary evaluation on Android API migrations demonstrates that our approach successfully identifies hallucinations and significantly reduces false positives compared to standard metrics and probabilistic judges

Related papers

• Leveraging Low-Level Symbolic Competences for Unsupervised Grounding in Hallucination Detection [6.17293763300421]
  Hallucination detection is difficult due to opaque reasoning processes of language models.<n>In this work, we investigate whether an LLM can use an alternative, low level, symbolic competence such assql for unsupervised hallucination detection.<n>We find that our approach improves on direct prediction and competes with state-of-the-art hallucination detection methods.
  arXiv  Detail & Related papers  (2026-09-04T11:44:11Z)
• Consistency Is the Key: Detecting Hallucinations in LLM Generated Text By Checking Inconsistencies About Key Facts [21.081815261690444]
  Large language models (LLMs) often hallucinate and generate text that is factually incorrect and not grounded in real-world knowledge.<n>This poses serious risks in domains like healthcare, finance, and customer support.<n>We introduce CONFACTCHECK, an efficient detection approach that does not leverage any external knowledge base.
  arXiv  Detail & Related papers  (2025-11-15T14:33:02Z)
• A Systematic Literature Review of Code Hallucinations in LLMs: Characterization, Mitigation Methods, Challenges, and Future Directions for Reliable AI [54.34738767990601]
  As Large Language Models become increasingly integrated into software engineering tasks, understanding and mitigating hallucination in code becomes essential.<n>We provide a systematic review of hallucination phenomena in code-oriented LLMs from four key perspectives.
  arXiv  Detail & Related papers  (2025-11-02T02:58:41Z)
• Mitigating Hallucinations in Multimodal LLMs via Object-aware Preference Optimization [55.543583937522804]
  Multimodal Large Language Models (MLLMs) emerge as a unified interface to address a multitude of tasks.<n>Despite showcasing state-of-the-art results in many benchmarks, a long-standing issue is the tendency of MLLMs to hallucinate.<n>In this paper, we address the problem of hallucinations as an alignment problem, seeking to steer the MLLM so that it prefers generating content without hallucinations.
  arXiv  Detail & Related papers  (2025-08-27T18:02:04Z)
• Generate, but Verify: Reducing Hallucination in Vision-Language Models with Retrospective Resampling [78.78822033285938]
  Vision-Language Models (VLMs) excel at visual understanding but often suffer from visual hallucinations.<n>In this work, we introduce REVERSE, a unified framework that integrates hallucination-aware training with on-the-fly self-verification.
  arXiv  Detail & Related papers  (2025-04-17T17:59:22Z)
• Identifying and Mitigating API Misuse in Large Language Models [26.4403427473915]
  API misuse in code generated by large language models (LLMs) represents a serious emerging challenge in software development.<n>This paper presents the first comprehensive study of API misuse patterns in LLM-generated code, analyzing both method selection and parameter usage across Python and Java.<n>We propose Dr.Fix, a novel LLM-based automatic program repair approach for API misuse based on the aforementioned taxonomy.
  arXiv  Detail & Related papers  (2025-03-28T18:43:12Z)
• ETF: An Entity Tracing Framework for Hallucination Detection in Code Summaries [36.63923402713602]
  Large language models (LLMs) are prone to hallucination, outputs that stray from intended meanings.<n>We introduce a first-of-its-kind dataset, CodeSumEval, with 10K samples, curated specifically for hallucination detection in code summarisation.
  arXiv  Detail & Related papers  (2024-10-17T19:38:55Z)
• MLLM can see? Dynamic Correction Decoding for Hallucination Mitigation [50.73561815838431]
  Multimodal Large Language Models (MLLMs) frequently exhibit hallucination phenomena.<n>We present an empirical analysis and find that, although MLLMs incorrectly generate the objects in the final output, they are actually able to recognize visual objects in the preceding layers.<n>Motivated by this, we propose a novel dynamic correction decoding method for MLLMs DeCo, which adaptively selects the appropriate preceding layers and proportionally integrates knowledge into the final layer to adjust the output logits.
  arXiv  Detail & Related papers  (2024-10-15T16:57:44Z)
• De-Hallucinator: Mitigating LLM Hallucinations in Code Generation Tasks via Iterative Grounding [18.129031749321058]
  Large language models (LLMs) trained on datasets of publicly available source code have established a new state of the art in code generation tasks. LLMs are mostly unaware of the code that exists within a specific project, preventing the models from making good use of existing APIs. This paper presents De-Hallucinator, a technique that grounds the predictions of an LLM through a novel combination of retrieving suitable API references.
  arXiv  Detail & Related papers  (2024-01-03T12:09:43Z)
• A New Benchmark and Reverse Validation Method for Passage-level Hallucination Detection [63.56136319976554]
  Large Language Models (LLMs) generate hallucinations, which can cause significant damage when deployed for mission-critical tasks. We propose a self-check approach based on reverse validation to detect factual errors automatically in a zero-resource fashion. We empirically evaluate our method and existing zero-resource detection methods on two datasets.
  arXiv  Detail & Related papers  (2023-10-10T10:14:59Z)
• A Token-level Reference-free Hallucination Detection Benchmark for Free-form Text Generation [50.55448707570669]
  We propose a novel token-level, reference-free hallucination detection task and an associated annotated dataset named HaDes. To create this dataset, we first perturb a large number of text segments extracted from English language Wikipedia, and then verify these with crowd-sourced annotations.
  arXiv  Detail & Related papers  (2021-04-18T04:09:48Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.