The Range Shrinks, the Threat Remains: Re-evaluating LLM Package Hallucinations on the 2026 Frontier-Model Cohort

• URL: http://arxiv.org/abs/2605.17062v1
• Date: Sat, 16 May 2026 16:08:52 GMT
• Title: The Range Shrinks, the Threat Remains: Re-evaluating LLM Package Hallucinations on the 2026 Frontier-Model Cohort
• Abstract summary: Spracklen et al. showed that code-generating large language models hallucinate package names that do not exist on PyPI or npm.<n>We measure hallucination rates across 199,845 paired Python and JavaScript prompts validated against PyPI and npm master lists.<n>We identify a set of 127 package names (109 on PyPI, 18 on npm) that all five evaluated models invent identically.
• Score: 51.56484100374058
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Spracklen et al. (USENIX Security '25) showed that code-generating large language models hallucinate package names that do not exist on PyPI or npm at rates ranging from 5.2% on commercial models to 21.7% on open-source models, creating an attack surface for slopsquatting -- the registration of malicious packages under hallucinated names. We replicate their methodology on five frontier code-capable LLMs released between October 2025 and March 2026: Claude Sonnet 4.6, Claude Haiku 4.5, GPT-5.4-mini, Gemini 2.5 Pro, and DeepSeek V3.2. Across 199,845 paired Python and JavaScript prompts validated against PyPI and npm master lists, we measure overall hallucination rates between 4.62% (Claude Haiku 4.5) and 6.10% (GPT-5.4-mini) -- an order-of-magnitude compression of the inter-model spread observed by Spracklen, but not a retirement of the threat. Beyond replication, we identify a set of 127 package names (109 on PyPI, 18 on npm) that all five evaluated models invent identically, constituting a model-agnostic supply-chain attack surface that no single-model study can reveal. We further document a Python-over-JavaScript hallucination asymmetry that inverts Spracklen's 2024 finding, identify a Haiku-below-Sonnet inversion within the Anthropic family, and observe a Jaccard-similarity peak between DeepSeek V3.2 and GPT-5.4-mini (J = 0.343) suggestive of shared training-data origins.

Related papers

• CogEvol: Towards Efficient and Reliable Learning Environment Generation [49.18054685846454]
  We present CogEvol, a family of models trained specifically for Learning Environment Generation.<n>CogEvol completes a slide in a median of 17 seconds and an interactive page in 59, replacing minutes-long multi-turn agent scaffolding.<n>A production-grounded data pipeline turns real failures into 53,687 verified SFT samples.
  arXiv  Detail & Related papers  (2026-08-31T15:33:38Z)
• Names Can Hurt: Spotting Slopsquatting Risks Caused by Package Name Hallucinations in Local Coding LLMs [0.0]
  When a code generating language model fabricates a Python package name, an adversary who has pre-registered that name on PyPI can convert that hallucination into a supply chain compromise.<n>We propose a two layer detector to counter this issue.<n>The detector is embedded in a LangGraph state machine that retries at escalating temperatures and, on repeated failure, routes to a stronger fallback model.
  arXiv  Detail & Related papers  (2026-08-24T23:10:49Z)
• Who Gets Named: Citation Type Predicts Individual Naming by Grounded Language Models, and a Roster Instrument Captures 0.5% of It [0.0]
  It issued 2,400 grounded API calls in one two-hour window on 24 July 2026.<n>Every response was coded for whether it named an individual professional.<n>Models named an individual in 25.8% of responses.
  arXiv  Detail & Related papers  (2026-07-26T23:37:34Z)
• Opaque Epistemic Mediation: How LLM Deployment Configurations Shape the Validation of Pseudo-Science [0.0]
  Commercial large language models are increasingly used as knowledge references, yet their stance on contested scientific claims is neither stable nor transparent.<n>We tested how four major LLM families evaluate ethnonationalist pseudo-science derived from Frank Salter's biosocial framework.<n>Grok's Fast versions consistently assigned credibility scores of 70-75, two to five times higher than all other models.
  arXiv  Detail & Related papers  (2026-07-24T17:32:43Z)
• Knowledge Index of Noah's Ark [63.143852586221534]
  We introduce KINA, an 899-item benchmark across 261 fine-grained disciplines.<n>We prove a bonus-on-bar tournament weakly FOSD-dominates flat payment in released-review quality.<n>The top model, Gemini-3.1-Pro-Preview, reaches 53.17%, followed by Claude-Opus-4.6 at 49.92% and GPT-5.4 at 48.55%.
  arXiv  Detail & Related papers  (2026-06-03T17:06:49Z)
• How Well Do Models Follow Their Constitutions? [11.533708129682887]
  We propose a multi-method audit pipeline that treats each lab's published specification as an auditable target.<n>We find that models follow their own lab's specification substantially better with each generation.
  arXiv  Detail & Related papers  (2026-05-22T21:17:16Z)
• How Far Is Document Parsing from Solved? PureDocBench: A Source-TraceableBenchmark across Clean, Degraded, and Real-World Settings [56.70440596502351]
  The past year has seen over 20 open document parsing models, yet benchmarks almost exclusively on OmniDocBench.<n>We present PureDocBench, a benchmark thatrenders document images from HTML/CSS, covering 10 domains, 66 pages, and 1,475 pages.
  arXiv  Detail & Related papers  (2026-05-08T09:30:31Z)
• Correct Code, Vulnerable Dependencies: A Large Scale Measurement Study of LLM-Specified Library Versions [52.50730821321986]
  We present the first large-scale measurement study of version-level risk in large language models (LLMs)<n>We evaluate 10 LLMs on PinTrace, a curated benchmark of 1,000 Stack Overflow programming tasks.<n>Our findings reveal LLM version selection as a first-class, previously overlooked risk surface in LLM-based development.
  arXiv  Detail & Related papers  (2026-05-07T13:52:59Z)
• ZenBrain: A Neuroscience-Inspired 7-Layer Memory Architecture for Autonomous AI Systems [51.56484100374058]
  On LongMemEval-500, ZenBrain matches a long-context oracle's binary-judge accuracy to within 4.5 pp.<n> ZenBrain is a 7-layer neuroscience-inspired memory architecture.
  arXiv  Detail & Related papers  (2026-04-26T20:39:19Z)
• Many Hands Make Light Work: An LLM-based Multi-Agent System for Detecting Malicious PyPI Packages [3.7667883869699597]
  Malicious code in open-source repositories such as PyPI poses a growing threat to software supply chains.<n>This paper presents LAMPS, a multi-agent system that employs collaborative language models to detect malicious PyPI packages.
  arXiv  Detail & Related papers  (2026-01-17T19:43:22Z)
• AI Transparency Atlas: Framework, Scoring, and Real-Time Model Card Evaluation Pipeline [2.1787849426740364]
  We analyzed documentation from five frontier models (Gemini 3, Grok 4.1, Llama 4, GPT-5, and Claude 4.5) and 100 Hugging Face model cards.<n>We developed a weighted transparency framework with 8 sections and 23 subsections that prioritizes safety-critical disclosures.
  arXiv  Detail & Related papers  (2025-12-13T19:48:44Z)
• How Deep Does Your Dependency Tree Go? An Empirical Study of Dependency Amplification Across 10 Package Ecosystems [0.0]
  We study 500 projects across ten major ecosystems, including Maven, npm Registry, RubyGems, Go Modules, PyPI, CocoaPods, and Pub.<n>We find significant differences with large effect sizes in 22 of 45 pairwise comparisons, challenging the assumption that npm has the highest amplification.<n>Our findings suggest adopting ecosystem specific security strategies, including systematic auditing for Maven environments, targeted outlier detection for npm and RubyGems, and continuation of current practices for ecosystems with controlled amplification.
  arXiv  Detail & Related papers  (2025-12-12T05:53:32Z)
• We Have a Package for You! A Comprehensive Analysis of Package Hallucinations by Code Generating LLMs [3.515912713354746]
  Package hallucinations arise from fact-conflicting errors when generating code using Large Language Models.<n>This paper conducts a rigorous and comprehensive evaluation of package hallucinations across different programming languages.<n>We show that the average percentage of hallucinated packages is at least 5.2% for commercial models and 21.7% for open-source models.
  arXiv  Detail & Related papers  (2024-06-12T03:29:06Z)
• Prometheus: Inducing Fine-grained Evaluation Capability in Language Models [66.12432440863816]
  We propose Prometheus, a fully open-source Large Language Model (LLM) that is on par with GPT-4's evaluation capabilities. Prometheus scores a Pearson correlation of 0.897 with human evaluators when evaluating with 45 customized score rubrics. Prometheus achieves the highest accuracy on two human preference benchmarks.
  arXiv  Detail & Related papers  (2023-10-12T16:50:08Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.