LLMPEDIA: Browsing, Verifying, and Comparing the Parametric Encyclopedic Knowledge of LLMs

• URL: http://arxiv.org/abs/2609.01182v1
• Date: Tue, 01 Sep 2026 12:56:02 GMT
• Title: LLMPEDIA: Browsing, Verifying, and Comparing the Parametric Encyclopedic Knowledge of LLMs
• Abstract summary: LLMPEDIA lets visitors inspect this frontier one claim at a time through five one-click views.<n>On a uniform random sample the true rate is 68.4% - more than 21 pp below MMLU.<n>The resulting live, open encyclopedia lets visitors inspect this frontier one claim at a time through five one-click views.
• Score: 10.223945061944521
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Flagship language models appear saturated on benchmarks like MMLU (Hendrycks et al., 2021), scoring above 90% - yet benchmarks test only what the experimenter thought to ask, the availability bias of fixed question sets. LLMPEDIA makes this bias measurable and browsable. We recursively materialized ~1.3M articles from three model families' parametric memory (GPT-5-mini, DeepSeek-V3.2, Llama-3.3-70B) without retrieval, then audited a stratified sample of atomic claims against Wikipedia and a curated web stack, coloring every claim supported, refuted, or insufficient (Saeed and Razniewski, 2026). On a uniform random sample the true rate is 68.4% - more than 21 pp below MMLU - with 30.5% of claims insufficient: assertions no benchmark probes and the world's largest encyclopedia cannot adjudicate - long-tail knowledge or plausible hallucination, the evidence cannot tell - extending to free text the coverage gap GPTKB established for triples (Hu et al., 2025). The resulting live, open encyclopedia lets visitors inspect this frontier one claim at a time through five one-click views - link-traversal exploration, claim-level factuality, cross-model and political-persona comparison, and a guided topic drill-down - each page, claim, and verdict at a stable URL. LLMPEDIA is live at https://llmpedia.net

Related papers

• NovGauge: A Fine-Grained Benchmark for Diagnosing LLMs' Capability in Paper Novelty Assessment [54.4265627247104]
  Large language models (LLMs) are increasingly used in peer review at major AI conferences.<n>Existing benchmarks assess novelty as a single holistic score.<n>We present NovGauge, a human-anchored benchmark for fine-grained novelty assessment diagnosis.
  arXiv  Detail & Related papers  (2026-09-10T08:34:56Z)
• BeTaL-GBI: Admission-Aware Benchmark Tuning and Full-Stack Verification of Geometric Belief Interfaces [2.894286977279531]
  This study examines whether an enterprise verification architecture can isolate interface failure, task competence, policy admissibility, and control integrity while keeping claims auditable.<n>BeTaL-GBI v0.2 applies Benchmark Tuning with an LLM-in-the-loop over 2,218,750,380 grid points, separating format admission from conditional performance.<n>Across 512 synthetic tasks, a 16-gate policy detects all 116 injected severe contradictions and accepts all 99 clean records. Hallucinator and evidence-forger surrogates are blocked with zero silent promotions.
  arXiv  Detail & Related papers  (2026-08-21T16:52:38Z)
• Opaque Epistemic Mediation: How LLM Deployment Configurations Shape the Validation of Pseudo-Science [0.0]
  Commercial large language models are increasingly used as knowledge references, yet their stance on contested scientific claims is neither stable nor transparent.<n>We tested how four major LLM families evaluate ethnonationalist pseudo-science derived from Frank Salter's biosocial framework.<n>Grok's Fast versions consistently assigned credibility scores of 70-75, two to five times higher than all other models.
  arXiv  Detail & Related papers  (2026-07-24T17:32:43Z)
• Before the Action: Benchmarking LLMs on Prospective Hypothesis Discovery [95.44432473367836]
  Large language models (LLMs) excel at answering pre-specified questions, yet their ability to navigate the open-ended, pre-conclusion stage of discovery remains largely unmeasured.<n>We introduce Prospective Hypothesis Discovery (PHD), which asks models to autonomously construct grounded, discnative, and testable hypothesis spaces from inconclusive evidence.<n>We introduce HypoArena, comprising HypoData, a benchmark of 988 cases across six scientific and analytical domains, and HypoEval, an evaluation framework for open-ended hypothesis sets.
  arXiv  Detail & Related papers  (2026-07-17T08:56:43Z)
• Can Large Language Models Revolutionize Survey Research? Experiments with Disaster Preparedness Responses [6.004875368104112]
  Large language models (LLMs) have been proposed as a remedy, yet rigorous evaluations across the full survey workflow remain scarce.<n>We present and evaluate a five-stage framework for LLM integration covering questionnaire design, sample selection, pilot testing, missing-data imputation, and post-collection analysis.<n>We introduce a Protection Motivation Theory (PMT)-constrained co-occurrence knowledge graph and develop seven LLM configurations spanning zero-shot inference, retrieval-augmented baselines, and novel theory-informed variants.
  arXiv  Detail & Related papers  (2026-05-19T00:58:36Z)
• How Far Is Document Parsing from Solved? PureDocBench: A Source-TraceableBenchmark across Clean, Degraded, and Real-World Settings [56.70440596502351]
  The past year has seen over 20 open document parsing models, yet benchmarks almost exclusively on OmniDocBench.<n>We present PureDocBench, a benchmark thatrenders document images from HTML/CSS, covering 10 domains, 66 pages, and 1,475 pages.
  arXiv  Detail & Related papers  (2026-05-08T09:30:31Z)
• LLMpedia: A Transparent Framework to Materialize an LLM's Encyclopedic Knowledge at Scale [10.223945061944521]
  LLMpedia generates encyclopedic articles entirely from parametric memory.<n>The verifiable true rate on Wikipedia-covered subjects is only 74.7%.<n>Unlike Grokipedia, every prompt, artifact, and evaluation verdict is publicly released.
  arXiv  Detail & Related papers  (2026-03-25T08:37:26Z)
• One Thousand and One Pairs: A "novel" challenge for long-context language models [56.60667988954638]
  NoCha is a dataset of 1,001 pairs of true and false claims about 67 fictional books. Our annotators confirm that the largest share of pairs in NoCha require global reasoning over the entire book to verify. On average, models perform much better on pairs that require only sentence-level retrieval vs. global reasoning.
  arXiv  Detail & Related papers  (2024-06-24T02:03:57Z)
• Long-form factuality in large language models [60.07181269469043]
  Large language models (LLMs) often generate content that contains factual errors when responding to fact-seeking prompts on open-ended topics. We benchmark a model's long-form factuality in open domains, using GPT-4 to generate LongFact. We then propose that LLM agents can be used as automated evaluators for long-form factuality through a method which we call Search-Augmented Factuality Evaluator (SAFE)
  arXiv  Detail & Related papers  (2024-03-27T17:48:55Z)
• LV-Eval: A Balanced Long-Context Benchmark with 5 Length Levels Up to 256K [57.0244259406764]
  This paper introduces LV-Eval, a challenging long-context benchmark with five length levels reaching up to 256k words.<n>The advantages of LV-Eval include controllable evaluation across different context lengths, challenging test instances with confusing facts, and more objective evaluations.
  arXiv  Detail & Related papers  (2024-02-06T13:11:19Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.