KnowledgeBerg: Evaluating Systematic Knowledge Coverage and Compositional Reasoning in Large Language Models

• URL: http://arxiv.org/abs/2604.17621v1
• Date: Sun, 19 Apr 2026 21:18:42 GMT
• Title: KnowledgeBerg: Evaluating Systematic Knowledge Coverage and Compositional Reasoning in Large Language Models
• Abstract summary: Many real-world questions appear deceptively simple yet implicitly demand two capabilities.<n>We formalize this challenge through two dimensions: knowledge width, the cardinality of the required universe, and reasoning depth.<n>We introduce KnowledgeBerg, a benchmark of 4,800 multiple-choice questions from 1,183 enumeration seeds spanning 10 domains and 17 languages.
• Score: 8.584257473147568
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Many real-world questions appear deceptively simple yet implicitly demand two capabilities: (i) systematic coverage of a bounded knowledge universe and (ii) compositional set-based reasoning over that universe, a phenomenon we term "the tip of the iceberg." We formalize this challenge through two orthogonal dimensions: knowledge width, the cardinality of the required universe, and reasoning depth, the number of compositional set operations. We introduce KnowledgeBerg, a benchmark of 4,800 multiple-choice questions derived from 1,183 enumeration seeds spanning 10 domains and 17 languages, with universes grounded in authoritative sources to ensure reproducibility. Representative open-source LLMs demonstrate severe limitations, achieving only 5.26-36.88 F1 on universe enumeration and 16.00-44.19 accuracy on knowledge-grounded reasoning. Diagnostic analyses reveal three stages of failure: completeness, or missing knowledge; awareness, or failure to identify requirements; and application, or incorrect reasoning execution. This pattern persists across languages and model scales. Although test-time compute and retrieval augmentation yield measurable gains -- up to 4.35 and 3.78 points, respectively -- substantial gaps remain, exposing limitations in how current LLMs organize structured knowledge and execute compositional reasoning over bounded domains. The dataset is available at https://huggingface.co/datasets/2npc/KnowledgeBerg

Related papers

• On the Fundamental Limits of LLMs at Scale [15.459708840379975]
  Large Language Models (LLMs) have benefited enormously from scaling, yet these gains are bounded by five fundamental limitations.<n>This work presents a unified, proof-informed framework that formalizes the innate theoretical ceilings of LLM scaling.<n>We show how likelihood-based training favors pattern completion over inference, how retrieval under token limits suffers from semantic drift and coupling noise, and how multimodal scaling inherits shallow cross-modal alignment.
  arXiv  Detail & Related papers  (2025-11-17T01:55:33Z)
• Knowledge Homophily in Large Language Models [75.12297135039776]
  We investigate an analogous knowledge homophily pattern in Large Language Models (LLMs)<n>We map LLM knowledge into a graph representation through knowledge checking at both the triplet and entity levels.<n>Motivated by this homophily principle, we propose a Graph Neural Network (GNN) regression model to estimate entity-level knowledgeability scores for triplets.
  arXiv  Detail & Related papers  (2025-09-28T09:40:27Z)
• MDK12-Bench: A Comprehensive Evaluation of Multimodal Large Language Models on Multidisciplinary Exams [50.293164501645975]
  Multimodal large language models (MLLMs) integrate language and visual cues for problem-solving.<n>Current benchmarks for measuring the intelligence of MLLMs suffer from limited scale, narrow coverage, and unstructured knowledge.<n>We introduce MDK12-Bench, a large-scale multidisciplinary benchmark built from real-world K-12 exams spanning six disciplines.
  arXiv  Detail & Related papers  (2025-08-09T06:21:10Z)
• OneEval: Benchmarking LLM Knowledge-intensive Reasoning over Diverse Knowledge Bases [38.58409057214189]
  textbftextscOneEval is a benchmark to assess the knowledge-intensive reasoning capabilities of Large Language Models (LLMs)<n>textscOneEval comprises 4,019 carefully curated instances and includes a challenging subset, textscOneEvaltextsubscriptHard, consisting of 1,285 particularly difficult cases.<n>We release the textscOneEval datasets, evaluation scripts, and baseline results publicly, accompanied by a leaderboard to facilitate ongoing advancements in structured knowledge reasoning.
  arXiv  Detail & Related papers  (2025-06-14T17:16:05Z)
• Analyzing LLMs' Knowledge Boundary Cognition Across Languages Through the Lens of Internal Representations [72.62400923539234]
  We present the first study to analyze how LLMs recognize knowledge boundaries across different languages.<n>Our empirical studies reveal three key findings: 1) LLMs' perceptions of knowledge boundaries are encoded in the middle to middle-upper layers across different languages.
  arXiv  Detail & Related papers  (2025-04-18T17:44:12Z)
• Inside-Out: Hidden Factual Knowledge in LLMs [50.79758420289131]
  This work presents a framework for assessing whether large language models (LLMs) encode more factual knowledge in their parameters than what they express in their outputs.<n>We first propose a formal definition of knowledge, quantifying it for a given question as the fraction of correct-incorrect answer pairs where the correct one is ranked higher.<n>We then present a case study, applying this framework to three popular open-weights LLMs in a closed-book QA setup.
  arXiv  Detail & Related papers  (2025-03-19T15:21:48Z)
• Knowledge Boundary of Large Language Models: A Survey [75.67848187449418]
  Large language models (LLMs) store vast amount of knowledge in their parameters, but they still have limitations in the memorization and utilization of certain knowledge.<n>This highlights the critical need to understand the knowledge boundary of LLMs, a concept that remains inadequately defined in existing research.<n>We propose a comprehensive definition of the LLM knowledge boundary and introduce a formalized taxonomy categorizing knowledge into four distinct types.
  arXiv  Detail & Related papers  (2024-12-17T02:14:02Z)
• ChroKnowledge: Unveiling Chronological Knowledge of Language Models in Multiple Domains [19.428141279030527]
  ChroKnowBench is a benchmark dataset designed to evaluate chronologically accumulated knowledge.<n>ChroKnowledge is a novel sampling-based framework for evaluating LLMs' non-parametric chronological knowledge.<n>ChroKnowPrompt is an in-depth prompting to elicit chronological knowledge by traversing step-by-step through the surrounding time spans.
  arXiv  Detail & Related papers  (2024-10-13T15:08:49Z)
• Knowledge Crosswords: Geometric Knowledge Reasoning with Large Language Models [49.23348672822087]
  We propose Knowledge Crosswords, a benchmark consisting of incomplete knowledge networks bounded by structured factual constraints. The novel setting of geometric knowledge reasoning necessitates new LM abilities beyond existing atomic/linear multi-hop QA. We conduct extensive experiments to evaluate existing LLMs and approaches on Knowledge Crosswords.
  arXiv  Detail & Related papers  (2023-10-02T15:43:53Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.