Geo-Spatial Concept Probing of Large Language Models: Abstraction, Compositionality, and Grounding

• URL: http://arxiv.org/abs/2608.07353v1
• Date: Fri, 07 Aug 2026 15:46:38 GMT
• Title: Geo-Spatial Concept Probing of Large Language Models: Abstraction, Compositionality, and Grounding
• Abstract summary: Large Language Models (LLMs) still struggle with genuine concept understanding.<n>We set up a concept-centric benchmark, targeting spatial concepts such as direction, distance, topology, and their compositions.<n>We conduct extensive experiments across multiple LLM architectures and training regimes to analyze how model scale and design impact conceptual understanding.
• Score: 1.6658912537684456
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Understanding concepts is fundamental to generalization. Despite their impressive performance on a wide range of tasks, Large Language Models (LLMs) still struggle with genuine concept understanding. Prior work has evaluated conceptual understanding in LLMs using natural-language benchmarks or narrowly scoped synthetic tasks, but these settings often conflate multiple skills or lack precise control over the underlying concepts and their properties. To support controlled probing of concepts in LLMs, we design tests on their core properties: abstraction, compositionality, and groundness. We set up a concept-centric benchmark, targeting spatial concepts such as direction, distance, topology, and their compositions, and use question answering tasks serving as a proxy. We conduct extensive experiments across multiple LLM architectures and training regimes to analyze how model scale and design impact conceptual understanding. The results reveal clear limitations in current LLMs and provide insights into the factors shaping their ability to acquire and compose structured concepts. Our findings shed light on how concept-based LLMs can be redesigned for improved information access and knowledge management. The code will be available at https://github.com/rd20karim/concept-probing.

Related papers

• Concept Component Analysis: A Principled Approach for Concept Extraction in LLMs [51.378834857406325]
  Mechanistic interpretability seeks to mitigate the issues through extracts from large language models.<n>Sparse autoencoders (SAEs) have emerged as a popular approach for extracting interpretable and monosemantic concepts.<n>We show that SAEs suffer from a fundamental theoretical ambiguity: the well-defined correspondence between LLM representations and human-interpretable concepts remains unclear.
  arXiv  Detail & Related papers  (2026-01-28T09:27:05Z)
• FaCT: Faithful Concept Traces for Explaining Neural Network Decisions [56.796533084868884]
  Deep networks have shown remarkable performance across a wide range of tasks, yet getting a global concept-level understanding of how they function remains a key challenge.<n>We put emphasis on the faithfulness of concept-based explanations and propose a new model with model-inherent mechanistic concept-explanations.<n>Our concepts are shared across classes and, from any layer, their contribution to the logit and their input-visualization can be faithfully traced.
  arXiv  Detail & Related papers  (2025-10-29T13:35:46Z)
• ConceptViz: A Visual Analytics Approach for Exploring Concepts in Large Language Models [18.456737929856125]
  ConceptViz is a visual analytics system designed to explore concepts in large language models (LLMs)<n>Our results show that ConceptViz enhances interpretability research by streamlining the discovery and validation of meaningful concept representations in LLMs.
  arXiv  Detail & Related papers  (2025-09-20T04:57:20Z)
• Probe by Gaming: A Game-based Benchmark for Assessing Conceptual Knowledge in LLMs [17.753896112412942]
  CK-Arena is a multi-agent interaction game built upon the Undercover game.<n>It is designed to evaluate the capacity of Large Language Models to reason with concepts in interactive settings.<n> CK-Arena provides a scalable and realistic benchmark for assessing conceptual reasoning in dynamic environments.
  arXiv  Detail & Related papers  (2025-05-23T06:06:28Z)
• Reasoning about concepts with LLMs: Inconsistencies abound [13.042591838719936]
  Large language models (LLMs) often display and demonstrate significant inconsistencies in their knowledge. In particular, we have been able to significantly enhance the performance of LLMs of various sizes with openly available weights.
  arXiv  Detail & Related papers  (2024-05-30T15:38:54Z)
• Exploring Concept Depth: How Large Language Models Acquire Knowledge and Concept at Different Layers? [57.04803703952721]
  Large language models (LLMs) have shown remarkable performances across a wide range of tasks.<n>However, the mechanisms by which these models encode tasks of varying complexities remain poorly understood.<n>We introduce the idea of "Concept Depth" to suggest that more complex concepts are typically acquired in deeper layers.
  arXiv  Detail & Related papers  (2024-04-10T14:56:40Z)
• Towards Concept-Aware Large Language Models [56.48016300758356]
  Concepts play a pivotal role in various human cognitive functions, including learning, reasoning and communication. There is very little work on endowing machines with the ability to form and reason with concepts. In this work, we analyze how well contemporary large language models (LLMs) capture human concepts and their structure.
  arXiv  Detail & Related papers  (2023-11-03T12:19:22Z)
• COPEN: Probing Conceptual Knowledge in Pre-trained Language Models [60.10147136876669]
  Conceptual knowledge is fundamental to human cognition and knowledge bases. Existing knowledge probing works only focus on factual knowledge of pre-trained language models (PLMs) and ignore conceptual knowledge. We design three tasks to probe whether PLMs organize entities by conceptual similarities, learn conceptual properties, and conceptualize entities in contexts. For the tasks, we collect and annotate 24k data instances covering 393 concepts, which is COPEN, a COnceptual knowledge Probing bENchmark.
  arXiv  Detail & Related papers  (2022-11-08T08:18:06Z)
• Concept Learners for Few-Shot Learning [76.08585517480807]
  We propose COMET, a meta-learning method that improves generalization ability by learning to learn along human-interpretable concept dimensions. We evaluate our model on few-shot tasks from diverse domains, including fine-grained image classification, document categorization and cell type annotation.
  arXiv  Detail & Related papers  (2020-07-14T22:04:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.