SpaRC and SpaRP: Spatial Reasoning Characterization and Path Generation for Understanding Spatial Reasoning Capability of Large Language Models

• URL: http://arxiv.org/abs/2406.04566v1
• Date: Fri, 7 Jun 2024 01:06:34 GMT
• Title: SpaRC and SpaRP: Spatial Reasoning Characterization and Path Generation for Understanding Spatial Reasoning Capability of Large Language Models
• Authors: Md Imbesat Hassan Rizvi, Xiaodan Zhu, Iryna Gurevych,
• Abstract summary: spatial reasoning is a crucial component of both biological and artificial intelligence. We present a comprehensive study of the capability of current state-of-the-art large language models (LLMs) on spatial reasoning.
• Score: 70.01883340129204
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Spatial reasoning is a crucial component of both biological and artificial intelligence. In this work, we present a comprehensive study of the capability of current state-of-the-art large language models (LLMs) on spatial reasoning. To support our study, we created and contribute a novel Spatial Reasoning Characterization (SpaRC) framework and Spatial Reasoning Paths (SpaRP) datasets, to enable an in-depth understanding of the spatial relations and compositions as well as the usefulness of spatial reasoning chains. We found that all the state-of-the-art LLMs do not perform well on the datasets -- their performances are consistently low across different setups. The spatial reasoning capability improves substantially as model sizes scale up. Finetuning both large language models (e.g., Llama-2-70B) and smaller ones (e.g., Llama-2-13B) can significantly improve their F1-scores by 7--32 absolute points. We also found that the top proprietary LLMs still significantly outperform their open-source counterparts in topological spatial understanding and reasoning.

Related papers

• SpatialCoT: Advancing Spatial Reasoning through Coordinate Alignment and Chain-of-Thought for Embodied Task Planning [42.487500113839666]
  We propose a novel approach to bolster the spatial reasoning capabilities of Vision-Language Models (VLMs) Our approach comprises two stages: spatial coordinate bi-directional alignment, and chain-of-thought spatial grounding. We evaluate our method on challenging navigation and manipulation tasks, both in simulation and real-world settings.
  arXiv  Detail & Related papers  (2025-01-17T09:46:27Z)
• Adaptive Pruning for Large Language Models with Structural Importance Awareness [66.2690963378878]
  Large language models (LLMs) have significantly improved language understanding and generation capabilities. LLMs are difficult to deploy on resource-constrained edge devices due to their high computational and storage resource demands. We propose structurally-aware adaptive pruning (SAAP) to significantly reduce the computational and memory costs while maintaining model performance.
  arXiv  Detail & Related papers  (2024-12-19T18:08:04Z)
• SPHERE: Unveiling Spatial Blind Spots in Vision-Language Models Through Hierarchical Evaluation [7.659514491338669]
  Current vision-language models may grasp basic spatial cues but struggle with the multi-dimensional spatial reasoning necessary for human-like understanding and real-world applications. We develop SPHERE, a hierarchical evaluation framework supported by a new human-annotated dataset. Benchmark evaluation of state-of-the-art models reveals significant deficiencies, especially in reasoning about distance and proximity. These findings expose critical blind spots in existing models and underscore the need for more advanced spatial reasoning techniques.
  arXiv  Detail & Related papers  (2024-12-17T09:10:55Z)
• An Empirical Analysis on Spatial Reasoning Capabilities of Large Multimodal Models [56.537253374781876]
  Large Multimodal Models (LMMs) have achieved strong performance across a range of vision and language tasks. However, their spatial reasoning capabilities are under-investigated. We construct a novel VQA dataset, Spatial-MM, to comprehensively study LMMs' spatial understanding and reasoning capabilities.
  arXiv  Detail & Related papers  (2024-11-09T03:07:33Z)
• Language Agents Meet Causality -- Bridging LLMs and Causal World Models [50.79984529172807]
  We propose a framework that integrates causal representation learning with large language models. This framework learns a causal world model, with causal variables linked to natural language expressions. We evaluate the framework on causal inference and planning tasks across temporal scales and environmental complexities.
  arXiv  Detail & Related papers  (2024-10-25T18:36:37Z)
• Sparkle: Mastering Basic Spatial Capabilities in Vision Language Models Elicits Generalization to Composite Spatial Reasoning [19.399925987942204]
  Vision language models (VLMs) have demonstrated impressive performance across a wide range of downstream tasks. Our evaluation reveals that state-of-the-art VLMs frequently generate implausible and incorrect responses to composite spatial reasoning problems. To address this, we explore an effective approach to enhance 2D spatial reasoning within VLMs by training the model solely on basic spatial capabilities.
  arXiv  Detail & Related papers  (2024-10-21T16:26:09Z)
• Exploring Language Model Generalization in Low-Resource Extractive QA [57.14068405860034]
  We investigate Extractive Question Answering (EQA) with Large Language Models (LLMs) under domain drift. We devise a series of experiments to explain the performance gap empirically.
  arXiv  Detail & Related papers  (2024-09-27T05:06:43Z)
• SpatialRGPT: Grounded Spatial Reasoning in Vision Language Models [68.13636352687257]
  We introduce Spatial Region GPT (SpatialRGPT) to enhance VLMs' spatial perception and reasoning capabilities. During inference, when provided with user-specified region proposals, SpatialRGPT can accurately perceive their relative directions and distances. Our results demonstrate that SpatialRGPT significantly enhances performance in spatial reasoning tasks, both with and without local region prompts.
  arXiv  Detail & Related papers  (2024-06-03T17:59:06Z)
• More than Correlation: Do Large Language Models Learn Causal Representations of Space? [6.293100288400849]
  This study focused on uncovering the causality of the spatial representations in large language models. Experiments showed that the spatial representations influenced the model's performance on next word prediction and a downstream task that relies on geospatial information.
  arXiv  Detail & Related papers  (2023-12-26T01:27:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.