Evaluation of Multilingual Ability to Use Spatial Deictic Expressions in Vision-Language Models

• URL: http://arxiv.org/abs/2607.07251v1
• Date: Wed, 08 Jul 2026 10:34:54 GMT
• Title: Evaluation of Multilingual Ability to Use Spatial Deictic Expressions in Vision-Language Models
• Abstract summary: This paper develops a benchmark to evaluate the multilingual ability of vision-language models to use spatial deictic expressions.<n>Our experiments reveal that the tested models use demonstratives in a manner different from that of humans.
• Score: 14.511507458575542
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: One of the expected abilities of vision-language models (VLMs) is spatial reasoning ability based on a given text and image. To evaluate the spatial reasoning abilities of VLMs, we focus on the use of spatial deictic expressions, which are defined as spatial expressions whose referent is determined by their situational context, such as ``this'' and ``that''. To handle spatial deictic expressions, VLMs must jointly reason over language and visual space, grounding context-dependent references in the image's spatial structure. In addition, selecting appropriate spatial deictic expressions across languages requires VLMs to understand the language-specific spatial distinctions encoded by these expressions. In this paper, we develop a benchmark to evaluate the multilingual ability of VLMs to use spatial deictic expressions in four languages. Our experiments using this benchmark reveal that the tested models use demonstratives in a manner different from that of humans, particularly in selecting the appropriate demonstratives based on the distance to the object.

Related papers

• Contextual Observer Grounding: Evaluating Situated Spatial Reasoning in Vision-Language Models [22.823647299309712]
  Reasoning over language instructions in embodied tasks often requires understanding spatial relations from a speaker's situated perspective.<n>Recent vision-language models (VLMs) appear capable of spatial reasoning, but their ability to infer a speaker's viewpoint from contextual cues remains unclear.
  arXiv  Detail & Related papers  (2026-09-07T00:00:13Z)
• OmniGF: A Dual-Branch Vision-Language Framework for Unified Gaze Following [59.53720386342017]
  We propose a unified vision-language framework that adapts foundational Vision-Language Models for scalable multi-person gaze reasoning.<n>By modeling all individuals, OmniGF seamlessly integrates precise spatial gaze target estimation, semantic gaze prediction, and complex social gaze reasoning.
  arXiv  Detail & Related papers  (2026-05-26T00:08:06Z)
• Referring Expressions as a Lens into Spatial Language Grounding in Vision-Language Models [22.672848927658958]
  We propose using the Referring Expression task as a platform for the evaluation of spatial reasoning by Vision-language models (VLMs)<n>This platform provides the opportunity for a deeper analysis of spatial comprehension and grounding abilities when there is 1) ambiguity in object detection, 2) complex spatial expressions with a longer sentence structure and multiple spatial relations, and 3) expressions with negation ('not')<n>Our results highlight these challenges and behaviors and provide insight into research gaps and future directions.
  arXiv  Detail & Related papers  (2025-11-08T21:43:09Z)
• A Neural Representation Framework with LLM-Driven Spatial Reasoning for Open-Vocabulary 3D Visual Grounding [78.99798110890157]
  Open-vocabulary 3D visual grounding aims to localize target objects based on free-form language queries.<n>Existing language field methods struggle to accurately localize instances using spatial relations in language queries.<n>We propose SpatialReasoner, a novel neural representation-based framework with large language model (LLM)-driven spatial reasoning.
  arXiv  Detail & Related papers  (2025-07-09T10:20:38Z)
• Mind the Gap: Benchmarking Spatial Reasoning in Vision-Language Models [14.442394137843923]
  We present a detailed analysis that first delineates the core elements of spatial reasoning.<n>We then assesses the performance of these models in both synthetic and real-world images.
  arXiv  Detail & Related papers  (2025-03-25T14:34:06Z)
• LayoutVLM: Differentiable Optimization of 3D Layout via Vision-Language Models [57.92316645992816]
  Spatial reasoning is a fundamental aspect of human cognition, enabling intuitive understanding and manipulation of objects in three-dimensional space.<n>We introduce LayoutVLM, a framework and scene layout representation that exploits the semantic knowledge of Vision-Language Models (VLMs)<n>We demonstrate that fine-tuning VLMs with the proposed scene layout representation extracted from existing scene datasets can improve their reasoning performance.
  arXiv  Detail & Related papers  (2024-12-03T06:15:04Z)
• Do Vision-Language Models Represent Space and How? Evaluating Spatial Frame of Reference Under Ambiguities [27.940469021840745]
  We present an evaluation protocol to assess the spatial reasoning capabilities of vision-language models (VLMs)<n>Despite some alignment with English conventions in resolving ambiguities, our experiments reveal significant shortcomings of VLMs.<n>With a growing effort to align vision-language models with human cognitive intuitions, we call for more attention to the ambiguous nature and cross-cultural diversity of spatial reasoning.
  arXiv  Detail & Related papers  (2024-10-22T19:39:15Z)
• Integrating Language Guidance into Vision-based Deep Metric Learning [78.18860829585182]
  We propose to learn metric spaces which encode semantic similarities as embedding space. These spaces should be transferable to classes beyond those seen during training. This causes learned embedding spaces to encode incomplete semantic context and misrepresent the semantic relation between classes.
  arXiv  Detail & Related papers  (2022-03-16T11:06:50Z)
• Spatial Language Understanding for Object Search in Partially Observed Cityscale Environments [21.528770932332474]
  We introduce the spatial language observation space and formulate a model under the framework of Partially Observable Markov Decision Process (POMDP) We propose a convolutional neural network model that learns to predict the language provider's relative frame of reference (FoR) given environment context. We demonstrate the generalizability of our FoR prediction model and object search system through cross-validation over areas of five cities, each with a 40,000m$2$ footprint.
  arXiv  Detail & Related papers  (2020-12-04T16:27:59Z)
• Learning Universal Representations from Word to Sentence [89.82415322763475]
  This work introduces and explores the universal representation learning, i.e., embeddings of different levels of linguistic unit in a uniform vector space. We present our approach of constructing analogy datasets in terms of words, phrases and sentences. We empirically verify that well pre-trained Transformer models incorporated with appropriate training settings may effectively yield universal representation.
  arXiv  Detail & Related papers  (2020-09-10T03:53:18Z)
• From Spatial Relations to Spatial Configurations [64.21025426604274]
  spatial relation language is able to represent a large, comprehensive set of spatial concepts crucial for reasoning. We show how we extend the capabilities of existing spatial representation languages with the fine-grained decomposition of semantics.
  arXiv  Detail & Related papers  (2020-07-19T02:11:53Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.