SceneFunRI: Reasoning the Invisible for Task-Driven Functional Object Localization

• URL: http://arxiv.org/abs/2605.14704v1
• Date: Thu, 14 May 2026 11:21:41 GMT
• Title: SceneFunRI: Reasoning the Invisible for Task-Driven Functional Object Localization
• Abstract summary: We introduce SceneFunRI, a benchmark for Reasoning the Invisible.<n>Based on the SceneFun3D dataset, SceneFunRI formulates the task as a 2D spatial reasoning problem.<n>It requires models to infer the locations of invisible functional objects from task instructions and commonsense reasoning.
• Score: 19.363096374067403
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: In real-world scenes, target objects may reside in regions that are not visible. While humans can often infer the locations of occluded objects from context and commonsense knowledge, this capability remains a major challenge for vision-language models (VLMs). To address this gap, we introduce SceneFunRI, a benchmark for Reasoning the Invisible. Based on the SceneFun3D dataset, SceneFunRI formulates the task as a 2D spatial reasoning problem via a semi-automatic pipeline and comprises 855 instances. It requires models to infer the locations of invisible functional objects from task instructions and commonsense reasoning. The strongest baseline model (Gemini 3 Flash) only achieves an CAcc@75 of 15.20, an mIoU of 0.74, and a Dist of 28.65. We group our prompting analysis into three categories: Strong Instruction Prompting, Reasoning-based Prompting, and Spatial Process of Elimination (SPoE). These findings indicate that invisible-region reasoning remains an unstable capability in current VLMs, motivating future work on models that more tightly integrate task intent, commonsense priors, spatial grounding, and uncertainty-aware search.

Related papers

• SceneBench: A Hierarchical Benchmark for Vision-Language Understanding of 3D Scenes [12.672496253081258]
  Vision-language models excel at 2D image understanding but remain limited in 3D spatial reasoning.<n>We introduce SceneBench, a benchmark of 966 photorealistic 3D scenes reconstructed with Gaussian Splatting.<n>SceneBench provides a realistic testbed for developing and evaluating models capable of fine-grained spatial reasoning.
  arXiv  Detail & Related papers  (2026-09-14T18:59:13Z)
• SceneTeract: Agentic Functional Affordances and VLM Grounding in 3D Scenes [49.117927512531715]
  We introduce SceneTeract, a framework that verifies 3D scene functionality under agent-specific constraints.<n> SceneTeract decomposes complex activities into sequences of atomic actions and validates each step against accessibility requirements.<n>We release the SceneTeract verification suite and data to bridge perception and physical reality in embodied 3D scene understanding.
  arXiv  Detail & Related papers  (2026-03-31T14:31:18Z)
• SpatiaLQA: A Benchmark for Evaluating Spatial Logical Reasoning in Vision-Language Models [60.088066516175026]
  We introduce a benchmark designed to evaluate the spatial logical reasoning capabilities of Vision-Language Models (VLMs)<n>We conduct extensive experiments on 41 mainstream VLMs, and the results show that even the most advanced models still struggle with spatial logical reasoning.<n>We propose a method called recursive scene graph assisted reasoning, which leverages visual foundation models to progressively decompose complex scenes into task-relevant scene graphs.
  arXiv  Detail & Related papers  (2026-02-24T13:38:37Z)
• Referring Expressions as a Lens into Spatial Language Grounding in Vision-Language Models [22.672848927658958]
  We propose using the Referring Expression task as a platform for the evaluation of spatial reasoning by Vision-language models (VLMs)<n>This platform provides the opportunity for a deeper analysis of spatial comprehension and grounding abilities when there is 1) ambiguity in object detection, 2) complex spatial expressions with a longer sentence structure and multiple spatial relations, and 3) expressions with negation ('not')<n>Our results highlight these challenges and behaviors and provide insight into research gaps and future directions.
  arXiv  Detail & Related papers  (2025-11-08T21:43:09Z)
• Spatial Reasoning in Foundation Models: Benchmarking Object-Centric Spatial Understanding [8.202861909913791]
  We present a benchmark for object-centric spatial reasoning in foundation models.<n>We find a stable trade-off: detectors such as GroundingDINO and OWLv2 deliver precise boxes with limited relational reasoning.<n>Our study highlights the gap between localization and true spatial understanding, and pointing toward the need for spatially-aware foundation models in the community.
  arXiv  Detail & Related papers  (2025-09-26T06:06:19Z)
• FlySearch: Exploring how vision-language models explore [5.7210882663967615]
  We introduce FlySearch, a 3D, outdoor, environment for searching and navigating to objects in complex scenes.<n>We observe that state-of-the-art Vision-Language Models (VLMs) cannot reliably solve even the simplest exploration tasks.<n>We identify a set of central causes, ranging from vision, through context misunderstanding, to task planning failures, and we show that some of them can be addressed by finetuning.
  arXiv  Detail & Related papers  (2025-06-03T14:03:42Z)
• CAPTURe: Evaluating Spatial Reasoning in Vision Language Models via Occluded Object Counting [59.830657530592255]
  Counting Amodally for Patterns Through Unseen REgions (CAPTURe) is a testbed for evaluating vision-language models.<n>We evaluate four strong vision-language models on CAPTURe, finding that models struggle to count on both occluded and unoccluded patterns.
  arXiv  Detail & Related papers  (2025-04-21T23:38:43Z)
• Mind the Gap: Benchmarking Spatial Reasoning in Vision-Language Models [14.442394137843923]
  We present a detailed analysis that first delineates the core elements of spatial reasoning.<n>We then assesses the performance of these models in both synthetic and real-world images.
  arXiv  Detail & Related papers  (2025-03-25T14:34:06Z)
• Why Is Spatial Reasoning Hard for VLMs? An Attention Mechanism Perspective on Focus Areas [69.56484419619919]
  We study the spatial reasoning challenge from the lens of mechanistic interpretability.<n>We observe that successful spatial reasoning correlates strongly with the model's ability to align its attention with actual object locations.<n>Motivated by these findings, we propose ADAPTVIS to sharpen the attention on highly relevant regions when confident.
  arXiv  Detail & Related papers  (2025-03-03T17:57:03Z)
• SpatialRGPT: Grounded Spatial Reasoning in Vision Language Models [68.13636352687257]
  We introduce Spatial Region GPT (SpatialRGPT) to enhance VLMs' spatial perception and reasoning capabilities. During inference, when provided with user-specified region proposals, SpatialRGPT can accurately perceive their relative directions and distances. Our results demonstrate that SpatialRGPT significantly enhances performance in spatial reasoning tasks, both with and without local region prompts.
  arXiv  Detail & Related papers  (2024-06-03T17:59:06Z)
• Object Manipulation via Visual Target Localization [64.05939029132394]
  Training agents to manipulate objects, poses many challenges. We propose an approach that explores the environment in search for target objects, computes their 3D coordinates once they are located, and then continues to estimate their 3D locations even when the objects are not visible. Our evaluations show a massive 3x improvement in success rate over a model that has access to the same sensory suite.
  arXiv  Detail & Related papers  (2022-03-15T17:59:01Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.