RenderMem: Rendering as Spatial Memory Retrieval

• URL: http://arxiv.org/abs/2603.14669v1
• Date: Sun, 15 Mar 2026 23:53:40 GMT
• Title: RenderMem: Rendering as Spatial Memory Retrieval
• Abstract summary: Embodied reasoning is inherently viewpoint-dependent.<n>We introduce RenderMem, a spatial memory framework that treats rendering as the interface between 3D world representations and spatial reasoning.
• Score: 3.781421673607643
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Embodied reasoning is inherently viewpoint-dependent: what is visible, occluded, or reachable depends critically on where the agent stands. However, existing spatial memory systems for embodied agents typically store either multi-view observations or object-centric abstractions, making it difficult to perform reasoning with explicit geometric grounding. We introduce RenderMem, a spatial memory framework that treats rendering as the interface between 3D world representations and spatial reasoning. Instead of storing fixed observations, RenderMem maintains a 3D scene representation and generates query-conditioned visual evidence by rendering the scene from viewpoints implied by the query. This enables embodied agents to reason directly about line-of-sight, visibility, and occlusion from arbitrary perspectives. RenderMem is fully compatible with existing vision-language models and requires no modification to standard architectures. Experiments in the AI2-THOR environment show consistent improvements on viewpoint-dependent visibility and occlusion queries over prior memory baselines.

Related papers

• SceneBench: A Hierarchical Benchmark for Vision-Language Understanding of 3D Scenes [12.672496253081258]
  Vision-language models excel at 2D image understanding but remain limited in 3D spatial reasoning.<n>We introduce SceneBench, a benchmark of 966 photorealistic 3D scenes reconstructed with Gaussian Splatting.<n>SceneBench provides a realistic testbed for developing and evaluating models capable of fine-grained spatial reasoning.
  arXiv  Detail & Related papers  (2026-09-14T18:59:13Z)
• G$^2$TAM: Geometry Grounded Track Anything Model [49.05553245076823]
  We propose a unified framework for promptable instance tracking in 3D using only unordered RGB images or videos.<n>G$2$TAM employs spatially aligned geometric representations as implicit memory, ensuring stable instance identity and localization across frames and views.<n>At its core is a cross-modal spatial encoder that integrates visual and textual prompts into a shared geometric space, enabling end-to-end spatial reconstruction and instance-consistent mask prediction.
  arXiv  Detail & Related papers  (2026-07-04T09:32:13Z)
• OmniView-Space: Reinforcing Spatial Reasoning via Multi-Perspective Spatial Mapping [69.48762031013514]
  We propose OmniView-Space, a framework designed to maintain spatial consistency through multimodal egocentric evidence.<n>Our approach consists of three core components: (1) Multi-Perspective Spatial Mapping (MPSM), which re-anchors reconstructed geometry into a query-aligned visual cognitive map and a textual spatial graph; (2) Tool-Guided Egocentric Reasoning, an interleaved policy trained to actively select the ego anchor required by the query and request the corresponding MPSM evidence; and (3) Cognitive-Map Distillation, which uses MPSM-generated trajectories and ego-frame rewards to train the model to reason with self-generated cognitive maps
  arXiv  Detail & Related papers  (2026-07-01T12:45:12Z)
• GARDEN: Gravity-Aligned Reconstruction of Disentangled ENvironments from RGB images [61.2170105836525]
  We propose an RGB-only framework that reformulates reconstruction as physically-grounded scene factorization and outputs a structured hybrid scene representation.<n>The resulting representation combines explicit rigid bodies with a decoupled background, enabling direct physics simulation while preserving visual realism.
  arXiv  Detail & Related papers  (2026-06-02T17:13:01Z)
• Reasmory: 3D Reconstruction as Explicit Memory for VLMs Spatial Reasoning [43.950190960277865]
  Vision-Language Models (VLMs) exhibit emerging spatial reasoning capabilities, yet they remain unreliable on tasks requiring precise spatial understanding.<n>We propose textbfReasmory, a framework that formulates spatial reasoning as structured program execution over reconstructed spatial memory.<n>Reasmory constructs explicit 3D memory, augments it with semantically grounded 3D object instances, and introduces a lightweight Domain-Specific Language.
  arXiv  Detail & Related papers  (2026-05-31T02:36:57Z)
• E3VS-Bench: A Benchmark for Viewpoint-Dependent Active Perception in 3D Gaussian Splatting Scenes [42.5758395227096]
  E3VS-Bench is a benchmark for embodied 3D visual search in real-world 3D environments.<n>It consists of 99 high-fidelity 3D scenes reconstructed using 3D Gaussian Splatting and 2,014 question-driven episodes.<n>We evaluate multiple state-of-the-art VLMs and compare their performance with humans.
  arXiv  Detail & Related papers  (2026-04-20T08:49:39Z)
• RieMind: Geometry-Grounded Spatial Agent for Scene Understanding [47.34079422330063]
  Current approaches rely on end-to-end video understanding or large-scale spatial question answering fine-tuning.<n>We propose an agentic framework for static 3D indoor scene reasoning that grounds an LLM in an explicit 3D scene graph (3DSG)<n>We find that our agentic variant achieves significantly better performance, with average improvements between 33% to 50%.
  arXiv  Detail & Related papers  (2026-03-16T15:02:24Z)
• Context-Nav: Context-Driven Exploration and Viewpoint-Aware 3D Spatial Reasoning for Instance Navigation [4.039245878626346]
  Text-goal instance navigation (TGIN) asks an agent to resolve a single, free-form description into actions that reach the correct object instance.<n>We present textitContext-Nav that elevates contextual captions from a local cue to a global exploration prior and verifies candidates through 3D spatial reasoning.
  arXiv  Detail & Related papers  (2026-03-10T11:08:35Z)
• RnG: A Unified Transformer for Complete 3D Modeling from Partial Observations [70.83499963694238]
  RnG (Reconstruction and Generation) is a novel feed-forward Transformer that unifies reconstruction and generation.<n>It reconstructs visible geometry and generates plausible, coherent unseen geometry and appearance.<n>Our method achieves state-of-the-art performance in both generalizable 3D reconstruction and novel view generation.
  arXiv  Detail & Related papers  (2026-03-01T17:25:32Z)
• SeeThrough3D: Occlusion Aware 3D Control in Text-to-Image Generation [32.15143378003745]
  occlusion reasoning is essential for partially occluded objects with depth-consistent geometry and scale.<n>We propose SeeThrough3D, a model for 3D layout conditioned generation that explicitly models occlusions.
  arXiv  Detail & Related papers  (2026-02-26T18:59:05Z)
• Thinking with Blueprints: Assisting Vision-Language Models in Spatial Reasoning via Structured Object Representation [52.605647992080485]
  spatial reasoning advances vision-language models from visual perception toward semantic understanding.<n>We integrate the cognitive concept of an object-centric blueprint into spatial reasoning.<n>Our method consistently outperforms existing vision-language models.
  arXiv  Detail & Related papers  (2026-01-05T10:38:26Z)
• OST-Bench: Evaluating the Capabilities of MLLMs in Online Spatio-temporal Scene Understanding [50.72259772580637]
  We introduce OST-Bench, a benchmark designed to evaluate Online Spatio-Temporal understanding from the perspective of an agent.<n>Built on an efficient data collection pipeline, OST-Bench consists of 1.4k scenes and 10k question-answer pairs collected from ScanNet, Matterport3D, and ARKitScenes.<n>We find that both complex-based spatial reasoning demands and long-term memory retrieval requirements significantly drop model performance along two separate axes.
  arXiv  Detail & Related papers  (2025-07-10T17:56:07Z)
• Move to Understand a 3D Scene: Bridging Visual Grounding and Exploration for Efficient and Versatile Embodied Navigation [54.04601077224252]
  Embodied scene understanding requires not only comprehending visual-spatial information but also determining where to explore next in the 3D physical world.<n>underlinetextbf3D vision-language learning enables embodied agents to effectively explore and understand their environment.<n>model's versatility enables navigation using diverse input modalities, including categories, language descriptions, and reference images.
  arXiv  Detail & Related papers  (2025-07-05T14:15:52Z)
• SCORP: Scene-Consistent Object Refinement via Proxy Generation and Tuning [46.441761732998536]
  We introduce Scene-Consistent Object Refinement via Proxy Generation and Tuning (SCORP)<n>SCORP is a novel 3D enhancement framework that leverages 3D generative priors to recover fine-grained object geometry and appearance under missing views.<n>It achieves consistent gains over recent state-of-the-art baselines on both novel view synthesis and geometry completion tasks.
  arXiv  Detail & Related papers  (2025-06-30T13:26:21Z)
• Beyond Viewpoint: Robust 3D Object Recognition under Arbitrary Views through Joint Multi-Part Representation [22.8031613567025]
  Part-aware Network (PANet) aims to localize and understand different parts of 3D objects, such as airplane wings and tails. Our results on benchmark datasets clearly demonstrate that our proposed method outperforms existing view-based aggregation baselines for the task of 3D object recognition under arbitrary views.
  arXiv  Detail & Related papers  (2024-07-04T11:16:47Z)
• Know Your Neighbors: Improving Single-View Reconstruction via Spatial Vision-Language Reasoning [119.99066522299309]
  KYN is a novel method for single-view scene reconstruction that reasons about semantic and spatial context to predict each point's density. We show that KYN improves 3D shape recovery compared to predicting density for each 3D point in isolation. We achieve state-of-the-art results in scene and object reconstruction on KITTI-360, and show improved zero-shot generalization compared to prior work.
  arXiv  Detail & Related papers  (2024-04-04T17:59:59Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.