Grounding by Remembering: Cross-Scene and In-Scene Memory for 3D Functional Affordances

• URL: http://arxiv.org/abs/2605.11616v1
• Date: Tue, 12 May 2026 06:44:47 GMT
• Title: Grounding by Remembering: Cross-Scene and In-Scene Memory for 3D Functional Affordances
• Abstract summary: AFFORDMEM is a framework that grounds 3D functional affordances by remembering geometry at two levels.<n>As the agent processes the scene, it organizes candidate instances and their 3D spatial relations into a structured scene graph.<n>Our method improves AP50 over the prior training-free state of the art by 3.23 on Split 0 and 3.7 on Split 1.
• Score: 23.103482730965595
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Functional affordance grounding requires more than recognizing an object: an agent must localize the specific region that supports an interaction, such as the handle to pull or the button to press. This is difficult for training-free vision-language pipelines because actionable regions are often small, visually ambiguous, and repeated across multiple same-category instances in a scene. We propose AFFORDMEM, a framework that grounds 3D functional affordances by remembering geometry at two levels. The first is cross-scene affordance memory: the agent maintains a category-level memory bank of RGB images with affordance regions rendered as overlays, and recalls the most informative examples at query time to guide a frozen VLM toward small operable subregions that text-only prompting consistently misses. The second is in-scene spatial memory: as the agent processes the scene, it organizes candidate instances and their 3D spatial relations into a structured scene graph, enabling the language model to resolve references over distant or currently unobserved candidates such as "the second handle from the top." AFFORDMEM requires no model fine-tuning and no target-scene annotation, using a reusable memory bank built from source scenes. On SceneFun3D, our method improves AP50 over the prior training-free state of the art by 3.23 on Split 0 and 3.7 on Split 1. Ablation studies support complementary benefits: cross-scene affordance memory improves fine-grained localization, while in-scene spatial memory provides the larger gain on spatially qualified queries. The project homepage is available at the project page.

Related papers

• CreaMem: A Scene-Aware Memory Architecture for Personalized Agents [10.59532182467324]
  We propose the CreaMem architecture, which enables scene-aware memory organization by partitioning memory into several Life Scene Memories.<n>CreaMem improves QA accuracy across all evaluation metrics, with particularly large gains on multi-hop reasoning performance.<n>To enhance, we release our code in a public GitHub repository.
  arXiv  Detail & Related papers  (2026-09-08T10:33:05Z)
• Reasmory: 3D Reconstruction as Explicit Memory for VLMs Spatial Reasoning [43.950190960277865]
  Vision-Language Models (VLMs) exhibit emerging spatial reasoning capabilities, yet they remain unreliable on tasks requiring precise spatial understanding.<n>We propose textbfReasmory, a framework that formulates spatial reasoning as structured program execution over reconstructed spatial memory.<n>Reasmory constructs explicit 3D memory, augments it with semantically grounded 3D object instances, and introduces a lightweight Domain-Specific Language.
  arXiv  Detail & Related papers  (2026-05-31T02:36:57Z)
• Memory Over Maps: 3D Object Localization Without Reconstruction [44.29345996003297]
  Target localization is a prerequisite for embodied tasks such as navigation and manipulation.<n>Recent advances in vision-language models suggest that rich semantic reasoning can be performed directly on 2D observations.<n>We propose a map-free pipeline that stores only posed RGB-Ds as a lightweight visual memory without constructing any global 3D representation of the scene.
  arXiv  Detail & Related papers  (2026-03-20T21:57:51Z)
• SpatialMem: Unified 3D Memory with Metric Anchoring and Fast Retrieval [19.68937683078205]
  SpatialMem is a memory-centric system that unifies 3D geometry, semantics, and language into a single representation.<n>It reconstructs metrically scaled indoor environments, detects structural 3D anchors, and populates a hierarchical memory with open-vocabulary object nodes.<n>It supports downstream tasks such as language-guided navigation and object retrieval without specialized sensors.
  arXiv  Detail & Related papers  (2026-01-21T11:32:24Z)
• ChangingGrounding: 3D Visual Grounding in Changing Scenes [92.00984845186679]
  Real-world robots localize objects from natural-language instructions while scenes around them keep changing.<n>Most of the existing 3D visual grounding (3DVG) method still assumes a reconstructed and up-to-date point cloud.<n>We introduce ChangingGrounding, the first benchmark that explicitly measures how well an agent can exploit past observations.
  arXiv  Detail & Related papers  (2025-10-16T17:59:16Z)
• A Neural Representation Framework with LLM-Driven Spatial Reasoning for Open-Vocabulary 3D Visual Grounding [78.99798110890157]
  Open-vocabulary 3D visual grounding aims to localize target objects based on free-form language queries.<n>Existing language field methods struggle to accurately localize instances using spatial relations in language queries.<n>We propose SpatialReasoner, a novel neural representation-based framework with large language model (LLM)-driven spatial reasoning.
  arXiv  Detail & Related papers  (2025-07-09T10:20:38Z)
• Point3R: Streaming 3D Reconstruction with Explicit Spatial Pointer Memory [72.75478398447396]
  We propose Point3R, an online framework targeting dense streaming 3D reconstruction.<n>To be specific, we maintain an explicit spatial pointer memory directly associated with the 3D structure of the current scene.<n>Our method achieves competitive or state-of-the-art performance on various tasks with low training costs.
  arXiv  Detail & Related papers  (2025-07-03T17:59:56Z)
• TSDASeg: A Two-Stage Model with Direct Alignment for Interactive Point Cloud Segmentation [3.615396917221689]
  We propose TSDASeg, a Two-Stage model coupled with a Direct cross-modal alignment module and memory module for interactive point cloud.<n>We introduce the direct cross-modal alignment module to establish explicit alignment between 3D point clouds and textual/2D image data.<n>Within the memory module, we employ multiple dedicated memory banks to separately store text features, visual features, and their cross-modal correspondence mappings.
  arXiv  Detail & Related papers  (2025-06-26T04:10:33Z)
• A-SCoRe: Attention-based Scene Coordinate Regression for wide-ranging scenarios [1.2093553114715083]
  A-ScoRe is an Attention-based model which leverage attention on descriptor map level to produce meaningful and high-semantic 2D descriptors.<n>Results show our methods achieve comparable performance with State-of-the-art methods on multiple benchmark while being light-weighted and much more flexible.
  arXiv  Detail & Related papers  (2025-03-18T07:39:50Z)
• 3D-Mem: 3D Scene Memory for Embodied Exploration and Reasoning [65.40458559619303]
  We propose 3D-Mem, a novel 3D scene memory framework for embodied agents.<n>3D-Mem employs informative multi-view images, termed Memory Snapshots, to represent the scene.<n>It further integrates frontier-based exploration by introducing Frontier Snapshots-glimpses of unexplored areas-enabling agents to make informed decisions.
  arXiv  Detail & Related papers  (2024-11-23T09:57:43Z)
• Self-supervised Learning of Neural Implicit Feature Fields for Camera Pose Refinement [32.335953514942474]
  This paper proposes to jointly learn the scene representation along with a 3D dense feature field and a 2D feature extractor. We learn the underlying geometry of the scene with an implicit field through volumetric rendering and design our feature field to leverage intermediate geometric information encoded in the implicit field. Visual localization is then achieved by aligning the image-based features and the rendered volumetric features.
  arXiv  Detail & Related papers  (2024-06-12T17:51:53Z)
• Visual Localization via Few-Shot Scene Region Classification [84.34083435501094]
  Visual (re)localization addresses the problem of estimating the 6-DoF camera pose of a query image captured in a known scene. Recent advances in structure-based localization solve this problem by memorizing the mapping from image pixels to scene coordinates. We propose a scene region classification approach to achieve fast and effective scene memorization with few-shot images.
  arXiv  Detail & Related papers  (2022-08-14T22:39:02Z)
• VS-Net: Voting with Segmentation for Visual Localization [72.8165619061249]
  We propose a novel visual localization framework that establishes 2D-to-3D correspondences between the query image and the 3D map with a series of learnable scene-specific landmarks. Our proposed VS-Net is extensively tested on multiple public benchmarks and can outperform state-of-the-art visual localization methods.
  arXiv  Detail & Related papers  (2021-05-23T08:44:11Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.