Expanding Spatial and Temporal Context for Robotic Imitation Learning With Scene Graphs

• URL: http://arxiv.org/abs/2606.01072v2
• Date: Fri, 05 Jun 2026 06:32:49 GMT
• Title: Expanding Spatial and Temporal Context for Robotic Imitation Learning With Scene Graphs
• Abstract summary: We propose using scene graphs as an explicit and structured memory mechanism in imitation learning.<n>By maintaining a dynamic scene graph that captures object-centric relationships and their evolution over time, our method allows the agent to retain relevant historical context.<n>Our experiments on simulated mobile manipulation and real-world tabletop manipulation demonstrate that our approach substantially improves policy performance.
• Score: 20.685320925756375
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Imitation learning enables robots to learn how to execute tasks via observation. However, real-world environments like homes and offices are often severely partially observed due to their large spatial scales. In addition, many tasks involve executing a series of subtasks requiring autonomous robots to reason over extended time horizons. To address these challenges, we propose using scene graphs as an explicit and structured memory mechanism in imitation learning. By maintaining a dynamic scene graph that captures object-centric relationships and their evolution over time, our method allows the agent to retain relevant historical context during task execution to efficiently reason over incrementally accrued scene information. Our experiments on simulated mobile manipulation and real-world tabletop manipulation demonstrate that our approach substantially improves policy performance, particularly in settings that demand long-term reasoning and robust generalization under partial observability.

Related papers

• Beyond Episodic Evaluation: Memory Architectural Bottlenecks in Sequential Embodied Question Answering [93.80043825360514]
  Embodied question answering (EQA) is traditionally evaluated under an episodic formulation, where agents solve each task independently and reset internal state between episodes.<n>We investigate how different memory architectures behave when EQA agents are evaluated sequentially, with multiple questions answered in the same scene while memory is carried forward across queries.<n>We find that simply preserving existing memory is often insufficient.<n>We highlight the necessity of structured, spatially grounded memory: architectures that map persistent visual observations onto metric 3D geometry preserve visual-semantic evidence in a coherent scene representation.
  arXiv  Detail & Related papers  (2026-07-23T17:50:45Z)
• Learning Semantic-Geometric Task Graph-Representations from Human Demonstrations [16.68801520494275]
  We introduce a semantic-geometric task graph-representation that encodes object identities, inter-object relations, and their temporal geometric evolution from human demonstrations.<n>We show that semantic-geometric task graph-representations are particularly beneficial for tasks with high action and object variability.
  arXiv  Detail & Related papers  (2026-01-16T17:35:00Z)
• Generalizable Geometric Prior and Recurrent Spiking Feature Learning for Humanoid Robot Manipulation [90.90219129619344]
  This paper presents a novel R-prior-S, Recurrent Geometric-priormodal Policy with Spiking features.<n>To ground high-level reasoning in physical reality, we leverage lightweight 2D geometric inductive biases.<n>For the data efficiency issue in robotic action generation, we introduce a Recursive Adaptive Spiking Network.
  arXiv  Detail & Related papers  (2026-01-13T23:36:30Z)
• FindingDory: A Benchmark to Evaluate Memory in Embodied Agents [49.18498389833308]
  We introduce a new benchmark for long-range embodied tasks in the Habitat simulator.<n>This benchmark evaluates memory-based capabilities across 60 tasks requiring sustained engagement and contextual awareness.
  arXiv  Detail & Related papers  (2025-06-18T17:06:28Z)
• Hi-Dyna Graph: Hierarchical Dynamic Scene Graph for Robotic Autonomy in Human-Centric Environments [41.80879866951797]
  Hi-Dyna Graph is a hierarchical dynamic scene graph architecture that integrates persistent global layouts with localized dynamic semantics for embodied robotic autonomy.<n>An agent powered by large language models (LLMs) is employed to interpret the unified graph, infer latent task triggers, and generate executable instructions grounded in robotic affordances.
  arXiv  Detail & Related papers  (2025-05-30T03:35:29Z)
• Tool-as-Interface: Learning Robot Policies from Observing Human Tool Use [20.644849892016122]
  We propose a framework that transfers tool-use knowledge from humans to robots.<n>We demonstrate our framework's effectiveness across a diverse suite of tool-use tasks.<n>Our method achieves a 71% improvement in task success over teleoperation-based diffusion policies.
  arXiv  Detail & Related papers  (2025-04-06T20:40:19Z)
• EmbodiedVSR: Dynamic Scene Graph-Guided Chain-of-Thought Reasoning for Visual Spatial Tasks [24.41705039390567]
  EmbodiedVSR (Embodied Visual Spatial Reasoning) is a novel framework that integrates dynamic scene graph-guided Chain-of-Thought (CoT) reasoning.<n>Our method enables zero-shot spatial reasoning without task-specific fine-tuning.<n>Experiments demonstrate that our framework significantly outperforms existing MLLM-based methods in accuracy and reasoning coherence.
  arXiv  Detail & Related papers  (2025-03-14T05:06:07Z)
• Reasoning in visual navigation of end-to-end trained agents: a dynamical systems approach [23.52028824411467]
  We present a large-scale experimental study involving numepisodes navigation episodes in a real environment with a physical robot.<n>We analyze the type of reasoning emerging from end-to-end training.<n>We show in a post-hoc analysis that the value function learned by the agent relates to long-term planning.
  arXiv  Detail & Related papers  (2025-03-11T11:16:47Z)
• Interactive Planning Using Large Language Models for Partially Observable Robotics Tasks [54.60571399091711]
  Large Language Models (LLMs) have achieved impressive results in creating robotic agents for performing open vocabulary tasks. We present an interactive planning technique for partially observable tasks using LLMs.
  arXiv  Detail & Related papers  (2023-12-11T22:54:44Z)
• Instruction-driven history-aware policies for robotic manipulations [82.25511767738224]
  We propose a unified transformer-based approach that takes into account multiple inputs. In particular, our transformer architecture integrates (i) natural language instructions and (ii) multi-view scene observations. We evaluate our method on the challenging RLBench benchmark and on a real-world robot.
  arXiv  Detail & Related papers  (2022-09-11T16:28:25Z)
• Modeling Long-horizon Tasks as Sequential Interaction Landscapes [75.5824586200507]
  We present a deep learning network that learns dependencies and transitions across subtasks solely from a set of demonstration videos. We show that these symbols can be learned and predicted directly from image observations. We evaluate our framework on two long horizon tasks: (1) block stacking of puzzle pieces being executed by humans, and (2) a robot manipulation task involving pick and place of objects and sliding a cabinet door with a 7-DoF robot arm.
  arXiv  Detail & Related papers  (2020-06-08T18:07:18Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.