Beyond Placement and Articulation: Usage-Driven Code Scenes for Embodied Interaction

• URL: http://arxiv.org/abs/2608.18840v1
• Date: Wed, 19 Aug 2026 12:00:35 GMT
• Title: Beyond Placement and Articulation: Usage-Driven Code Scenes for Embodied Interaction
• Abstract summary: We present RoomWright, an agentic usage-driven framework for generating 3D scenes represented entirely as code for embodied interaction.<n>RoomWright performs usage-driven object reasoning, which treats each anchor as a task centre and admits task-required objects and their affordances.<n>The resulting scenes are executable, editable, and simulation-ready, providing interactive environments for embodied AI and policy learning.
• Score: 29.693557719848187
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Indoor scene synthesis provides essential environments for embodied AI, robotic manipulation, and simulation-based policy learning. Recent code-based scene generation methods produce editable and extensible environments, yet they remain focused on visual construction and object-level articulation, leaving the functional usage of scenes largely unmodeled. To address this problem, we present RoomWright, an agentic usage-driven framework for generating 3D scenes represented entirely as code for embodied interaction. RoomWright performs usage-driven object reasoning, which treats each anchor as a task centre and admits task-required objects and their affordances. A code agent further enables multi-part interaction by compiling each interaction into a trigger, condition, effect rule that updates structured object states, capturing causal dependencies across objects. Moreover, since manipuland orientation is ambiguous and hard to recover from pixels, RoomWright alleviates this via annotation-informed usage-guided orientation. Extensive experiments demonstrate the effectiveness of our method. The resulting scenes are executable, editable, and simulation-ready, providing interactive environments for embodied AI and policy learning.

Related papers

• SAGE: Scalable Agentic 3D Scene Generation for Embodied AI [67.43935343696982]
  Existing scene-generation systems often rely on rule-based or task-specific pipelines, yielding artifacts and physically invalid scenes.<n>We present SAGE, an agentic framework that, given a user-specified embodied task, understands the intent and automatically generates simulation-ready environments at scale.<n>The resulting environments are realistic, diverse, and directly deployable in modern simulators for policy training.
  arXiv  Detail & Related papers  (2026-02-10T18:59:55Z)
• SceneWeaver: All-in-One 3D Scene Synthesis with an Extensible and Self-Reflective Agent [28.12183839499528]
  SceneWeaver is a framework that unifies diverse scene synthesis paradigms through tool-based iterative refinement.<n>It can identify semantic inconsistencies, invoke targeted tools, and update the environment over successive iterations.<n>It generalizes effectively to complex scenes with diverse instructions, marking a step toward general-purpose 3D environment generation.
  arXiv  Detail & Related papers  (2025-09-24T09:06:41Z)
• HOSIG: Full-Body Human-Object-Scene Interaction Generation with Hierarchical Scene Perception [57.37135310143126]
  HO SIG is a novel framework for synthesizing full-body interactions through hierarchical scene perception.<n>Our framework supports unlimited motion length through autoregressive generation and requires minimal manual intervention.<n>This work bridges the critical gap between scene-aware navigation and dexterous object manipulation.
  arXiv  Detail & Related papers  (2025-06-02T12:08:08Z)
• HiScene: Creating Hierarchical 3D Scenes with Isometric View Generation [50.206100327643284]
  HiScene is a novel hierarchical framework that bridges the gap between 2D image generation and 3D object generation.<n>We generate 3D content that aligns with 2D representations while maintaining compositional structure.
  arXiv  Detail & Related papers  (2025-04-17T16:33:39Z)
• IAAO: Interactive Affordance Learning for Articulated Objects in 3D Environments [56.85804719947]
  We present IAAO, a framework that builds an explicit 3D model for intelligent agents to gain understanding of articulated objects in their environment through interaction.<n>We first build hierarchical features and label fields for each object state using 3D Gaussian Splatting (3DGS) by distilling mask features and view-consistent labels from multi-view images.<n>We then perform object- and part-level queries on the 3D Gaussian primitives to identify static and articulated elements, estimating global transformations and local articulation parameters along with affordances.
  arXiv  Detail & Related papers  (2025-04-09T12:36:48Z)
• Flex: End-to-End Text-Instructed Visual Navigation from Foundation Model Features [59.892436892964376]
  We investigate the minimal data requirements and architectural adaptations necessary to achieve robust closed-loop performance with vision-based control policies.<n>Our findings are synthesized in Flex (Fly lexically), a framework that uses pre-trained Vision Language Models (VLMs) as frozen patch-wise feature extractors.<n>We demonstrate the effectiveness of this approach on a quadrotor fly-to-target task, where agents trained via behavior cloning successfully generalize to real-world scenes.
  arXiv  Detail & Related papers  (2024-10-16T19:59:31Z)
• Controllable Human-Object Interaction Synthesis [77.56877961681462]
  We propose Controllable Human-Object Interaction Synthesis (CHOIS) to generate synchronized object motion and human motion in 3D scenes. Here, language descriptions inform style and intent, and waypoints, which can be effectively extracted from high-level planning, ground the motion in the scene. Our module seamlessly integrates with a path planning module, enabling the generation of long-term interactions in 3D environments.
  arXiv  Detail & Related papers  (2023-12-06T21:14:20Z)
• Learning Extrinsic Dexterity with Parameterized Manipulation Primitives [8.7221770019454]
  We learn a sequence of actions that utilize the environment to change the object's pose. Our approach can control the object's state through exploiting interactions between the object, the gripper, and the environment. We evaluate our approach on picking box-shaped objects of various weight, shape, and friction properties from a constrained table-top workspace.
  arXiv  Detail & Related papers  (2023-10-26T21:28:23Z)
• Learning Environment-Aware Affordance for 3D Articulated Object Manipulation under Occlusions [23.913350548394405]
  We propose an environment-aware affordance framework that incorporates both object-level actionable priors and environment constraints.<n>We introduce a novel contrastive affordance learning framework capable of training on scenes containing a single occluder and generalizing to scenes with complex occluder combinations.
  arXiv  Detail & Related papers  (2023-09-14T08:24:32Z)
• Object-agnostic Affordance Categorization via Unsupervised Learning of Graph Embeddings [6.371828910727037]
  Acquiring knowledge about object interactions and affordances can facilitate scene understanding and human-robot collaboration tasks. We address the problem of affordance categorization for class-agnostic objects with an open set of interactions. A novel depth-informed qualitative spatial representation is proposed for the construction of Activity Graphs.
  arXiv  Detail & Related papers  (2023-03-30T15:04:04Z)
• SceneGen: Generative Contextual Scene Augmentation using Scene Graph Priors [3.1969855247377827]
  We introduce SceneGen, a generative contextual augmentation framework that predicts virtual object positions and orientations within existing scenes. SceneGen takes a semantically segmented scene as input, and outputs positional and orientational probability maps for placing virtual content. We formulate a novel spatial Scene Graph representation, which encapsulates explicit topological properties between objects, object groups, and the room. To demonstrate our system in action, we develop an Augmented Reality application, in which objects can be contextually augmented in real-time.
  arXiv  Detail & Related papers  (2020-09-25T18:36:27Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.