ReSpace: Text-Driven 3D Scene Synthesis and Editing with Preference Alignment

• URL: http://arxiv.org/abs/2506.02459v2
• Date: Tue, 10 Jun 2025 20:08:12 GMT
• Title: ReSpace: Text-Driven 3D Scene Synthesis and Editing with Preference Alignment
• Authors: Martin JJ. Bucher, Iro Armeni,
• Abstract summary: ReSpace is a generative framework for text-driven 3D indoor scene synthesis and editing.<n>We leverage a dual-stage training approach combining supervised fine-tuning and preference alignment.<n>For scene editing, we employ a zero-shot LLM to handle object removal and prompts for addition.
• Score: 1.0918065824771606
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Scene synthesis and editing has emerged as a promising direction in computer graphics. Current trained approaches for 3D indoor scenes either oversimplify object semantics through one-hot class encodings (e.g., 'chair' or 'table'), require masked diffusion for editing, ignore room boundaries, or rely on floor plan renderings that fail to capture complex layouts. In contrast, LLM-based methods enable richer semantics via natural language (e.g., 'modern studio with light wood furniture') but do not support editing, remain limited to rectangular layouts or rely on weak spatial reasoning from implicit world models. We introduce ReSpace, a generative framework for text-driven 3D indoor scene synthesis and editing using autoregressive language models. Our approach features a compact structured scene representation with explicit room boundaries that frames scene editing as a next-token prediction task. We leverage a dual-stage training approach combining supervised fine-tuning and preference alignment, enabling a specially trained language model for object addition that accounts for user instructions, spatial geometry, object semantics, and scene-level composition. For scene editing, we employ a zero-shot LLM to handle object removal and prompts for addition. We further introduce a novel voxelization-based evaluation that captures fine-grained geometry beyond 3D bounding boxes. Experimental results surpass state-of-the-art on object addition while maintaining competitive results on full scene synthesis.

Related papers

• Scenethesis: A Language and Vision Agentic Framework for 3D Scene Generation [36.44409268300039]
  Scenethesis is a framework that integrates text-based scene planning with vision-guided layout refinement.<n>It generates diverse, realistic, and physically plausible 3D interactive scenes, making it valuable for virtual content creation, simulation environments, and embodied AI research.
  arXiv  Detail & Related papers  (2025-05-05T17:59:58Z)
• FreeInsert: Disentangled Text-Guided Object Insertion in 3D Gaussian Scene without Spatial Priors [67.26107732326948]
  FreeInsert is a novel framework that disentangles object generation from spatial placement.<n>It achieves semantically coherent, spatially precise, and visually realistic 3D insertions without relying on spatial priors.
  arXiv  Detail & Related papers  (2025-05-02T14:53:56Z)
• HiScene: Creating Hierarchical 3D Scenes with Isometric View Generation [50.206100327643284]
  HiScene is a novel hierarchical framework that bridges the gap between 2D image generation and 3D object generation.<n>We generate 3D content that aligns with 2D representations while maintaining compositional structure.
  arXiv  Detail & Related papers  (2025-04-17T16:33:39Z)
• EditRoom: LLM-parameterized Graph Diffusion for Composable 3D Room Layout Editing [114.14164860467227]
  We propose EditRoom, a framework capable of executing a variety of layout edits through natural language commands.<n>Specifically, EditRoom leverages Large Language Models (LLMs) for command planning and generates target scenes.<n>We have developed an automatic pipeline to augment existing 3D scene datasets and introduced EditRoom-DB, a large-scale dataset with 83k editing pairs.
  arXiv  Detail & Related papers  (2024-10-03T17:42:24Z)
• LLplace: The 3D Indoor Scene Layout Generation and Editing via Large Language Model [58.24851949945434]
  LLplace is a novel 3D indoor scene layout designer based on lightweight fine-tuned open-source LLM Llama3. LLplace circumvents the need for spatial relationship priors and in-context exemplars, enabling efficient and credible room layout generation. Our approach demonstrates that LLplace can effectively generate and edit 3D indoor layouts interactively and outperform existing methods in delivering high-quality 3D design solutions.
  arXiv  Detail & Related papers  (2024-06-06T08:53:01Z)
• SceneScript: Reconstructing Scenes With An Autoregressive Structured Language Model [7.707324214953882]
  We introduce SceneScript, a method that produces full scene models as a sequence of structured language commands. Our method infers the set of structured language commands directly from encoded visual data. Our method gives state-of-the art results in architectural layout estimation, and competitive results in 3D object detection.
  arXiv  Detail & Related papers  (2024-03-19T18:01:29Z)
• Set-the-Scene: Global-Local Training for Generating Controllable NeRF Scenes [68.14127205949073]
  We propose a novel GlobalLocal training framework for synthesizing a 3D scene using object proxies. We show that using proxies allows a wide variety of editing options, such as adjusting the placement of each independent object. Our results show that Set-the-Scene offers a powerful solution for scene synthesis and manipulation.
  arXiv  Detail & Related papers  (2023-03-23T17:17:29Z)
• DisCoScene: Spatially Disentangled Generative Radiance Fields for Controllable 3D-aware Scene Synthesis [90.32352050266104]
  DisCoScene is a 3Daware generative model for high-quality and controllable scene synthesis. It disentangles the whole scene into object-centric generative fields by learning on only 2D images with the global-local discrimination. We demonstrate state-of-the-art performance on many scene datasets, including the challenging outdoor dataset.
  arXiv  Detail & Related papers  (2022-12-22T18:59:59Z)
• Learning Object-Compositional Neural Radiance Field for Editable Scene Rendering [42.37007176376849]
  We present a novel neural scene rendering system, which learns an object-compositional neural radiance field and produces realistic rendering for a clustered and real-world scene. To survive the training in heavily cluttered scenes, we propose a scene-guided training strategy to solve the 3D space ambiguity in the occluded regions and learn sharp boundaries for each object.
  arXiv  Detail & Related papers  (2021-09-04T11:37:18Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.