SceneAssistant: A Visual Feedback Agent for Open-Vocabulary 3D Scene Generation

• URL: http://arxiv.org/abs/2603.12238v1
• Date: Thu, 12 Mar 2026 17:55:07 GMT
• Title: SceneAssistant: A Visual Feedback Agent for Open-Vocabulary 3D Scene Generation
• Abstract summary: We introduce SceneAssistant, a visual-feedback-driven agent designed for open-vocabulary 3D scene generation.<n>Our framework leverages modern 3D object generation model along with the spatial reasoning and planning capabilities of Vision-Language Models.<n>Our method allows users to instruct the agent to edit existing scenes based on natural language commands.
• Score: 27.16255874731512
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Text-to-3D scene generation from natural language is highly desirable for digital content creation. However, existing methods are largely domain-restricted or reliant on predefined spatial relationships, limiting their capacity for unconstrained, open-vocabulary 3D scene synthesis. In this paper, we introduce SceneAssistant, a visual-feedback-driven agent designed for open-vocabulary 3D scene generation. Our framework leverages modern 3D object generation model along with the spatial reasoning and planning capabilities of Vision-Language Models (VLMs). To enable open-vocabulary scene composition, we provide the VLMs with a comprehensive set of atomic operations (e.g., Scale, Rotate, FocusOn). At each interaction step, the VLM receives rendered visual feedback and takes actions accordingly, iteratively refining the scene to achieve more coherent spatial arrangements and better alignment with the input text. Experimental results demonstrate that our method can generate diverse, open-vocabulary, and high-quality 3D scenes. Both qualitative analysis and quantitative human evaluations demonstrate the superiority of our approach over existing methods. Furthermore, our method allows users to instruct the agent to edit existing scenes based on natural language commands. Our code is available at https://github.com/ROUJINN/SceneAssistant

Related papers

• Chat-Scene++: Exploiting Context-Rich Object Identification for 3D LLM [71.01178857331969]
  Chat-Scene++ is an MLLM framework that represents 3D scenes as context-rich object sequences.<n>By structuring scenes as sequences of objects with contextual semantics, Chat-Scene++ enables object-centric representation and interaction.<n>Chat-Scene++ achieves state-of-the-art performance on five major 3D vision-language benchmarks.
  arXiv  Detail & Related papers  (2026-03-29T04:16:04Z)
• OpenVoxel: Training-Free Grouping and Captioning Voxels for Open-Vocabulary 3D Scene Understanding [51.20680201565003]
  We propose OpenVoxel, a training-free algorithm for grouping and captioningvoxels for the open-vocabulary 3D scene understanding tasks.<n>Our method demonstrates superior performance compared to recent studies, particularly in complex referring expression segmentation (RES) tasks.
  arXiv  Detail & Related papers  (2026-01-14T15:45:57Z)
• Video Perception Models for 3D Scene Synthesis [109.5543506037003]
  VIPScene is a novel framework that exploits the encoded commonsense knowledge of the 3D physical world in video generation models.<n>VIPScene seamlessly integrates video generation, feedforward 3D reconstruction, and open-vocabulary perception models to semantically and geometrically analyze each object in a scene.
  arXiv  Detail & Related papers  (2025-06-25T16:40:17Z)
• ReSpace: Text-Driven 3D Indoor Scene Synthesis and Editing with Preference Alignment [8.954070942391603]
  ReSpace is a generative framework for text-driven 3D indoor scene synthesis and editing.<n>We leverage a dual-stage training approach combining supervised fine-tuning and preference alignment.<n>For scene editing, we employ a zero-shot LLM to handle object removal and prompts for addition.
  arXiv  Detail & Related papers  (2025-06-03T05:22:04Z)
• Agentic 3D Scene Generation with Spatially Contextualized VLMs [67.31920821192323]
  We introduce a new paradigm that enables vision-language models to generate, understand, and edit complex 3D environments.<n>We develop an agentic 3D scene generation pipeline in which the VLM iteratively reads from and updates the spatial context.<n>Results show that our framework can handle diverse and challenging inputs, achieving a level of generalization not observed in prior work.
  arXiv  Detail & Related papers  (2025-05-26T15:28:17Z)
• Scenethesis: A Language and Vision Agentic Framework for 3D Scene Generation [36.44409268300039]
  Scenethesis is a framework that integrates text-based scene planning with vision-guided layout refinement.<n>It generates diverse, realistic, and physically plausible 3D interactive scenes, making it valuable for virtual content creation, simulation environments, and embodied AI research.
  arXiv  Detail & Related papers  (2025-05-05T17:59:58Z)
• Decorum: A Language-Based Approach For Style-Conditioned Synthesis of Indoor 3D Scenes [18.232310061758298]
  3D indoor scene generation is an important problem for the design of digital and real-world environments.<n>Existing methods for this task exhibit very limited control over these attributes.<n>Our proposed method Decorum enables users to control the scene generation process with natural language.
  arXiv  Detail & Related papers  (2025-03-23T17:48:44Z)
• Unlocking Textual and Visual Wisdom: Open-Vocabulary 3D Object Detection Enhanced by Comprehensive Guidance from Text and Image [70.02187124865627]
  Open-vocabulary 3D object detection (OV-3DDet) aims to localize and recognize both seen and previously unseen object categories within any new 3D scene. We leverage a vision foundation model to provide image-wise guidance for discovering novel classes in 3D scenes. We demonstrate significant improvements in accuracy and generalization, highlighting the potential of foundation models in advancing open-vocabulary 3D object detection.
  arXiv  Detail & Related papers  (2024-07-07T04:50:04Z)
• SceneWiz3D: Towards Text-guided 3D Scene Composition [134.71933134180782]
  Existing approaches either leverage large text-to-image models to optimize a 3D representation or train 3D generators on object-centric datasets. We introduce SceneWiz3D, a novel approach to synthesize high-fidelity 3D scenes from text.
  arXiv  Detail & Related papers  (2023-12-13T18:59:30Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.