DesignAgent3D: Interactive 3D Scene Editing via Designer-like Multimodal Reasoning

• URL: http://arxiv.org/abs/2608.21438v1
• Date: Tue, 18 Aug 2026 01:16:13 GMT
• Title: DesignAgent3D: Interactive 3D Scene Editing via Designer-like Multimodal Reasoning
• Abstract summary: Text guided 3D scene editing provides an intuitive interface for modifying reconstructed environments.<n>Existing methods typically formulate the task as one-shot conditional generation from a single prompt.<n>We present DesignAgent3D, an interactive multimodal agentic framework that reformulates 3D scene editing as a designer-like Plan-Perceive-Act paradigm.
• Score: 32.91291969659809
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Text guided 3D scene editing provides an intuitive interface for modifying reconstructed environments, but remains difficult because natural language design requests are often semantically underspecified and must be grounded in cluttered 3D scenes. Existing methods typically formulate the task as one-shot conditional generation from a single prompt, failing to resolve ambiguous user intents or achieve precise spatial grounding. Consequently, they suffer from severe object localization drift, tracking failure under occlusions, and the notorious multi-view "sticker effect." To overcome these limitations, we present DesignAgent3D, an interactive multimodal agentic framework that reformulates 3D scene editing as a designer-like Plan-Perceive-Act paradigm. The agent first plans by interacting with the user to clarify underspecified design goals, then perceives by grounding the intended edit to specific objects or regions in the 3D scene, and finally acts by applying controlled visual modifications while preserving scene consistency. The edits are further integrated into the underlying 3D representation, supporting persistent and multi-view consistent novel-view rendering. Extensive experiments across both NeRF and 3D Gaussian Splatting backbones demonstrate that DesignAgent3D significantly outperforms state-of-the-art baselines, delivering superior semantic intent alignment, impeccable spatial localization accuracy, and high-fidelity multi-view consistency.

Related papers

• Native3D: End-to-End 3D Scene Generation via Unified Mesh-Texture Modeling and Semantic Alignment [8.98466538269363]
  Native3D is the first end-to-end 3D scene generation framework that completely bypasses 2D intermediate representations.<n>We propose a unified mesh-texture joint representation that simultaneously models both geometric structures and texture features through a Transformer-based scene encoder.
  arXiv  Detail & Related papers  (2026-06-05T10:13:17Z)
• VGGT-Edit: Feed-forward Native 3D Scene Editing with Residual Field Prediction [59.303842406260124]
  VGGT-Edit is a feed-forward framework for text-conditioned native 3D scene editing.<n>We introduce depth-synchronized text injection to align semantic guidance with the backbone's spatial poses.<n> VGGT-Edit substantially outperforms 2D-lifting baselines, producing sharper object details, stronger multi-view consistency, and near-instant inference speed.
  arXiv  Detail & Related papers  (2026-05-14T17:59:04Z)
• SDesc3D: Towards Layout-Aware 3D Indoor Scene Generation from Short Descriptions [71.54559024212976]
  3D indoor scene generation conditioned on short textual descriptions provides a promising avenue for interactive 3D environment construction.<n>Existing works suffer from poor physical plausibility and insufficient detail richness in such semantic condensation cases.<n>We propose SDesc3D, a short-text conditioned 3D indoor scene generation framework, that leverages multi-view structural priors and regional functionality implications.
  arXiv  Detail & Related papers  (2026-04-02T12:33:01Z)
• Unified Semantic Transformer for 3D Scene Understanding [55.415468022487005]
  We introduce UNITE, a novel feed-forward neural network that unifies a diverse set of 3D semantic tasks within a single model.<n>Our model operates on unseen scenes in a fully end-to-end manner and only takes a few seconds to infer the full 3D semantic geometry.<n>We demonstrate that UNITE achieves state-of-the-art performance on several different semantic tasks and even outperforms task-specific models.
  arXiv  Detail & Related papers  (2025-12-16T12:49:35Z)
• Error-Driven Scene Editing for 3D Grounding in Large Language Models [71.41120775319088]
  Despite recent progress in 3D-LLMs, they remain limited in accurately grounding language to visual and spatial elements in 3D environments.<n>This limitation stems in part from training data that focuses on language reasoning rather than spatial understanding due to scarce 3D resources.<n>We propose 3D scene editing as a key mechanism to generate precise visual counterfactuals that mitigate these biases.
  arXiv  Detail & Related papers  (2025-11-18T03:13:29Z)
• Mastering Regional 3DGS: Locating, Initializing, and Editing with Diverse 2D Priors [67.22744959435708]
  3D semantic parsing often underperforms compared to its 2D counterpart, making targeted manipulations within 3D spaces more difficult and limiting the fidelity of edits.<n>We address this problem by leveraging 2D diffusion editing to accurately identify modification regions in each view, followed by inverse rendering for 3D localization.<n> Experiments demonstrate that our method achieves state-of-the-art performance while delivering up to a $4times$ speedup.
  arXiv  Detail & Related papers  (2025-07-07T19:15:43Z)
• InterGSEdit: Interactive 3D Gaussian Splatting Editing with 3D Geometry-Consistent Attention Prior [3.6657066043195585]
  InterGSEdit is a novel framework for high-quality 3DGS editing via interactively selecting key views with users' preferences.<n>We project $GAP3D$ to obtain 3D-constrained attention, which are fused with 2D cross-attention via Attention Fusion Network (AFN)
  arXiv  Detail & Related papers  (2025-07-07T13:04:26Z)
• 3DSceneEditor: Controllable 3D Scene Editing with Gaussian Splatting [31.98493679748211]
  We propose 3DSceneEditor, a fully 3D-based paradigm for real-time, precise editing of 3D scenes using Gaussian Splatting.<n>Unlike conventional methods, 3DSceneEditor operates through a streamlined 3D pipeline, enabling direct manipulation of Gaussians for efficient, high-quality edits.
  arXiv  Detail & Related papers  (2024-12-02T15:03:55Z)
• Chat-Edit-3D: Interactive 3D Scene Editing via Text Prompts [76.73043724587679]
  We propose a dialogue-based 3D scene editing approach, termed CE3D. Hash-Atlas represents 3D scene views, which transfers the editing of 3D scenes onto 2D atlas images. Results demonstrate that CE3D effectively integrates multiple visual models to achieve diverse editing visual effects.
  arXiv  Detail & Related papers  (2024-07-09T13:24:42Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.