Chat-Edit-3D++: Interactive 3D and 4D Scene Editing via Large Language Models
Abstract Overview
This paper presents CE3D++, an interactive framework for editing 3D and monocular 4D scenes through multi-round dialogue with a large language model. Its core technical idea is Hash-Atlas, which maps scene views to 2D foreground and background atlases so that editing can be performed in atlas space and projected back to the scene, decoupling 2D editing from 3D/4D reconstruction. On top of this representation, the system uses an LLM to interpret open-ended user instructions, decide when visual tools are needed, and orchestrate a broad suite of external vision models. The framework also supports dynamic 4D scenes by introducing pivot-view initialization and a motion tracking loss to maintain atlas stability for moving objects.
Novelty
The main novelty is combining a decoupled hash-based atlas representation with an LLM-driven dialogue agent for interactive 3D and 4D scene editing. The method introduces a pivot-view initialization and motion loss specifically for dynamic 4D scenes, alongside a trajectory-tuning dataset that enables smaller LLMs to reliably orchestrate over 30 external visual tools for multi-step editing tasks.
Results
Across multiple 3D and 4D datasets, Hash-Atlas outperforms LNA in atlas reconstruction with reported improvements of 1.1–5.1 dB PSNR, 14.2–18.6x faster training, and 7.6–9.0x higher inference FPS. In editing benchmarks, CE3D++ achieves higher CLIP similarity and CLIP directional scores with faster editing times compared to existing 3D and 4D editing baselines. Furthermore, trajectory tuning markedly reduces tool-calling failure rates for lightweight LLMs such as Qwen-14B and LLaMA-8B on complex multi-turn editing workflows.
Key Points
- Hash-Atlas reformulates 3D and 4D scene editing as 2D atlas manipulation, making the editing stage broadly compatible with standard 2D image models without retraining scene representations.
- CE3D++ employs an LLM-based dialogue agent with abstract scene identifiers and an executor merge-split strategy to parse free-form user instructions and orchestrate visual tools across multi-round interactions.
- For dynamic 4D scenes, pivot-view selection and a point-tracking motion loss stabilize moving-object atlases, while a specialized trajectory-tuning dataset ensures reliable tool scheduling on smaller LLMs.