VGGT-Edit: Feed-forward Native 3D Scene Editing with Residual Field Prediction

• URL: http://arxiv.org/abs/2605.15186v2
• Date: Tue, 19 May 2026 03:07:48 GMT
• Title: VGGT-Edit: Feed-forward Native 3D Scene Editing with Residual Field Prediction
• Abstract summary: VGGT-Edit is a feed-forward framework for text-conditioned native 3D scene editing.<n>We introduce depth-synchronized text injection to align semantic guidance with the backbone's spatial poses.<n> VGGT-Edit substantially outperforms 2D-lifting baselines, producing sharper object details, stronger multi-view consistency, and near-instant inference speed.
• Score: 59.303842406260124
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: High-quality 3D scene reconstruction has recently advanced toward generalizable feed-forward architectures, enabling the generation of complex environments in a single forward pass. However, despite their strong performance in static scene perception, these models remain limited in responding to dynamic human instructions, which restricts their use in interactive applications. Existing editing methods typically rely on a 2D-lifting strategy, where individual views are edited independently and then lifted back into 3D space. This indirect pipeline often leads to blurry textures and inconsistent geometry, as 2D editors lack the spatial awareness required to preserve structure across viewpoints. To address these limitations, we propose VGGT-Edit, a feed-forward framework for text-conditioned native 3D scene editing. VGGT-Edit introduces depth-synchronized text injection to align semantic guidance with the backbone's spatial poses, ensuring stable instruction grounding. This semantic signal is then processed by a residual transformation head, which directly predicts 3D geometric displacements to deform the scene while preserving background stability. To ensure high-fidelity results, we supervise the framework with a multi-term objective function that enforces geometric accuracy and cross-view consistency. We also construct the DeltaScene Dataset, a large-scale dataset generated through an automated pipeline with 3D agreement filtering to ensure ground-truth quality. Experiments show that VGGT-Edit substantially outperforms 2D-lifting baselines, producing sharper object details, stronger multi-view consistency, and near-instant inference speed. The project page is https://chriszkxxx.github.io/VGGT-Edit/.

Related papers

• Robust Prior-Guided Segmentation for Editable 3D Gaussian Splatting [6.290910602456353]
  Existing approaches that lift 2D segmentations to the 3D domain suffer from view inconsistencies and coarse masks.<n>We propose a novel framework that leverages the Segment Anything Model High Quality to generate accurate 2D masks.<n>Our approach achieves state-of-the-art segmentation accuracy and enables interactive, real-time object editing while maintaining high visual fidelity.
  arXiv  Detail & Related papers  (2026-05-15T15:29:30Z)
• GemDepth: Geometry-Embedded Features for 3D-Consistent Video Depth [12.866152238833104]
  Video depth estimation extends monocular prediction into the temporal domain to ensure coherence.<n>Current approaches, which primarily rely on temporal smoothing via Transformers, struggle to maintain strict 3D geometric consistency.<n>We propose GemDepth, a framework built on the insight that an explicit awareness of camera motion and global 3D structure is a prerequisite for 3D consistency.
  arXiv  Detail & Related papers  (2026-05-11T13:11:54Z)
• TRACE: High-Fidelity 3D Scene Editing via Tangible Reconstruction and Geometry-Aligned Contextual Video Masking [38.40189748889072]
  Mesh-guided 3DGS editing framework achieves automated, high-fidelity scene transformation.<n>By anchoring video diffusion with explicit 3D geometry, TRACE enables fine-grained, part-level manipulatio.
  arXiv  Detail & Related papers  (2026-04-01T17:51:00Z)
• Geometry-Guided Reinforcement Learning for Multi-view Consistent 3D Scene Editing [106.07976338405793]
  Leveraging the priors of 2D diffusion models for 3D editing has emerged as a promising paradigm.<n>We propose textbfRL3DEdit, a single-pass framework driven by reinforcement learning with novel rewards derived from the 3D foundation model, VGGT.<n>Experiments demonstrate that RL3DEdit achieves stable multi-view consistency and outperforms state-of-the-art methods in editing quality with high efficiency.
  arXiv  Detail & Related papers  (2026-03-03T16:31:10Z)
• Easy3E: Feed-Forward 3D Asset Editing via Rectified Voxel Flow [29.8200628539749]
  We propose an effective and fully feedforward 3D editing framework based on the TRELLIS generative backbone.<n>Our framework addresses two key issues: adapting training-free 2D editing to structured 3D representations, and overcoming the bottleneck of appearance fidelity in compressed 3D features.
  arXiv  Detail & Related papers  (2026-02-25T02:15:14Z)
• Interp3D: Correspondence-aware Interpolation for Generative Textured 3D Morphing [63.141976759536625]
  We propose Interp3D, a training-free framework for textured 3D morphing.<n>It harnesses generative priors and adopts a progressive alignment principle to ensure both geometric fidelity and texture coherence.<n>For comprehensive evaluations, we construct a dedicated dataset, Interp3DData, with graded difficulty levels and assess generation results from fidelity, transition smoothness, and plausibility.
  arXiv  Detail & Related papers  (2026-01-20T16:03:22Z)
• Error-Driven Scene Editing for 3D Grounding in Large Language Models [71.41120775319088]
  Despite recent progress in 3D-LLMs, they remain limited in accurately grounding language to visual and spatial elements in 3D environments.<n>This limitation stems in part from training data that focuses on language reasoning rather than spatial understanding due to scarce 3D resources.<n>We propose 3D scene editing as a key mechanism to generate precise visual counterfactuals that mitigate these biases.
  arXiv  Detail & Related papers  (2025-11-18T03:13:29Z)
• IGGT: Instance-Grounded Geometry Transformer for Semantic 3D Reconstruction [82.53307702809606]
  Humans naturally perceive the geometric structure and semantic content of a 3D world as intertwined dimensions.<n>We propose InstanceGrounded Geometry Transformer (IGGT) to unify the knowledge for both spatial reconstruction and instance-level contextual understanding.
  arXiv  Detail & Related papers  (2025-10-26T14:57:44Z)
• InterGSEdit: Interactive 3D Gaussian Splatting Editing with 3D Geometry-Consistent Attention Prior [3.6657066043195585]
  InterGSEdit is a novel framework for high-quality 3DGS editing via interactively selecting key views with users' preferences.<n>We project $GAP3D$ to obtain 3D-constrained attention, which are fused with 2D cross-attention via Attention Fusion Network (AFN)
  arXiv  Detail & Related papers  (2025-07-07T13:04:26Z)
• 3DOT: Texture Transfer for 3DGS Objects from a Single Reference Image [31.972069558992946]
  3D texture swapping allows for the customization of 3D object textures.<n>No dedicated method exists, but adapted 2D editing and text-driven 3D editing approaches can serve this purpose.<n>We introduce 3DSwapping, a 3D texture swapping method that integrates progressive generation, view-consistency gradient guidance, and prompt-tuned gradient guidance.
  arXiv  Detail & Related papers  (2025-03-24T16:31:52Z)
• SyncNoise: Geometrically Consistent Noise Prediction for Text-based 3D Scene Editing [58.22339174221563]
  We propose SyncNoise, a novel geometry-guided multi-view consistent noise editing approach for high-fidelity 3D scene editing. SyncNoise synchronously edits multiple views with 2D diffusion models while enforcing multi-view noise predictions to be geometrically consistent. Our method achieves high-quality 3D editing results respecting the textual instructions, especially in scenes with complex textures.
  arXiv  Detail & Related papers  (2024-06-25T09:17:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.