InstructMesh: Selective Refinement of Generative 3D Models for Fabrication
Abstract Overview
InstructMesh is an interactive post-generation tool designed to repair fabrication-relevant flaws in AI-generated 3D models through user region selection and targeted latent-space editing. The paper begins with a formative analysis of 120 Thingiverse-based reconstructions generated with Trellis, showing that fabrication-related geometric flaws are widespread and frequently co-occur. Based on these findings, the authors introduce a set of canonical additive and subtractive latent operations accessible via natural language or slider controls, accompanied by preview visualizations before model regeneration. A technical evaluation and two user studies demonstrate how novices can effectively identify and correct functional geometric flaws using the system.
Novelty
The paper distinguishes itself by framing fabrication repair of generative 3D outputs as selective latent-space voxel editing rather than explicit mesh manipulation or iterative text re-prompting. It couples region-based selection with a fixed vocabulary of parameterized canonical geometric operations, in-context LLM instruction mapping, and WYSIWYG-style preview visualizations.
Results
In technical benchmarks, the system achieved high flaw repair rates, including 96.3% for missing openings and 95.65% for wall-thickness issues, with GPT-4 correctly predicting canonical operations with 92.1% accuracy. In user study I, novices identified 90.4% and successfully repaired 89.7% of annotated fabrication flaws. In user study II, all 12 participants favored a hybrid workflow combining natural language and slider controls, rating the preview visualizations highly for decision support and clarity.
Key Points
- A formative study on 120 reconstructed Thingiverse models revealed that 78.3% had multiple fabrication-related flaws, averaging 2.4 issues per model.
- InstructMesh modifies intermediate voxel latent representations via six canonical additive and subtractive operations, supporting both natural language and slider interfaces alongside visual previews.
- Novice participants without 3D modeling experience successfully identified 90.4% and repaired 89.7% of evaluated flaws, expressing unanimous preference for a hybrid interaction mode.