InstructHumans: Editing Animated 3D Human Textures with Instructions

• URL: http://arxiv.org/abs/2404.04037v2
• Date: Mon, 15 Sep 2025 13:11:30 GMT
• Title: InstructHumans: Editing Animated 3D Human Textures with Instructions
• Authors: Jiayin Zhu, Linlin Yang, Angela Yao,
• Abstract summary: We present InstructHumans, a novel framework for instruction-driven animatable 3D human texture editing.<n>Existing text-based 3D editing methods often directly apply Score Distillation Sampling (SDS)<n>We propose a modified SDS for Editing (SDS-E) that selectively incorporates subterms of SDS across diffusion timesteps.
• Score: 59.8568078884315
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We present InstructHumans, a novel framework for instruction-driven {animatable} 3D human texture editing. Existing text-based 3D editing methods often directly apply Score Distillation Sampling (SDS). SDS, designed for generation tasks, cannot account for the defining requirement of editing -- maintaining consistency with the source avatar. This work shows that naively using SDS harms editing, as it may destroy consistency. We propose a modified SDS for Editing (SDS-E) that selectively incorporates subterms of SDS across diffusion timesteps. We further enhance SDS-E with spatial smoothness regularization and gradient-based viewpoint sampling for edits with sharp and high-fidelity detailing. Incorporating SDS-E into a 3D human texture editing framework allows us to outperform existing 3D editing methods. Our avatars faithfully reflect the textual edits while remaining consistent with the original avatars. Project page: https://jyzhu.top/instruct-humans/.

Related papers

• Towards Scalable and Consistent 3D Editing [32.16698854719098]
  3D editing has wide applications in immersive content creation, digital entertainment, and AR/VR.<n>Unlike 2D editing, it remains challenging due to the need for cross-view consistency, structural fidelity, and fine-grained controllability.<n>We introduce 3DEditVerse, the largest paired 3D editing benchmark to date, comprising 116,309 high-quality training pairs and 1,500 curated test pairs.<n>On the model side, we propose 3DEditFormer, a 3D-structure-preserving conditional transformer.
  arXiv  Detail & Related papers  (2025-10-03T13:34:55Z)
• Rethinking Score Distilling Sampling for 3D Editing and Generation [50.52808917055502]
  Unified Distillation Sampling (UDS) is a method that seamlessly integrates the generation and editing of 3D assets.<n>UDS not only outperforms baseline methods in generating 3D assets with richer details but also excels in editing tasks, thereby bridging the gap between 3D generation and editing.
  arXiv  Detail & Related papers  (2025-05-03T18:40:39Z)
• Semantic Score Distillation Sampling for Compositional Text-to-3D Generation [28.88237230872795]
  Generating high-quality 3D assets from textual descriptions remains a pivotal challenge in computer graphics and vision research. We introduce a novel SDS approach, designed to improve the expressiveness and accuracy of compositional text-to-3D generation. Our approach integrates new semantic embeddings that maintain consistency across different rendering views. By leveraging explicit semantic guidance, our method unlocks the compositional capabilities of existing pre-trained diffusion models.
  arXiv  Detail & Related papers  (2024-10-11T17:26:00Z)
• Revealing Directions for Text-guided 3D Face Editing [52.85632020601518]
  3D face editing is a significant task in multimedia, aimed at the manipulation of 3D face models across various control signals. We present Face Clan, a text-general approach for generating and manipulating 3D faces based on arbitrary attribute descriptions. Our method offers a precisely controllable manipulation method, allowing users to intuitively customize regions of interest with the text description.
  arXiv  Detail & Related papers  (2024-10-07T12:04:39Z)
• EditRoom: LLM-parameterized Graph Diffusion for Composable 3D Room Layout Editing [114.14164860467227]
  We propose EditRoom, a framework capable of executing a variety of layout edits through natural language commands.<n>Specifically, EditRoom leverages Large Language Models (LLMs) for command planning and generates target scenes.<n>We have developed an automatic pipeline to augment existing 3D scene datasets and introduced EditRoom-DB, a large-scale dataset with 83k editing pairs.
  arXiv  Detail & Related papers  (2024-10-03T17:42:24Z)
• Preserving Identity with Variational Score for General-purpose 3D Editing [48.314327790451856]
  Piva is a novel optimization-based method for editing images and 3D models based on diffusion models. We pinpoint the limitations in 2D and 3D editing, which causes detail loss and oversaturation. We propose an additional score distillation term that enforces identity preservation.
  arXiv  Detail & Related papers  (2024-06-13T09:32:40Z)
• GSEdit: Efficient Text-Guided Editing of 3D Objects via Gaussian Splatting [10.527349772993796]
  We present GSEdit, a pipeline for text-guided 3D object editing based on Gaussian Splatting models. Our method enables the editing of the style and appearance of 3D objects without altering their main details, all in a matter of minutes on consumer hardware.
  arXiv  Detail & Related papers  (2024-03-08T08:42:23Z)
• Plasticine3D: 3D Non-Rigid Editing with Text Guidance by Multi-View Embedding Optimization [21.8454418337306]
  We propose Plasticine3D, a novel text-guided controlled 3D editing pipeline that can perform 3D non-rigid editing. Our work divides the editing process into a geometry editing stage and a texture editing stage to achieve separate control of structure and appearance. For the purpose of fine-grained control, we propose Embedding-Fusion (EF) to blend the original characteristics with the editing objectives in the embedding space.
  arXiv  Detail & Related papers  (2023-12-15T09:01:54Z)
• SHAP-EDITOR: Instruction-guided Latent 3D Editing in Seconds [73.91114735118298]
  Shap-Editor is a novel feed-forward 3D editing framework. We demonstrate that direct 3D editing in this space is possible and efficient by building a feed-forward editor network.
  arXiv  Detail & Related papers  (2023-12-14T18:59:06Z)
• MaTe3D: Mask-guided Text-based 3D-aware Portrait Editing [61.014328598895524]
  We propose textbfMaTe3D: mask-guided text-based 3D-aware portrait editing. New SDF-based 3D generator learns local and global representations with proposed SDF and density consistency losses. Conditional Distillation on Geometry and Texture (CDGT) mitigates visual ambiguity and avoids mismatch between texture and geometry.
  arXiv  Detail & Related papers  (2023-12-12T03:04:08Z)
• PaintHuman: Towards High-fidelity Text-to-3D Human Texturing via Denoised Score Distillation [89.09455618184239]
  Recent advances in text-to-3D human generation have been groundbreaking. We propose a model called PaintHuman to address the challenges from two aspects. We use the depth map as a guidance to ensure realistic semantically aligned textures.
  arXiv  Detail & Related papers  (2023-10-14T00:37:16Z)
• Directional Texture Editing for 3D Models [51.31499400557996]
  ITEM3D is designed for automatic textbf3D object editing according to the text textbfInstructions. Leveraging the diffusion models and the differentiable rendering, ITEM3D takes the rendered images as the bridge of text and 3D representation.
  arXiv  Detail & Related papers  (2023-09-26T12:01:13Z)
• Delta Denoising Score [51.98288453616375]
  We introduce Delta Denoising Score (DDS), a novel scoring function for text-based image editing. It guides minimal modifications of an input image towards the content described in a target prompt.
  arXiv  Detail & Related papers  (2023-04-14T12:22:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.