MiLDEdit: Reasoning-Based Multi-Layer Design Document Editing

• URL: http://arxiv.org/abs/2601.04589v1
• Date: Thu, 08 Jan 2026 04:38:07 GMT
• Title: MiLDEdit: Reasoning-Based Multi-Layer Design Document Editing
• Authors: Zihao Lin, Wanrong Zhu, Jiuxiang Gu, Jihyung Kil, Christopher Tensmeyer, Lin Zhang, Shilong Liu, Ruiyi Zhang, Lifu Huang, Vlad I. Morariu, Tong Sun,
• Abstract summary: We introduce the Multi-Layer Document Editing Agent (MiLDEAgent)<n>MiLDEAgent is a reasoning-based framework that combines an RL-trained multimodal reasoner for layer-wise understanding with an image editor for targeted modifications.<n>MiLDEAgent achieves strong layer-aware reasoning and precise editing, significantly outperforming all open-source baselines and attaining performance comparable to closed-source models.
• Score: 67.28337411397062
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Real-world design documents (e.g., posters) are inherently multi-layered, combining decoration, text, and images. Editing them from natural-language instructions requires fine-grained, layer-aware reasoning to identify relevant layers and coordinate modifications. Prior work largely overlooks multi-layer design document editing, focusing instead on single-layer image editing or multi-layer generation, which assume a flat canvas and lack the reasoning needed to determine what and where to modify. To address this gap, we introduce the Multi-Layer Document Editing Agent (MiLDEAgent), a reasoning-based framework that combines an RL-trained multimodal reasoner for layer-wise understanding with an image editor for targeted modifications. To systematically benchmark this setting, we introduce the MiLDEBench, a human-in-the-loop corpus of over 20K design documents paired with diverse editing instructions. The benchmark is complemented by a task-specific evaluation protocol, MiLDEEval, which spans four dimensions including instruction following, layout consistency, aesthetics, and text rendering. Extensive experiments on 14 open-source and 2 closed-source models reveal that existing approaches fail to generalize: open-source models often cannot complete multi-layer document editing tasks, while closed-source models suffer from format violations. In contrast, MiLDEAgent achieves strong layer-aware reasoning and precise editing, significantly outperforming all open-source baselines and attaining performance comparable to closed-source models, thereby establishing the first strong baseline for multi-layer document editing.

Related papers

• MultiEdit: Advancing Instruction-based Image Editing on Diverse and Challenging Tasks [46.87912659985628]
  MultiEdit is a comprehensive dataset featuring over 107K high-quality image editing samples.<n>It encompasses 6 challenging editing tasks through a diverse collection of 18 non-style-transfer editing types and 38 style transfer operations.<n>We employ a novel dataset construction pipeline that utilizes two multi-modal large language models (MLLMs) to generate visual-adaptive editing instructions.
  arXiv  Detail & Related papers  (2025-09-18T05:33:38Z)
• Image Editing As Programs with Diffusion Models [69.05164729625052]
  We introduce Image Editing As Programs (IEAP), a unified image editing framework built upon the Diffusion Transformer (DiT) architecture.<n>IEAP approaches instructional editing through a reductionist lens, decomposing complex editing instructions into sequences of atomic operations.<n>Our framework delivers superior accuracy and semantic fidelity, particularly for complex, multi-step instructions.
  arXiv  Detail & Related papers  (2025-06-04T16:57:24Z)
• What Changed? Detecting and Evaluating Instruction-Guided Image Edits with Multimodal Large Language Models [88.398085358514]
  DICE is a model designed to detect localized differences between the original and the edited image.<n>It is trained using a strategy that leverages self-supervision, distillation from inpainting networks, and full supervision.<n>We demonstrate that DICE effectively identifies coherent edits, effectively evaluating images generated by different editing models with a strong correlation with human judgment.
  arXiv  Detail & Related papers  (2025-05-26T18:00:10Z)
• ImgEdit: A Unified Image Editing Dataset and Benchmark [14.185771939071149]
  We introduce ImgEdit, a large-scale, high-quality image-editing dataset comprising 1.2 million carefully curated edit pairs.<n>ImgEdit surpasses existing datasets in both task novelty and data quality.<n>For comprehensive evaluation, we introduce ImgEdit-Bench, a benchmark designed to evaluate image editing performance.
  arXiv  Detail & Related papers  (2025-05-26T17:53:33Z)
• BrushEdit: All-In-One Image Inpainting and Editing [76.93556996538398]
  BrushEdit is a novel inpainting-based instruction-guided image editing paradigm.<n>We devise a system enabling free-form instruction editing by integrating MLLMs and a dual-branch image inpainting model.<n>Our framework effectively combines MLLMs and inpainting models, achieving superior performance across seven metrics.
  arXiv  Detail & Related papers  (2024-12-13T17:58:06Z)
• GenArtist: Multimodal LLM as an Agent for Unified Image Generation and Editing [60.09562648953926]
  GenArtist is a unified image generation and editing system coordinated by a multimodal large language model (MLLM) agent. We integrate a comprehensive range of existing models into the tool library and utilize the agent for tool selection and execution. Experiments demonstrate that GenArtist can perform various generation and editing tasks, achieving state-of-the-art performance.
  arXiv  Detail & Related papers  (2024-07-08T04:30:53Z)
• LayeredDoc: Domain Adaptive Document Restoration with a Layer Separation Approach [9.643486775455841]
  This paper introduces a text-graphic layer separation approach that enhances domain adaptability in document image restoration systems. We propose LayeredDoc, which utilizes two layers of information: the first targets coarse-grained graphic components, while the second refines machine-printed textual content. We evaluate our approach both qualitatively and quantitatively using a new real-world dataset, LayeredDocDB, developed for this study.
  arXiv  Detail & Related papers  (2024-06-12T19:41:01Z)
• LayerDiff: Exploring Text-guided Multi-layered Composable Image Synthesis via Layer-Collaborative Diffusion Model [70.14953942532621]
  Layer-collaborative diffusion model, named LayerDiff, is designed for text-guided, multi-layered, composable image synthesis. Our model can generate high-quality multi-layered images with performance comparable to conventional whole-image generation methods. LayerDiff enables a broader range of controllable generative applications, including layer-specific image editing and style transfer.
  arXiv  Detail & Related papers  (2024-03-18T16:28:28Z)
• An Item is Worth a Prompt: Versatile Image Editing with Disentangled Control [21.624984690721842]
  D-Edit is a framework to disentangle the comprehensive image-prompt interaction into several item-prompt interactions.<n>It is based on pretrained diffusion models with cross-attention layers disentangled and adopts a two-step optimization to build item-prompt associations.<n>We demonstrate state-of-the-art results in four types of editing operations including image-based, text-based, mask-based editing, and item removal.
  arXiv  Detail & Related papers  (2024-03-07T20:06:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.