ScribbleEdit: Synthetic Data for Image Editing with Scribbles and Text

• URL: http://arxiv.org/abs/2605.01135v2
• Date: Tue, 05 May 2026 03:57:17 GMT
• Title: ScribbleEdit: Synthetic Data for Image Editing with Scribbles and Text
• Abstract summary: We introduce ScribbleEdit, a large-scale synthetic dataset designed to bridge this gap.<n>We evaluate and finetune both diffusion-based and autoregressive unified multimodal image editing models.<n>Our experiments reveal that while off-the-shelf models struggle with abstract scribble inputs, finetuning on our synthetic dataset significantly improves their ability to generate spatially aligned and semantically consistent edits.
• Score: 26.951462580463883
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Recent progress in generative models has significantly advanced image editing capabilities, yet precise and intuitive user control remains difficult. Specifically, users often struggle to communicate both exact spatial layouts and specific semantic details simultaneously. While natural language instructions effectively convey high-level semantics like texture and color, they lack spatial specificity. Conversely, freehand scribbles provide rough spatial boundaries but cannot express detailed visual attributes. Consequently, achieving precise control requires combining both modalities. However, existing models struggle to jointly interpret abstract scribbles alongside text due to a lack of specialized training data. In this work, we introduce ScribbleEdit, a large-scale synthetic dataset designed to bridge this gap by combining natural language instructions with freehand scribble inputs for more accurate, controllable edits. We construct this dataset through a synthetic pipeline that automatically generates source-target image pairs via inpainting, which are then paired with human-drawn scribbles and VLM-generated text instructions. Using ScribbleEdit, we evaluate and finetune both diffusion-based and autoregressive unified multimodal image editing models. Our experiments reveal that while off-the-shelf models struggle with abstract scribble inputs, finetuning on our synthetic dataset significantly improves their ability to generate spatially aligned and semantically consistent edits.

Related papers

• SceneCraft: Interactive System for Image Editing via Scene Graph [17.007784605446716]
  SceneCraft is a novel framework that bridges user intent and model execution by representing images as editable scene graphs.<n>Instead of guessing text prompts through trial and error, users interact directly with a visual graph to perform complex spatial and relational operations.<n>We show that SceneCraft provides a more intuitive control mechanism, significantly reducing the cognitive burden of manual prompt engineering.
  arXiv  Detail & Related papers  (2026-06-15T01:34:22Z)
• All-in-One Conditioning for Text-to-Image Synthesis [45.22434803596108]
  We propose a novel approach that grounds text-to-image synthesis within the framework of scene graph structures.<n>We introduce a zero-shot, scene graph-based conditioning mechanism that generates soft visual guidance during inference.<n>This enables the model to maintain text-image alignment while supporting lightweight, coherent, and diverse image synthesis.
  arXiv  Detail & Related papers  (2026-02-09T20:16:19Z)
• ScribbleSense: Generative Scribble-Based Texture Editing with Intent Prediction [5.109590115201006]
  ScribbleSense is an editing method that combines multimodal large language models (MLLMs) and image generation models.<n>We leverage the visual capabilities of MLLMs to predict the editing intent behind the scribbles.<n>Globally generated images are employed to extract local texture details.
  arXiv  Detail & Related papers  (2026-01-30T01:55:44Z)
• DreamOmni3: Scribble-based Editing and Generation [72.52583595391944]
  We introduce Dream Omni3, tackling two challenges: data creation and framework design.<n>For scribble-based editing, we define four tasks: scribble and instruction-based editing, scribble and multimodal instruction-based generation, and doodle generation.<n>For the framework, instead of using binary masks, we propose a joint input scheme that feeds both the original and scribbled source images into the model.
  arXiv  Detail & Related papers  (2025-12-27T09:07:12Z)
• SAEdit: Token-level control for continuous image editing via Sparse AutoEncoder [52.754326452329956]
  We introduce a method for disentangled and continuous editing through token-level manipulation of text embeddings.<n>The edits are applied by manipulating the embeddings along carefully chosen directions, which control the strength of the target attribute.<n>Our method operates directly on text embeddings without modifying the diffusion process, making it model agnostic and broadly applicable to various image backbones.
  arXiv  Detail & Related papers  (2025-10-06T17:51:04Z)
• Scribble-Guided Diffusion for Training-free Text-to-Image Generation [17.930032337081673]
  Scribble-Guided Diffusion (ScribbleDiff) is a training-free approach that utilizes simple user-provided scribbles as visual prompts to guide image generation. We introduce moment alignment and scribble propagation, which allow for more effective and flexible alignment between generated images and scribble inputs.
  arXiv  Detail & Related papers  (2024-09-12T13:13:07Z)
• DM-Align: Leveraging the Power of Natural Language Instructions to Make Changes to Images [55.546024767130994]
  We propose a novel model to enhance the text-based control of an image editor by explicitly reasoning about which parts of the image to alter or preserve. It relies on word alignments between a description of the original source image and the instruction that reflects the needed updates, and the input image. It is evaluated on a subset of the Bison dataset and a self-defined dataset dubbed Dream.
  arXiv  Detail & Related papers  (2024-04-27T22:45:47Z)
• Text2Data: Low-Resource Data Generation with Textual Control [100.5970757736845]
  Text2Data is a novel approach that utilizes unlabeled data to understand the underlying data distribution.<n>It undergoes finetuning via a novel constraint optimization-based learning objective that ensures controllability and effectively counteracts catastrophic forgetting.
  arXiv  Detail & Related papers  (2024-02-08T03:41:39Z)
• Paste, Inpaint and Harmonize via Denoising: Subject-Driven Image Editing with Pre-Trained Diffusion Model [22.975965453227477]
  We introduce a new framework called textitPaste, Inpaint and Harmonize via Denoising (PhD) In our experiments, we apply PhD to both subject-driven image editing tasks and explore text-driven scene generation given a reference subject.
  arXiv  Detail & Related papers  (2023-06-13T07:43:10Z)
• iEdit: Localised Text-guided Image Editing with Weak Supervision [53.082196061014734]
  We propose a novel learning method for text-guided image editing. It generates images conditioned on a source image and a textual edit prompt. It shows favourable results against its counterparts in terms of image fidelity, CLIP alignment score and qualitatively for editing both generated and real images.
  arXiv  Detail & Related papers  (2023-05-10T07:39:14Z)
• DiffEdit: Diffusion-based semantic image editing with mask guidance [64.555930158319]
  DiffEdit is a method to take advantage of text-conditioned diffusion models for the task of semantic image editing. Our main contribution is able to automatically generate a mask highlighting regions of the input image that need to be edited.
  arXiv  Detail & Related papers  (2022-10-20T17:16:37Z)
• Prompt-to-Prompt Image Editing with Cross Attention Control [41.26939787978142]
  We present an intuitive prompt-to-prompt editing framework, where the edits are controlled by text only. We show our results over diverse images and prompts, demonstrating high-quality synthesis and fidelity to the edited prompts.
  arXiv  Detail & Related papers  (2022-08-02T17:55:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.