SketchSense: Learning to Interpret Imperfect Sketch Guidance for Image Inpainting

• URL: http://arxiv.org/abs/2608.13186v1
• Date: Thu, 13 Aug 2026 12:50:02 GMT
• Title: SketchSense: Learning to Interpret Imperfect Sketch Guidance for Image Inpainting
• Abstract summary: Sketch-guided image inpainting provides intuitive structural control.<n>Real sketches often mix reliable global intent with locally crowded, displaced, incomplete, or deliberately unconventional strokes.<n>We propose SketchSense, a framework that interprets imperfect sketch guidance by synchronously denoising interacting RGB and structure streams.
• Score: 0.913755431537592
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Sketch-guided image inpainting provides intuitive structural control, yet real sketches often mix reliable global intent with locally crowded, displaced, incomplete, or deliberately unconventional strokes. Existing approaches typically either retain the input sketch as a fixed condition throughout denoising or refine it into a clean structure before RGB synthesis. The former assumes uniformly reliable strokes and can propagate local errors throughout generation; the latter must resolve ambiguous structure before emerging appearance and semantic context become available. We propose SketchSense, a framework that interprets imperfect sketch guidance by synchronously denoising interacting RGB and structure streams. Bidirectional Attention Fusion couples appearance generation with structural recovery, producing a refined structure that exposes the model's evolving sketch interpretation. A phrase-level objective aligns the semantic grounding of the two streams. Sketch-Aware Spatial Regulation further adapts sketch use to local generation states by modulating attention and the fusion process, while an optional signed prior injects preserve-versus-correct intent into feature representations and attention behavior. Experiments on natural and structurally complex imagery show substantial gains over existing methods in both restoration quality and structural fidelity.

Related papers

• SegDem: Segmentation helps Demosaicing [52.515779223891705]
  Image demosaicing reconstructs a full-color image from incomplete color measurements produced by a sensor covered with a color filter (CFA)<n>Most existing methods formulate demosaicing as pixel-level reconstruction.<n>SegDem is a cross-task decoder representation transfer framework for demosaicing.
  arXiv  Detail & Related papers  (2026-08-08T04:49:20Z)
• Compositional Video Generation via Inference-Time Guidance [69.53614395025632]
  Text-to-video diffusion models often fail on prompts requiring compositional understanding.<n>We propose textbfCVG, an inference-time guidance method for improving compositional faithfulness in frozen text-to-video models.
  arXiv  Detail & Related papers  (2026-05-14T15:50:25Z)
• Semantic-Structural Alignment for Generative Pictorial Charts [59.45629259524998]
  We present a generative framework for the automated synthesis of pictorial charts.<n>We frame the problem as a dual-conditioned generation task guided by two parallel external control signals.<n>Our method produces pictorial charts that are both artistically compelling and structurally consistent.
  arXiv  Detail & Related papers  (2026-05-05T05:20:46Z)
• Multi-Level Conditioning by Pairing Localized Text and Sketch for Fashion Image Generation [14.962452069195544]
  We present LOcalized Text and Sketch with multi-level guidance (LOTS)<n>LOTS combines global sketch guidance with multiple localized sketch-text pairs.<n>We develop Sketchy, the first fashion dataset where multiple text-sketch pairs are provided per image.
  arXiv  Detail & Related papers  (2026-02-20T16:07:31Z)
• Stroke2Sketch: Harnessing Stroke Attributes for Training-Free Sketch Generation [54.053878919317526]
  Stroke2Sketch is a training-free framework that introduces cross-image stroke attention.<n>We develop adaptive contrast enhancement and semantic-focused attention to reinforce content preservation and foreground emphasis.<n>Stroke2Sketch effectively synthesizes stylistically faithful sketches, outperforming existing methods in expressive stroke control and semantic coherence.
  arXiv  Detail & Related papers  (2025-10-18T03:07:56Z)
• StrokeFusion: Vector Sketch Generation via Joint Stroke-UDF Encoding and Latent Sequence Diffusion [13.862427684807486]
  StrokeFusion is a two-stage framework for vector sketch generation.<n>It contains a dual-modal sketch feature learning network that maps strokes into a high-quality latent space.<n>It exploits a stroke-level latent diffusion model that simultaneously adjusts stroke position, scale, and trajectory during generation.
  arXiv  Detail & Related papers  (2025-03-31T06:03:03Z)
• Recovering Partially Corrupted Objects via Sketch-Guided Bidirectional Feature Interaction [16.03488741913531]
  Text-guided diffusion models provide high-level semantic guidance through text prompts.<n>They often lack precise pixel-level spatial control in partially corrupted objects.<n>We propose a sketch-guided bidirectional feature interaction framework built upon a pretrained Stable Diffusion model.
  arXiv  Detail & Related papers  (2025-03-10T08:34:31Z)
• Sketch-Guided Scenery Image Outpainting [83.6612152173028]
  We propose an encoder-decoder based network to conduct sketch-guided outpainting. We apply a holistic alignment module to make the synthesized part be similar to the real one from the global view. Second, we reversely produce the sketches from the synthesized part and encourage them be consistent with the ground-truth ones.
  arXiv  Detail & Related papers  (2020-06-17T11:34:36Z)
• Guidance and Evaluation: Semantic-Aware Image Inpainting for Mixed Scenes [54.836331922449666]
  We propose a Semantic Guidance and Evaluation Network (SGE-Net) to update the structural priors and the inpainted image. It utilizes semantic segmentation map as guidance in each scale of inpainting, under which location-dependent inferences are re-evaluated. Experiments on real-world images of mixed scenes demonstrated the superiority of our proposed method over state-of-the-art approaches.
  arXiv  Detail & Related papers  (2020-03-15T17:49:20Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.