Imagine Before You Draw: Visual Prompt Engineering for Image Generation

• URL: http://arxiv.org/abs/2606.04457v1
• Date: Wed, 03 Jun 2026 05:01:36 GMT
• Title: Imagine Before You Draw: Visual Prompt Engineering for Image Generation
• Abstract summary: We propose Visual Prompt Engineering (VPE), which can be seamlessly integrated into internal frameworks.<n>We validate VPE across class-conditional generation, text-to-image generation, and image editing.<n>Results show that VPE can accelerate convergence, raise quality ceilings, and through internal integration, achieve substantially better editing preservation.
• Score: 67.81347924426714
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Incorporating visual semantic representations as an intermediate step before image generation can reduce the modeling difficulty between text and images, thereby improving generation quality. Recent works such as X-Omni and BLIP3o-Next have explored this direction, but they typically use a two-stage external pipeline: a separate autoregressive model first generates semantic tokens, which are then fed as conditioning to an independent diffusion decoder. Since the decoder cannot jointly access the original input and the semantic plan, this design introduces an information bottleneck that limits detail preservation in downstream tasks such as editing. Internal architectures such as Transfusion, BAGEL, and Show-o2 avoid this bottleneck by enabling cross-modal interaction within a single model, but they still face the difficult text-to-pixel modeling gap without intermediate semantic guidance. We propose Visual Prompt Engineering (VPE), which can be seamlessly integrated into such internal frameworks. Specifically, the model first autoregressively generates visual semantic tokens (e.g., SigLIP 2) as "visual prompts" that capture the semantic layout, then generates the full image tokens conditioned on this plan. We validate VPE across class-conditional generation, text-to-image generation, and image editing, covering various token types and model architectures. Results show that VPE can accelerate convergence, raise quality ceilings, and through internal integration, achieve substantially better editing preservation (PSNR: 26.76 vs. 19.92) than external alternatives of the same parameter scale, while maintaining competitive editing responsiveness.

Related papers

• UniSpace: Unified Visual Representation and Scalable Multimodal Modeling [38.3945115074626]
  We study whether understanding, generation, and editing can be modeled in a single visual representation space built from a pretrained semantic ViT.<n>We introduce emphPatch Re parameterization, which preserves the original semantic pathway while adding a reconstruction-aware patch embedding.<n>The resulting unified representation preserves multimodal understanding while enabling high-fidelity image reconstruction and a favorable reconstruction--generation trade-off.
  arXiv  Detail & Related papers  (2026-08-09T12:46:56Z)
• Aura: Consistent Multi-Subject Video Generation via VLM-Grounded Semantic Alignment [20.811984469709508]
  Aura is a unified framework for high-fidelity and identity-consistent video generation.<n>We introduce AI director-level captions that provide dense and structured descriptions of video content.<n>We build a high-quality video-subject image dataset through a dedicated data construction pipeline.
  arXiv  Detail & Related papers  (2026-07-05T13:54:33Z)
• OmniGen-AR: AutoRegressive Any-to-Image Generation [89.27614937810836]
  We present OmniGen-AR, a unified autoregressive framework for Any-to-Image generation.<n>By discretizing various visual conditions through a shared visual tokenizer and text prompts with a text tokenizer, OmniGen-AR supports a broad spectrum of conditional inputs within a single model.
  arXiv  Detail & Related papers  (2026-06-08T07:47:53Z)
• End-to-End Training for Unified Tokenization and Latent Denoising [82.91537591286554]
  We propose UNITE - an autoencoder architecture for unified tokenization and latent diffusion.<n>UNITE consists of a Generative that serves as both image tokenizer and latent generator via weight sharing.<n>We show that single stage joint training of tokenization & generation from scratch is feasible.
  arXiv  Detail & Related papers  (2026-03-23T17:59:49Z)
• Edit2Interp: Adapting Image Foundation Models from Spatial Editing to Video Frame Interpolation with Few-Shot Learning [5.6616478694920405]
  We show that a large image editing model (Qwen-Image-Edit) can be adapted for Video Frame Interpolation (VFI) using only 64-256 training samples via Low-Rank Adaptation (LoRA)<n>Our work establishes that foundation image editing models possess untapped potential for temporal tasks, offering a data-efficient pathway for video synthesis in resource-constrained scenarios.
  arXiv  Detail & Related papers  (2026-03-16T09:07:25Z)
• Multimodal Representation Alignment for Image Generation: Text-Image Interleaved Control Is Easier Than You Think [38.258453761376586]
  We propose Dream Engine, an efficient framework designed for arbitrary text-image interleaved control in image generation models.<n>Our approach utilizes a two-stage training paradigm, consisting of joint text-image alignment and multimodal interleaved instruction tuning.<n>Our experiments demonstrate that this training method is effective, achieving a 0.69 overall score on the GenEval benchmark.
  arXiv  Detail & Related papers  (2025-02-27T15:08:39Z)
• Prompt-Consistency Image Generation (PCIG): A Unified Framework Integrating LLMs, Knowledge Graphs, and Controllable Diffusion Models [20.19571676239579]
  We introduce a novel diffusion-based framework to enhance the alignment of generated images with their corresponding descriptions. Our framework is built upon a comprehensive analysis of inconsistency phenomena, categorizing them based on their manifestation in the image. We then integrate a state-of-the-art controllable image generation model with a visual text generation module to generate an image that is consistent with the original prompt.
  arXiv  Detail & Related papers  (2024-06-24T06:12:16Z)
• MM-Interleaved: Interleaved Image-Text Generative Modeling via Multi-modal Feature Synchronizer [106.79844459065828]
  This paper presents MM-Interleaved, an end-to-end generative model for interleaved image-text data. It introduces a multi-scale and multi-image feature synchronizer module, allowing direct access to fine-grained image features in the previous context. Experiments demonstrate the versatility of MM-Interleaved in recognizing visual details following multi-modal instructions and generating consistent images following both textual and visual conditions.
  arXiv  Detail & Related papers  (2024-01-18T18:50:16Z)
• BLIP-Diffusion: Pre-trained Subject Representation for Controllable Text-to-Image Generation and Editing [73.74570290836152]
  BLIP-Diffusion is a new subject-driven image generation model that supports multimodal control. Unlike other subject-driven generation models, BLIP-Diffusion introduces a new multimodal encoder which is pre-trained to provide subject representation.
  arXiv  Detail & Related papers  (2023-05-24T04:51:04Z)
• eDiffi: Text-to-Image Diffusion Models with an Ensemble of Expert Denoisers [87.52504764677226]
  Large-scale diffusion-based generative models have led to breakthroughs in text-conditioned high-resolution image synthesis. We train an ensemble of text-to-image diffusion models specialized for different stages synthesis. Our ensemble of diffusion models, called eDiffi, results in improved text alignment while maintaining the same inference cost.
  arXiv  Detail & Related papers  (2022-11-02T17:43:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.