InstanceControl: Controllable Complex Image Generation without Instance Labeling

• URL: http://arxiv.org/abs/2606.31924v1
• Date: Tue, 30 Jun 2026 16:33:03 GMT
• Title: InstanceControl: Controllable Complex Image Generation without Instance Labeling
• Abstract summary: We propose InstanceControl, a novel controllable generation method that eliminates the need for instance labeling.<n>We leverage the Vision-Language Model (VLM) to establish instance-level correspondences between text prompts and visual conditions.<n>Our approach outperforms state-of-the-art methods, achieving superior fidelity and precise instance-level control.
• Score: 57.00586546940602
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Controllable image generation methods, such as ControlNet, have demonstrated a remarkable capacity to introduce visual conditions(e.g., depth maps) to guide image generation. However, these methods often struggle with complex multi-instance scenes, frequently leading to attribute confusion among instances. While recent approaches attempt to mitigate this via manual instance labeling, such requirements are labor-intensive. In this paper, we propose InstanceControl, a novel multi-instance controllable generation method that eliminates the need for instance labeling. We identify the primary bottleneck in existing methods as the inability to accurately associate instance descriptions with their corresponding regions within visual conditions. To address this, we leverage the Vision-Language Model (VLM) to establish instance-level correspondences between text prompts and visual conditions. Specifically, the VLM automatically parses instance descriptions from the text prompts and simultaneously predicts instance masks based on the visual conditions. Furthermore, since the predicted masks may contain noise, we introduce an adaptive mask refinement strategy that dynamically refines these instance masks during the generation process. Extensive experiments demonstrate that our approach outperforms state-of-the-art methods, achieving superior fidelity and precise instance-level control.

Related papers

• MoGen: A Unified Collaborative Framework for Controllable Multi-Object Image Generation [76.94658056824422]
  MoGen is a user-friendly multi-object image generation method.<n>First, we design a Regional Semantic Anchor (RSA) module that precisely anchors phrase units in language descriptions to their corresponding image regions.<n>We introduce an Adaptive Multi-modal Guidance (AMG) module, which adaptively parses and integrates various combinations of multi-source control signals.
  arXiv  Detail & Related papers  (2026-01-09T05:57:48Z)
• A Simple yet Powerful Instance-Aware Prompting Framework for Training-free Camouflaged Object Segmentation [6.712332323439369]
  We propose a training-free Camouflaged Object pipeline that explicitly converts a task-generic prompt into fine-grained instance masks.<n>The proposed IAPF significantly surpasses existing state-of-the-art training-free COS methods.
  arXiv  Detail & Related papers  (2025-08-09T09:35:32Z)
• LLM-guided Instance-level Image Manipulation with Diffusion U-Net Cross-Attention Maps [5.836227628651603]
  We propose a pipeline leveraging Large Language Models, open-vocabulary detectors, cross-attention maps and diffusion U-Net for instance-level image manipulation.<n>Our method detects objects mentioned in the prompt and present in the generated image, enabling precise manipulation without extensive training or input masks.
  arXiv  Detail & Related papers  (2025-01-23T19:26:14Z)
• Generating Compositional Scenes via Text-to-image RGBA Instance Generation [82.63805151691024]
  Text-to-image diffusion generative models can generate high quality images at the cost of tedious prompt engineering. We propose a novel multi-stage generation paradigm that is designed for fine-grained control, flexibility and interactivity. Our experiments show that our RGBA diffusion model is capable of generating diverse and high quality instances with precise control over object attributes.
  arXiv  Detail & Related papers  (2024-11-16T23:44:14Z)
• Open-Vocabulary Segmentation with Unpaired Mask-Text Supervision [87.15580604023555]
  Unpair-Seg is a novel weakly-supervised open-vocabulary segmentation framework. It learns from unpaired image-mask and image-text pairs, which can be independently and efficiently collected. It achieves 14.6% and 19.5% mIoU on the ADE-847 and PASCAL Context-459 datasets.
  arXiv  Detail & Related papers  (2024-02-14T06:01:44Z)
• InstanceDiffusion: Instance-level Control for Image Generation [89.31908006870422]
  InstanceDiffusion adds precise instance-level control to text-to-image diffusion models. We propose three major changes to text-to-image models that enable precise instance-level control.
  arXiv  Detail & Related papers  (2024-02-05T18:49:17Z)
• Mask-free OVIS: Open-Vocabulary Instance Segmentation without Manual Mask Annotations [86.47908754383198]
  Open-Vocabulary (OV) methods leverage large-scale image-caption pairs and vision-language models to learn novel categories. Our method generates pseudo-mask annotations by leveraging the localization ability of a pre-trained vision-language model for objects present in image-caption pairs. Our method trained with just pseudo-masks significantly improves the mAP scores on the MS-COCO dataset and OpenImages dataset.
  arXiv  Detail & Related papers  (2023-03-29T17:58:39Z)
• GANSeg: Learning to Segment by Unsupervised Hierarchical Image Generation [16.900404701997502]
  We propose a GAN-based approach that generates images conditioned on latent masks. We show that such mask-conditioned image generation can be learned faithfully when conditioning the masks in a hierarchical manner. It also lets us generate image-mask pairs for training a segmentation network, which outperforms the state-of-the-art unsupervised segmentation methods on established benchmarks.
  arXiv  Detail & Related papers  (2021-12-02T07:57:56Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.