UniCustom: Unified Visual Conditioning for Multi-Reference Image Generation

• URL: http://arxiv.org/abs/2605.12088v2
• Date: Wed, 13 May 2026 15:41:37 GMT
• Title: UniCustom: Unified Visual Conditioning for Multi-Reference Image Generation
• Abstract summary: We propose a unified visual conditioning framework that fuses ViT and VAE features before VLM encoding.<n>Experiments on two multi-reference generation benchmarks demonstrate that UniCustom consistently improves subject consistency, instruction following, and compositional fidelity.
• Score: 65.53694602893042
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Multi-reference image generation aims to synthesize images from textual instructions while faithfully preserving subject identities from multiple reference images. Existing VLM-enhanced diffusion models commonly rely on decoupled visual conditioning: semantic ViT features are processed by the VLM for instruction understanding, whereas appearance-rich VAE features are injected later into the diffusion backbone. Despite its intuitive design, this separation makes it difficult for the model to associate each semantically grounded subject with visual details from the correct reference image. As a result, the model may recognize which subject is being referred to, but fail to preserve its identity and fine-grained appearance, leading to attribute leakage and cross-reference confusion in complex multi-reference settings. To address this issue, we propose UniCustom, a unified visual conditioning framework that fuses ViT and VAE features before VLM encoding. This early fusion exposes the VLM to both semantic cues and appearance-rich details, enabling its hidden states to jointly encode the referred subject and corresponding visual appearance with only a lightweight linear fusion layer. To learn such unified representations, we adopt a two-stage training strategy: reconstruction-oriented pretraining that preserves reference-specific appearance details in the fused hidden states, followed by supervised finetuning on single- and multi-reference generation tasks. We further introduce a slot-wise binding regularization that encourages each image slot to preserve low-level details of its corresponding reference, thereby reducing cross-reference entanglement. Experiments on two multi-reference generation benchmarks demonstrate that UniCustom consistently improves subject consistency, instruction following, and compositional fidelity over strong baselines.

Related papers

• Aura: Consistent Multi-Subject Video Generation via VLM-Grounded Semantic Alignment [20.811984469709508]
  Aura is a unified framework for high-fidelity and identity-consistent video generation.<n>We introduce AI director-level captions that provide dense and structured descriptions of video content.<n>We build a high-quality video-subject image dataset through a dedicated data construction pipeline.
  arXiv  Detail & Related papers  (2026-07-05T13:54:33Z)
• Unified Multimodal Autoregressive Modeling with Shared Context-Visual Tokenizer is Key to Unification [80.62512020268626]
  UniAR is a unified autoregressive framework where a single visual tokenizer serves as the key bridge between understanding and generation.<n>UniAR adapts a pretrained vision encoder with multi-level feature fusion and a lookup-free bitwise quantization scheme.<n>A diffusion-based visual decoder operates on discrete visual tokens to decode high-fidelity images.
  arXiv  Detail & Related papers  (2026-06-16T17:59:22Z)
• Squeezing Capacity from Multimodal Large Language Models for Subject-driven Generation [22.419513267677278]
  Existing approaches often encode text and reference images separately.<n>Recent frameworks that connect multimodal models and diffusion models improve instruction following, but largely overlook identity preservation.<n>We condition diffusion models on Multimodal Large Language Models that jointly encode text and reference images, and augment it with VAE-based identity conditioning.<n>Our approach harmonizes multimodal understanding with identity preservation, mitigates copy-paste issues, and achieves superior performance regarding human preference on subject-driven image generation.
  arXiv  Detail & Related papers  (2026-05-25T17:59:35Z)
• Multimodal Large Language Models for Multi-Subject In-Context Image Generation [56.20395856287325]
  Music is the first MLLM specifically designed for textbfMUlti-textbfSubject textbfIn-textbfContext image generation.<n>We enhance the model's understanding of multi-subject semantic relationships through a vision chain-of-thought mechanism.<n>By incorporating complex subject images during training, we improve the model's capacity for chained reasoning.
  arXiv  Detail & Related papers  (2026-04-08T15:37:42Z)
• UniRef-Image-Edit: Towards Scalable and Consistent Multi-Reference Image Editing [33.64590153603506]
  We present UniRef-Image-Edit, a high-performance multi-modal generation system.<n>It unifies single-image editing and multi-image composition within a single framework.
  arXiv  Detail & Related papers  (2026-02-15T15:24:03Z)
• Hierarchical Concept-to-Appearance Guidance for Multi-Subject Image Generation [22.845591588026366]
  We propose a framework that provides explicit, structured supervision from high-level concepts to fine-grained appearances.<n>At the conceptual level, we introduce a VAE dropout training strategy that randomly omits reference VAE features.<n>At the appearance level, we integrate the VLM-derived correspondences into a correspondence-aware masked attention module.
  arXiv  Detail & Related papers  (2026-02-03T12:13:29Z)
• Towards Generalized Multi-Image Editing for Unified Multimodal Models [56.620038824933566]
  Unified Multimodal Models (UMMs) integrate multimodal understanding and generation.<n>UMMs are limited to maintaining visual consistency and disambiguating visual cues when referencing details across multiple input images.<n>We propose a scalable multi-image editing framework for UMMs that explicitly distinguishes image identities and generalizes to variable input counts.
  arXiv  Detail & Related papers  (2026-01-09T06:42:49Z)
• Omni-Attribute: Open-vocabulary Attribute Encoder for Visual Concept Personalization [82.31106470150844]
  We introduce Omni-Attribute, the first open-vocabulary image attribute encoder to learn attribute-specific representations.<n>We use a dual-objective training paradigm that balances generative fidelity with contrastive disentanglement.<n>The resulting embeddings prove effective for open-vocabulary attribute retrieval, personalization, and compositional generation.
  arXiv  Detail & Related papers  (2025-12-11T18:59:56Z)
• DRC: Enhancing Personalized Image Generation via Disentangled Representation Composition [69.10628479553709]
  We introduce DRC, a novel personalized image generation framework that enhances Large Multimodal Models (LMMs)<n> DRC explicitly extracts user style preferences and semantic intentions from history images and the reference image, respectively.<n>It involves two critical learning stages: 1) Disentanglement learning, which employs a dual-tower disentangler to explicitly separate style and semantic features, optimized via a reconstruction-driven paradigm with difficulty-aware importance sampling; and 2) Personalized modeling, which applies semantic-preserving augmentations to effectively adapt the disentangled representations for robust personalized generation.
  arXiv  Detail & Related papers  (2025-04-24T08:10:10Z)
• LeftRefill: Filling Right Canvas based on Left Reference through Generalized Text-to-Image Diffusion Model [55.20469538848806]
  LeftRefill is an innovative approach to harness large Text-to-Image (T2I) diffusion models for reference-guided image synthesis. This paper introduces LeftRefill, an innovative approach to efficiently harness large Text-to-Image (T2I) diffusion models for reference-guided image synthesis.
  arXiv  Detail & Related papers  (2023-05-19T10:29:42Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.