Correlation-Weighted Multi-Reward Optimization for Compositional Generation

• URL: http://arxiv.org/abs/2603.18528v1
• Date: Thu, 19 Mar 2026 06:19:18 GMT
• Title: Correlation-Weighted Multi-Reward Optimization for Compositional Generation
• Abstract summary: ours is a framework that balances competing reward signals and emphasizes concepts that are partially satisfied yet inconsistently generated across samples.<n>We apply our approach to train state-of-the-art diffusion models, SD3.5 and FLUX.1-dev, and demonstrate consistent improvements on challenging multi-concept benchmarks.
• Score: 5.347765461028618
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Text-to-image models produce images that align well with natural language prompts, but compositional generation has long been a central challenge. Models often struggle to satisfy multiple concepts within a single prompt, frequently omitting some concepts and resulting in partial success. Such failures highlight the difficulty of jointly optimizing multiple concepts during reward optimization, where competing concepts can interfere with one another. To address this limitation, we propose Correlation-Weighted Multi-Reward Optimization (\ours), a framework that leverages the correlation structure among concept rewards to adaptively weight each attribute concept in optimization. By accounting for interactions among concepts, \ours balances competing reward signals and emphasizes concepts that are partially satisfied yet inconsistently generated across samples, improving compositional generation. Specifically, we decompose multi-concept prompts into pre-defined concept groups (\eg, objects, attributes, and relations) and obtain reward signals from dedicated reward models for each concept. We then adaptively reweight these rewards, assigning higher weights to conflicting or hard-to-satisfy concepts using correlation-based difficulty estimation. By focusing optimization on the most challenging concepts within each group, \ours encourages the model to consistently satisfy all requested attributes simultaneously. We apply our approach to train state-of-the-art diffusion models, SD3.5 and FLUX.1-dev, and demonstrate consistent improvements on challenging multi-concept benchmarks, including ConceptMix, GenEval 2, and T2I-CompBench.

Related papers

• CPC-VAR:Continual Personalized and Compositional Generation in Visual Autoregressive Models [69.9266117908314]
  Visual autoregressive ( VAR) models have emerged as efficient paradigm for text-to-image generation.<n>Existing VAR-based personalization methods fail to accommodate evolving user demands.<n>We present the first systematic study of continual personalized generation in VAR models.
  arXiv  Detail & Related papers  (2026-05-19T12:18:15Z)
• TILT: Improving Compositional Generation in Diffusion Models with a Model-Intrinsic Reward [7.47572316039482]
  We present TILT, a training-free framework for compositional text-to-image generation via test-time reward alignment.<n>We interpret compositional failures as overlap modes between joint and single-concept distributions, and define a reward that favors samples where all concepts are jointly present.<n>Experiments on prompts from T2ICompBench show that our method improves compositional alignment while preserving image quality compared to previous baselines.
  arXiv  Detail & Related papers  (2026-05-16T16:56:52Z)
• Concept-wise Attention for Fine-grained Concept Bottleneck Models [8.484128495313607]
  Concept-wise Attention for Fine-grained Concept Bottleneck Models (CoAt-CBM) is a novel framework that achieves adaptive fine-grained image-concept alignment and high interpretability.<n>CoAt-CBM employs learnable concept-wise visual queries to adaptively obtain fine-grained concept-wise visual embeddings, which are then used to produce a concept score vector.
  arXiv  Detail & Related papers  (2026-04-17T06:43:30Z)
• Beyond Heuristic Prompting: A Concept-Guided Bayesian Framework for Zero-Shot Image Recognition [81.2779530670268]
  Vision-Language Models (VLMs) have significantly advanced zero-shot image recognition.<n>This paper enhances prompt by incorporating class-specific concepts.<n>Our method consistently outperforms state-of-the-art approaches.
  arXiv  Detail & Related papers  (2026-03-09T03:11:11Z)
• RMLer: Synthesizing Novel Objects across Diverse Categories via Reinforcement Mixing Learning [16.682831359982064]
  Reinforcement Mixing Learning (RMLer) is a framework that formulates cross-category concept fusion.<n>Our work provides a robust framework for generating novel visual concepts, with promising applications in film, gaming, and design.
  arXiv  Detail & Related papers  (2025-12-22T11:44:32Z)
• LoRACLR: Contrastive Adaptation for Customization of Diffusion Models [84.04930416829264]
  LoRACLR is a novel approach for multi-concept image generation that merges multiple LoRA models, each fine-tuned for a distinct concept, into a single, unified model.<n>LoRACLR uses a contrastive objective to align and merge the weight spaces of these models, ensuring compatibility while minimizing interference.<n>Our results highlight the effectiveness of LoRACLR in accurately merging multiple concepts, advancing the capabilities of personalized image generation.
  arXiv  Detail & Related papers  (2024-12-12T18:59:55Z)
• MC$^2$: Multi-concept Guidance for Customized Multi-concept Generation [59.00909718832648]
  We propose MC$2$, a novel approach for multi-concept customization.<n>By adaptively refining attention weights between visual and textual tokens, our method ensures that image regions accurately correspond to their associated concepts.<n>Experiments demonstrate that MC$2$ outperforms training-based methods in terms of prompt-reference alignment.
  arXiv  Detail & Related papers  (2024-04-08T07:59:04Z)
• LoRA-Composer: Leveraging Low-Rank Adaptation for Multi-Concept Customization in Training-Free Diffusion Models [33.379758040084894]
  Multi-concept customization emerges as the challenging task within this domain. Existing approaches often rely on training a fusion matrix of multiple Low-Rank Adaptations (LoRAs) to merge various concepts into a single image. LoRA-Composer is a training-free framework designed for seamlessly integrating multiple LoRAs.
  arXiv  Detail & Related papers  (2024-03-18T09:58:52Z)
• Separable Multi-Concept Erasure from Diffusion Models [52.51972530398691]
  We propose a Separable Multi-concept Eraser (SepME) to eliminate unsafe concepts from large-scale diffusion models. The latter separates optimizable model weights, making each weight increment correspond to a specific concept erasure. Extensive experiments indicate the efficacy of our approach in eliminating concepts, preserving model performance, and offering flexibility in the erasure or recovery of various concepts.
  arXiv  Detail & Related papers  (2024-02-03T11:10:57Z)
• Multi-Concept Customization of Text-to-Image Diffusion [51.8642043743222]
  We propose Custom Diffusion, an efficient method for augmenting existing text-to-image models. We find that only optimizing a few parameters in the text-to-image conditioning mechanism is sufficiently powerful to represent new concepts. Our model generates variations of multiple new concepts and seamlessly composes them with existing concepts in novel settings.
  arXiv  Detail & Related papers  (2022-12-08T18:57:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.