Discrete Preference Learning for Personalized Multimodal Generation

• URL: http://arxiv.org/abs/2604.20434v1
• Date: Wed, 22 Apr 2026 10:53:28 GMT
• Title: Discrete Preference Learning for Personalized Multimodal Generation
• Abstract summary: We present a framework called Discrete Preference learning for Personalized Multimodal Generation (DPPMG)<n>In the first stage, to accurately learn discrete modal-specific preferences, we introduce a modal-specific graph neural network.<n>In the second stage, the discrete modal-specific preference tokens are injected into downstream text and image generators.
• Score: 30.948146467300802
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The emergence of generative models enables the creation of texts and images tailored to users' preferences. Existing personalized generative models have two critical limitations: lacking a dedicated paradigm for accurate preference modeling, and generating unimodal content despite real-world multimodal-driven user interactions. Therefore, we propose personalized multimodal generation, which captures modal-specific preferences via a dedicated preference model from multimodal interactions, and then feeds them into downstream generators for personalized multimodal content. However, this task presents two challenges: (1) Gap between continuous preferences from dedicated modeling and discrete token inputs intrinsic to generator architectures; (2) Potential inconsistency between generated images and texts. To tackle these, we present a two-stage framework called Discrete Preference learning for Personalized Multimodal Generation (DPPMG). In the first stage, to accurately learn discrete modal-specific preferences, we introduce a modal-specific graph neural network (a dedicated preference model) to learn users' modal-specific preferences, which preferences are then quantized into discrete preference tokens. In the second stage, the discrete modal-specific preference tokens are injected into downstream text and image generators. To further enhance cross-modal consistency while preserving personalization, we design a cross-modal consistent and personalized reward to fine-tune token-associated parameters. Extensive experiments on two real-world datasets demonstrate the effectiveness of our model in generating personalized and consistent multimodal content.

Related papers

• Omni-Diffusion: Unified Multimodal Understanding and Generation with Masked Discrete Diffusion [66.78268790956493]
  We introduce Omni-Diffusion, the first any-to-any multimodal language model built entirely on mask-based discrete diffusion models.<n>Our method outperforms or performs on par with existing multimodal systems that process two or more modalities.
  arXiv  Detail & Related papers  (2026-03-06T18:59:57Z)
• PrefGen: Multimodal Preference Learning for Preference-Conditioned Image Generation [44.638427866399205]
  Preference-conditioned image generation seeks to adapt generative models to individual users, producing outputs that reflect personal aesthetic choices beyond the given prompt.<n>We propose a framework that leverages multimodal large language models to extract rich user representations and inject them into diffusion-based image generation.<n>Our method substantially outperforms strong baselines in both image quality and preference alignment.
  arXiv  Detail & Related papers  (2025-12-04T02:57:29Z)
• NExT-OMNI: Towards Any-to-Any Omnimodal Foundation Models with Discrete Flow Matching [64.10695425442164]
  We introduce NExT-OMNI, an open-source omnimodal foundation model that achieves unified modeling through discrete flow paradigms.<n>Trained on large-scale interleaved text, image, video, and audio data, NExT-OMNI delivers competitive performance on multimodal generation and understanding benchmarks.<n>To advance further research, we release training details, data protocols, and open-source both the code and model checkpoints.
  arXiv  Detail & Related papers  (2025-10-15T16:25:18Z)
• Progressive Semantic Residual Quantization for Multimodal-Joint Interest Modeling in Music Recommendation [6.790539226766362]
  We propose a novel multimodal recommendation framework with two stages.<n>In the first stage, our method generates modal-specific and modal-joint semantic IDs.<n>In the second stage, to model multimodal interest of users, a Multi-Codebook Cross-Attention network is designed.
  arXiv  Detail & Related papers  (2025-08-28T02:16:57Z)
• Draw Your Mind: Personalized Generation via Condition-Level Modeling in Text-to-Image Diffusion Models [5.282669911393826]
  We propose DrUM, a novel method that integrates user profiling with a transformer-based adapter to enable personalized generation.<n>DrUM demonstrates strong performance on large-scale datasets and seamlessly integrates with open-source text encoders.
  arXiv  Detail & Related papers  (2025-08-05T14:14:55Z)
• Mogao: An Omni Foundation Model for Interleaved Multi-Modal Generation [54.588082888166504]
  We present Mogao, a unified framework that enables interleaved multi-modal generation through a causal approach.<n>Mogoo integrates a set of key technical improvements in architecture design, including a deep-fusion design, dual vision encoders, interleaved rotary position embeddings, and multi-modal classifier-free guidance.<n>Experiments show that Mogao achieves state-of-the-art performance in multi-modal understanding and text-to-image generation, but also excels in producing high-quality, coherent interleaved outputs.
  arXiv  Detail & Related papers  (2025-05-08T17:58:57Z)
• Personalized Preference Fine-tuning of Diffusion Models [75.22218338096316]
  We introduce PPD, a multi-reward optimization objective that aligns diffusion models with personalized preferences.<n>With PPD, a diffusion model learns the individual preferences of a population of users in a few-shot way.<n>Our approach achieves an average win rate of 76% over Stable Cascade, generating images that more accurately reflect specific user preferences.
  arXiv  Detail & Related papers  (2025-01-11T22:38:41Z)
• QARM: Quantitative Alignment Multi-Modal Recommendation at Kuaishou [23.818456863262494]
  We introduce a quantitative multi-modal framework to customize the specialized and trainable multi-modal information for different downstream models. Inspired by the two difficulties challenges in downstream tasks usage, we introduce a quantitative multi-modal framework to customize the specialized and trainable multi-modal information for different downstream models.
  arXiv  Detail & Related papers  (2024-11-18T17:08:35Z)
• U3M: Unbiased Multiscale Modal Fusion Model for Multimodal Semantic Segmentation [63.31007867379312]
  We introduce U3M: An Unbiased Multiscale Modal Fusion Model for Multimodal Semantics. We employ feature fusion at multiple scales to ensure the effective extraction and integration of both global and local features. Experimental results demonstrate that our approach achieves superior performance across multiple datasets.
  arXiv  Detail & Related papers  (2024-05-24T08:58:48Z)
• Towards Unified Multi-Modal Personalization: Large Vision-Language Models for Generative Recommendation and Beyond [87.1712108247199]
  Our goal is to establish a Unified paradigm for Multi-modal Personalization systems (UniMP) We develop a generic and personalization generative framework, that can handle a wide range of personalized needs. Our methodology enhances the capabilities of foundational language models for personalized tasks.
  arXiv  Detail & Related papers  (2024-03-15T20:21:31Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.