MAUGen: A Unified Diffusion Approach for Multi-Identity Facial Expression and AU Label Generation

• URL: http://arxiv.org/abs/2602.00583v1
• Date: Sat, 31 Jan 2026 07:56:22 GMT
• Title: MAUGen: A Unified Diffusion Approach for Multi-Identity Facial Expression and AU Label Generation
• Authors: Xiangdong Li, Ye Lou, Ao Gao, Wei Zhang, Siyang Song,
• Abstract summary: We propose MAUGen, a diffusion-based multi-modal framework that jointly generates a large collection of photorealistic facial expressions.<n>Under this framework, we introduce Multi-Identity Facial Action (MIFA), a large-scale multimodal synthetic dataset featuring comprehensive AU annotations and identity variations.
• Score: 18.996319133901473
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: The lack of large-scale, demographically diverse face images with precise Action Unit (AU) occurrence and intensity annotations has long been recognized as a fundamental bottleneck in developing generalizable AU recognition systems. In this paper, we propose MAUGen, a diffusion-based multi-modal framework that jointly generates a large collection of photorealistic facial expressions and anatomically consistent AU labels, including both occurrence and intensity, conditioned on a single descriptive text prompt. Our MAUGen involves two key modules: (1) a Multi-modal Representation Learning (MRL) module that captures the relationships among the paired textual description, facial identity, expression image, and AU activations within a unified latent space; and (2) a Diffusion-based Image label Generator (DIG) that decodes the joint representation into aligned facial image-label pairs across diverse identities. Under this framework, we introduce Multi-Identity Facial Action (MIFA), a large-scale multimodal synthetic dataset featuring comprehensive AU annotations and identity variations. Extensive experiments demonstrate that MAUGen outperforms existing methods in synthesizing photorealistic, demographically diverse facial images along with semantically aligned AU labels.

Related papers

• MICON-Bench: Benchmarking and Enhancing Multi-Image Context Image Generation in Unified Multimodal Models [89.89575486159795]
  We introduce textbfMICON-Bench, a benchmark for multi-image context generation.<n>We propose an MLLM-driven Evaluation-by-Checkpoint framework for automatic verification of semantic and visual consistency.<n>We also present textbfDynamic Attention Rebalancing (DAR), a training-free, plug-and-play mechanism that dynamically adjusts attention during inference to enhance coherence and reduce hallucinations.
  arXiv  Detail & Related papers  (2026-02-23T04:32:52Z)
• Towards Generalized Multi-Image Editing for Unified Multimodal Models [56.620038824933566]
  Unified Multimodal Models (UMMs) integrate multimodal understanding and generation.<n>UMMs are limited to maintaining visual consistency and disambiguating visual cues when referencing details across multiple input images.<n>We propose a scalable multi-image editing framework for UMMs that explicitly distinguishes image identities and generalizes to variable input counts.
  arXiv  Detail & Related papers  (2026-01-09T06:42:49Z)
• Seg4Diff: Unveiling Open-Vocabulary Segmentation in Text-to-Image Diffusion Transformers [56.76198904599581]
  Text-to-image diffusion models excel at translating language prompts into implicitly grounding concepts through their cross-modal attention mechanisms.<n>Recent multi-modal diffusion transformers extend this by introducing joint self-attentiond image and text tokens, enabling richer and more scalable cross-modal alignment.<n>We introduce Seg4Diff, a systematic framework for analyzing the attention structures of MM-DiT, with a focus on how specific layers propagate semantic information from text to image.
  arXiv  Detail & Related papers  (2025-09-22T17:59:54Z)
• Identity-Preserving Text-to-Image Generation via Dual-Level Feature Decoupling and Expert-Guided Fusion [35.67333978414322]
  We propose a novel framework that improves the separation of identity-related and identity-unrelated features.<n>Our framework consists of two key components: an Implicit-Explicit foreground-background Decoupling Module and a Feature Fusion Module.
  arXiv  Detail & Related papers  (2025-05-28T13:40:46Z)
• Multimodal Representation Learning Techniques for Comprehensive Facial State Analysis [5.795431510723275]
  We present a comprehensive pipeline for multimodal facial state analysis.<n>We introduce a novel Multilevel Multimodal Face Foundation model (MF2) tailored for Action Unit (AU) and emotion recognition.<n>Experimentation show superior performance for AU and emotion detection tasks.
  arXiv  Detail & Related papers  (2025-04-14T16:00:57Z)
• Harmonizing Visual Representations for Unified Multimodal Understanding and Generation [53.01486796503091]
  We present emphHarmon, a unified autoregressive framework that harmonizes understanding and generation tasks with a shared MAR encoder.<n>Harmon achieves state-of-the-art image generation results on the GenEval, MJHQ30K and WISE benchmarks.
  arXiv  Detail & Related papers  (2025-03-27T20:50:38Z)
• Omni-ID: Holistic Identity Representation Designed for Generative Tasks [75.29174595706533]
  Omni-ID encodes holistic information about an individual's appearance across diverse expressions.<n>It consolidates information from a varied number of unstructured input images into a structured representation.<n>It demonstrates substantial improvements over conventional representations across various generative tasks.
  arXiv  Detail & Related papers  (2024-12-12T19:21:20Z)
• Infinite-ID: Identity-preserved Personalization via ID-semantics Decoupling Paradigm [31.06269858216316]
  We propose Infinite-ID, an ID-semantics decoupling paradigm for identity-preserved personalization. We introduce an identity-enhanced training, incorporating an additional image cross-attention module to capture sufficient ID information. We also introduce a feature interaction mechanism that combines a mixed attention module with an AdaIN-mean operation to seamlessly merge the two streams.
  arXiv  Detail & Related papers  (2024-03-18T13:39:53Z)
• Multimodal Face Synthesis from Visual Attributes [85.87796260802223]
  We propose a novel generative adversarial network that simultaneously synthesizes identity preserving multimodal face images. multimodal stretch-in modules are introduced in the discriminator which discriminates between real and fake images.
  arXiv  Detail & Related papers  (2021-04-09T13:47:23Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.