RealDiffusion: Physics-informed Attention for Multi-character Storybook Generation

• URL: http://arxiv.org/abs/2605.11927v1
• Date: Tue, 12 May 2026 10:39:45 GMT
• Title: RealDiffusion: Physics-informed Attention for Multi-character Storybook Generation
• Abstract summary: RealDiffusion is a framework designed to reconcile narrative dynamism with narrative coherence.<n>Heat diffusion serves as a dissipative prior that averages neighboring features along the sequence.<n>A region-aware process removes perturbations that explore nearby modes and prevent collapse so the story maintains pose change and scene evolution.
• Score: 31.320456668679554
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: While modern diffusion models excel at generating diverse single images, extending this to sequential generation reveals a fundamental challenge: balancing narrative dynamism with multi-character coherence. Existing methods often falter at this trade-off, leading to artifacts where characters lose their identity or the story stagnates. To resolve this critical tension, we introduce RealDiffusion, a unified framework designed to reconcile robust coherence with narrative dynamism. Heat diffusion serves as a dissipative prior that averages neighboring features along the sequence and removes high-frequency noise within the subject region. This suppresses attribute drift and stabilizes identity across frames. A region-aware stochastic process then introduces small perturbations that explore nearby modes and prevent collapse so the story maintains pose change and scene evolution. We thus introduce a lightweight, training-free Physics-informed Attention mechanism that injects controllable physical priors into the self-attention layers during inference. By modeling feature evolution as a configurable physical system, our method regularizes spatio-temporal relationships without suppressing intentional, prompt-driven changes. Extensive experiments demonstrate that RealDiffusion achieves substantial gains in character coherence while preserving narrative dynamism, outperforming state-of-the-art approaches. Code is available at https://github.com/ShmilyQi-CN/RealDiffusion.

Related papers

• DSAR: Dual-Stream Autoregressive Modeling of Temporal Cloth Dynamics for Photorealistic Animatable Avatars [7.198815179389743]
  We introduce a dual-stream autoregressive framework that explicitly models both observable geometric information and implicit internal state.<n> Experiments on challenging datasets demonstrate significant improvements in rendering quality, temporal consistency, and generalization to motion patterns beyond training distributions.
  arXiv  Detail & Related papers  (2026-08-11T05:19:21Z)
• UniNDM: A Unified Noise-driven Detection and Mitigation Framework Against Sexual Content in Text-to-Image Generation [90.0896070825457]
  We propose a unified noise-driven framework that rethinks safety mechanisms through the lens of noise dynamics in diffusion processes.<n>Our key insight is that early-stage predicted noise exhibits inherent separability between normal and sexually explicit content.<n>We develop a lightweight noise-based detector achieving superior accuracy with virtually no computational overhead.
  arXiv  Detail & Related papers  (2026-07-18T14:03:39Z)
• ConsistTalk: Intensity Controllable Temporally Consistent Talking Head Generation with Diffusion Noise Search [8.993664585683055]
  We introduce textbfTalk, a novel intensity-controllable talking head generation framework with diffusion noise search inference.<n>First, we propose textbfan optical flow-guided temporal module (OFT) that decouples motion features from static appearance.<n>Second, we present an textbfAudio-to-Intensity (A2I) model obtained through multimodal teacher-student knowledge distillation.
  arXiv  Detail & Related papers  (2025-11-10T08:28:13Z)
• Boosting Fidelity for Pre-Trained-Diffusion-Based Low-Light Image Enhancement via Condition Refinement [63.54516423266521]
  Pre-Trained Diffusion-Based (PTDB) methods often sacrifice content fidelity to attain higher perceptual realism.<n>We propose a novel optimization strategy for conditioning in pre-trained diffusion models, enhancing fidelity while preserving realism and aesthetics.<n>Our approach is plug-and-play, seamlessly integrating into existing diffusion networks to provide more effective control.
  arXiv  Detail & Related papers  (2025-10-20T02:40:06Z)
• RainDiff: End-to-end Precipitation Nowcasting Via Token-wise Attention Diffusion [64.49056527678606]
  We propose a Token-wise Attention integrated into not only the U-Net diffusion model but also the radar-temporal encoder.<n>Unlike prior approaches, our method integrates attention into the architecture without incurring the high resource cost typical of pixel-space diffusion.<n>Our experiments and evaluations demonstrate that the proposed method significantly outperforms state-of-the-art approaches, robustness local fidelity, generalization, and superior in complex precipitation forecasting scenarios.
  arXiv  Detail & Related papers  (2025-10-16T17:59:13Z)
• Equilibrium flow: From Snapshots to Dynamics [4.741100658955037]
  We introduce the Equilibrium flow method, a framework that learns continuous dynamics that preserve a given pattern distribution.<n>For high-dimensional Turing patterns from the Gray-Scott model, we develop an efficient, training-free variant that achieves high fidelity to the ground truth.<n>This capability extends beyond recovering known systems, enabling a new paradigm of inverse design for Artificial Life.
  arXiv  Detail & Related papers  (2025-09-22T16:33:20Z)
• Stable Video-Driven Portraits [52.008400639227034]
  Animation aims to generate photo-realistic videos from a single source image by reenacting the expression and pose from a driving video.<n>Recent advances using diffusion models have demonstrated improved quality but remain constrained by weak control signals and architectural limitations.<n>We propose a novel diffusion based framework that leverages masked facial regions specifically the eyes, nose, and mouth from the driving video as strong motion control cues.
  arXiv  Detail & Related papers  (2025-09-22T08:11:08Z)
• StorySync: Training-Free Subject Consistency in Text-to-Image Generation via Region Harmonization [31.250596607318364]
  Existing approaches, which typically rely on fine-tuning or retraining models, are computationally expensive, time-consuming, and often interfere with the model's pre-existing capabilities.<n>This paper proposes an efficient consistent-subject-generation method.<n> Experimental results demonstrate that our approach successfully generates visually consistent subjects across a variety of scenarios.
  arXiv  Detail & Related papers  (2025-07-31T11:24:40Z)
• Object Isolated Attention for Consistent Story Visualization [16.721634474902036]
  Open-ended story visualization is a challenging task that involves generating coherent image sequences from a given storyline.<n>One of the main difficulties is maintaining character consistency while creating natural and contextually fitting scenes.<n>We propose an enhanced Transformer module that uses separate self attention and cross attention mechanisms.
  arXiv  Detail & Related papers  (2025-03-30T08:16:52Z)
• Dynamical Diffusion: Learning Temporal Dynamics with Diffusion Models [71.63194926457119]
  We introduce Dynamical Diffusion (DyDiff), a theoretically sound framework that incorporates temporally aware forward and reverse processes.<n>Experiments across scientifictemporal forecasting, video prediction, and time series forecasting demonstrate that Dynamical Diffusion consistently improves performance in temporal predictive tasks.
  arXiv  Detail & Related papers  (2025-03-02T16:10:32Z)
• ACDiT: Interpolating Autoregressive Conditional Modeling and Diffusion Transformer [95.80384464922147]
  ACDiT is a blockwise Conditional Diffusion Transformer.<n>It offers a flexible between token-wise autoregression and full-sequence diffusion.<n>We show that ACDiT performs best among all autoregressive baselines on image and video generation tasks.
  arXiv  Detail & Related papers  (2024-12-10T18:13:20Z)
• ModeRNN: Harnessing Spatiotemporal Mode Collapse in Unsupervised Predictive Learning [75.2748374360642]
  We propose ModeRNN, which introduces a novel method to learn hidden structured representations between recurrent states. Across the entire dataset, different modes result in different responses on the mixtures of slots, which enhances the ability of ModeRNN to build structured representations.
  arXiv  Detail & Related papers  (2021-10-08T03:47:54Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.