Semantics Lead the Way: Harmonizing Semantic and Texture Modeling with Asynchronous Latent Diffusion

• URL: http://arxiv.org/abs/2512.04926v2
• Date: Fri, 05 Dec 2025 07:41:30 GMT
• Title: Semantics Lead the Way: Harmonizing Semantic and Texture Modeling with Asynchronous Latent Diffusion
• Abstract summary: Latent Diffusion Models (LDMs) inherently follow a coarse-to-fine generation process.<n>Recent advances have integrated semantic priors from pretrained visual encoders to further enhance LDMs.<n>We propose Semantic-First Diffusion (SFD), a latent diffusion paradigm that explicitly prioritizes semantic formation.
• Score: 59.30874672131483
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Latent Diffusion Models (LDMs) inherently follow a coarse-to-fine generation process, where high-level semantic structure is generated slightly earlier than fine-grained texture. This indicates the preceding semantics potentially benefit texture generation by providing a semantic anchor. Recent advances have integrated semantic priors from pretrained visual encoders to further enhance LDMs, yet they still denoise semantic and VAE-encoded texture synchronously, neglecting such ordering. Observing these, we propose Semantic-First Diffusion (SFD), a latent diffusion paradigm that explicitly prioritizes semantic formation. SFD first constructs composite latents by combining a compact semantic latent, which is extracted from a pretrained visual encoder via a dedicated Semantic VAE, with the texture latent. The core of SFD is to denoise the semantic and texture latents asynchronously using separate noise schedules: semantics precede textures by a temporal offset, providing clearer high-level guidance for texture refinement and enabling natural coarse-to-fine generation. On ImageNet 256x256 with guidance, SFD achieves FID 1.06 (LightningDiT-XL) and FID 1.04 (1.0B LightningDiT-XXL), while achieving up to 100x faster convergence than the original DiT. SFD also improves existing methods like ReDi and VA-VAE, demonstrating the effectiveness of asynchronous, semantics-led modeling. Project page and code: https://yuemingpan.github.io/SFD.github.io/.

Related papers

• Beyond Text Conditioning: A Systematic Study of MLLM-DiT Fusion for Video Generation [50.92925099642139]
  Diffusion Transformers (DiTs) have become the dominant paradigm for high-fidelity video generation, yet their ability to perform high-level semantic planning remains limited.<n>We propose BiVidGen, a hybrid framework where an MLLM first generates semantic visual tokens and a DiT renders videos conditioned on both text and these tokens via multi-layer cross-attention.
  arXiv  Detail & Related papers  (2026-08-14T07:40:41Z)
• Imagine Before You Draw: Visual Prompt Engineering for Image Generation [67.81347924426714]
  We propose Visual Prompt Engineering (VPE), which can be seamlessly integrated into internal frameworks.<n>We validate VPE across class-conditional generation, text-to-image generation, and image editing.<n>Results show that VPE can accelerate convergence, raise quality ceilings, and through internal integration, achieve substantially better editing preservation.
  arXiv  Detail & Related papers  (2026-06-03T05:01:36Z)
• TextLDM: Language Modeling with Continuous Latent Diffusion [89.69255520673248]
  Diffusion Transformers (DiT) trained with flow matching in a VAE latent space have unified visual generation across images and videos.<n>We propose TextLDM, which transfers the visual latent diffusion recipe to text generation with minimal architectural modification.
  arXiv  Detail & Related papers  (2026-05-08T13:54:34Z)
• LoST: Level of Semantics Tokenization for 3D Shapes [50.847769883816085]
  State-of-the-art methods rely on geometric level-of-detail (LoD) hierarchies, originally designed for rendering and compression.<n>We propose Level-of-Semantics Tokenization (LoST), which orders salience, such that early prefixes decode complete, plausible shapes.<n>LoST achieves SOTA reconstruction, surpassing previous LoD-based 3D shape tokenizers by large margins on both geometric and semantic reconstruction metrics.
  arXiv  Detail & Related papers  (2026-03-18T17:56:06Z)
• EVLF: Early Vision-Language Fusion for Generative Dataset Distillation [6.913041518668593]
  We introduce an Early Vision-Language Fusion (EVLF) method that aligns textual and visual embeddings at the transition between the encoder and the generative backbone.<n>EVLF is plug-and-play and can be easily integrated into any diffusion-based dataset distillation pipeline with an encoder.<n>Experiments demonstrate that EVLF generates semantically faithful and visually coherent synthetic data, yielding consistent improvements in downstream classification accuracy across varied settings.
  arXiv  Detail & Related papers  (2026-03-08T05:34:07Z)
• Hyperdimensional Cross-Modal Alignment of Frozen Language and Image Models for Efficient Image Captioning [0.8594140167290097]
  HDFLIM is a framework that establishes cross-modal mappings while keeping pretrained vision and language models fully frozen.<n>We show that HDFLIM performance comparable to end-to-end vision-language training methods.
  arXiv  Detail & Related papers  (2026-02-27T01:38:58Z)
• Bridging the Discrete-Continuous Gap: Unified Multimodal Generation via Coupled Manifold Discrete Absorbing Diffusion [60.186310080523135]
  Bifurcation of generative modeling into autoregressive approaches for discrete data (text) and diffusion approaches for continuous data (images) hinders development of truly unified multimodal systems.<n>We propose textbfCoM-DAD, a novel probabilistic framework that reformulates multimodal generation as a hierarchical dual-process.<n>Our method demonstrates superior stability over standard masked modeling, establishing a new paradigm for scalable, unified text-image generation.
  arXiv  Detail & Related papers  (2026-01-07T16:21:19Z)
• ResTok: Learning Hierarchical Residuals in 1D Visual Tokenizers for Autoregressive Image Generation [64.84095852784714]
  Residual Tokenizer (ResTok) is a 1D visual tokenizer that builds hierarchical residuals for both image tokens and latent tokens.<n>We show that restoring hierarchical residual priors in visual tokenization significantly improves AR image generation, achieving a gFID of 2.34 on ImageNet-256 with only 9 sampling steps.
  arXiv  Detail & Related papers  (2026-01-07T14:09:18Z)
• EDITS: Enhancing Dataset Distillation with Implicit Textual Semantics [12.818622596576775]
  EDITS is a novel framework that exploits the implicit textual semantics within the image data to achieve enhanced distillation.<n>In this paper, we propose EDITS, a novel framework that exploits the implicit textual semantics within the image data to achieve enhanced distillation.
  arXiv  Detail & Related papers  (2025-09-17T09:48:39Z)
• Noise Diffusion for Enhancing Semantic Faithfulness in Text-to-Image Synthesis [19.109259539368]
  This paper proposes leveraging the language comprehension capabilities of large vision-language models to guide the optimization of the initial noisy latent.<n>We introduce the Noise Diffusion process, which updates the noisy latent to generate semantically faithful images while preserving distribution consistency.<n> Experimental results demonstrate the effectiveness and adaptability of our framework, consistently enhancing semantic alignment across various diffusion models.
  arXiv  Detail & Related papers  (2024-11-25T15:40:47Z)
• IIDM: Image-to-Image Diffusion Model for Semantic Image Synthesis [8.080248399002663]
  In this paper, semantic image synthesis is treated as an image denoising task. The style reference is first contaminated with random noise and then progressively denoised by IIDM. Three techniques, refinement, color-transfer and model ensembles are proposed to further boost the generation quality.
  arXiv  Detail & Related papers  (2024-03-20T08:21:00Z)
• Improving Diffusion-Based Image Synthesis with Context Prediction [49.186366441954846]
  Existing diffusion models mainly try to reconstruct input image from a corrupted one with a pixel-wise or feature-wise constraint along spatial axes. We propose ConPreDiff to improve diffusion-based image synthesis with context prediction. Our ConPreDiff consistently outperforms previous methods and achieves a new SOTA text-to-image generation results on MS-COCO, with a zero-shot FID score of 6.21.
  arXiv  Detail & Related papers  (2024-01-04T01:10:56Z)
• Semantic-Conditional Diffusion Networks for Image Captioning [116.86677915812508]
  We propose a new diffusion model based paradigm tailored for image captioning, namely Semantic-Conditional Diffusion Networks (SCD-Net) In SCD-Net, multiple Diffusion Transformer structures are stacked to progressively strengthen the output sentence with better visional-language alignment and linguistical coherence. Experiments on COCO dataset demonstrate the promising potential of using diffusion models in the challenging image captioning task.
  arXiv  Detail & Related papers  (2022-12-06T16:08:16Z)
• SLLEN: Semantic-aware Low-light Image Enhancement Network [92.80325772199876]
  We develop a semantic-aware LLE network (SSLEN) composed of a LLE main-network (LLEmN) and a SS auxiliary-network (SSaN) Unlike currently available approaches, the proposed SLLEN is able to fully lever the semantic information, e.g., IEF, HSF, and SS dataset, to assist LLE. Comparisons between the proposed SLLEN and other state-of-the-art techniques demonstrate the superiority of SLLEN with respect to LLE quality.
  arXiv  Detail & Related papers  (2022-11-21T15:29:38Z)
• Semantic Image Synthesis via Diffusion Models [174.24523061460704]
  Denoising Diffusion Probabilistic Models (DDPMs) have achieved remarkable success in various image generation tasks.<n>Recent work on semantic image synthesis mainly follows the de facto GAN-based approaches.<n>We propose a novel framework based on DDPM for semantic image synthesis.
  arXiv  Detail & Related papers  (2022-06-30T18:31:51Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.