Semantic Routing: Exploring Multi-Layer LLM Feature Weighting for Diffusion Transformers

• URL: http://arxiv.org/abs/2602.03510v1
• Date: Tue, 03 Feb 2026 13:30:13 GMT
• Title: Semantic Routing: Exploring Multi-Layer LLM Feature Weighting for Diffusion Transformers
• Authors: Bozhou Li, Yushuo Guan, Haolin Li, Bohan Zeng, Yiyan Ji, Yue Ding, Pengfei Wan, Kun Gai, Yuanxing Zhang, Wentao Zhang,
• Abstract summary: We introduce a unified normalized convex fusion framework equipped with lightweight gates to systematically organize multi-layer LLM hidden states.<n>Experiments establish Depth-wise Semantic Routing as the superior conditioning strategy.<n>We find that purely time-wise fusion can paradoxically degrade visual generation fidelity.
• Score: 31.67315012315044
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Recent DiT-based text-to-image models increasingly adopt LLMs as text encoders, yet text conditioning remains largely static and often utilizes only a single LLM layer, despite pronounced semantic hierarchy across LLM layers and non-stationary denoising dynamics over both diffusion time and network depth. To better match the dynamic process of DiT generation and thereby enhance the diffusion model's generative capability, we introduce a unified normalized convex fusion framework equipped with lightweight gates to systematically organize multi-layer LLM hidden states via time-wise, depth-wise, and joint fusion. Experiments establish Depth-wise Semantic Routing as the superior conditioning strategy, consistently improving text-image alignment and compositional generation (e.g., +9.97 on the GenAI-Bench Counting task). Conversely, we find that purely time-wise fusion can paradoxically degrade visual generation fidelity. We attribute this to a train-inference trajectory mismatch: under classifier-free guidance, nominal timesteps fail to track the effective SNR, causing semantically mistimed feature injection during inference. Overall, our results position depth-wise routing as a strong and effective baseline and highlight the critical need for trajectory-aware signals to enable robust time-dependent conditioning.

Related papers

• AIVD: Adaptive Edge-Cloud Collaboration for Accurate and Efficient Industrial Visual Detection [15.419663374345845]
  This paper proposes the AIVD framework, which achieves unified precise localization and high-quality semantic generation.<n>To enhance the cloud MLLM's robustness against edge cropped-box noise and scenario variations, we design an efficient fine-tuning strategy.<n>To maintain high throughput and low latency across heterogeneous edge devices and dynamic network conditions, we propose a heterogeneous resource-aware dynamic scheduling algorithm.
  arXiv  Detail & Related papers  (2026-01-08T08:56:07Z)
• Towards Unified Semantic and Controllable Image Fusion: A Diffusion Transformer Approach [99.80480649258557]
  DiTFuse is an instruction-driven framework that performs semantics-aware fusion within a single model.<n>Experiments on public IVIF, MFF, and MEF benchmarks confirm superior quantitative and qualitative performance, sharper textures, and better semantic retention.
  arXiv  Detail & Related papers  (2025-12-08T05:04:54Z)
• LUMA: Low-Dimension Unified Motion Alignment with Dual-Path Anchoring for Text-to-Motion Diffusion Model [18.564067196226436]
  We propose a text-to-motion diffusion model that incorporates dual-path anchoring to enhance semantic alignment.<n>LUMA achieves state-of-the-art performance, with FID scores of 0.035 and 0.123, respectively.
  arXiv  Detail & Related papers  (2025-09-29T17:58:28Z)
• Unsupervised Online 3D Instance Segmentation with Synthetic Sequences and Dynamic Loss [52.28880405119483]
  Unsupervised online 3D instance segmentation is a fundamental yet challenging task.<n>Existing methods, such as UNIT, have made progress in this direction but remain constrained by limited training diversity.<n>We propose a new framework that enriches the training distribution through synthetic point cloud sequence generation.
  arXiv  Detail & Related papers  (2025-09-27T08:53:27Z)
• Integrating Time Series into LLMs via Multi-layer Steerable Embedding Fusion for Enhanced Forecasting [44.91360223102709]
  Time series (TS) data are ubiquitous across various application areas, rendering time series forecasting (TSF) a fundamental task.<n>Existing methods are inherently constrained by their shallow integration of TS information.<n>We propose the Multi-layer Steerable Embedding Fusion (MSEF) to mitigate the progressive loss of TS information in deeper layers.
  arXiv  Detail & Related papers  (2025-08-22T03:22:10Z)
• Fast-dLLM: Training-free Acceleration of Diffusion LLM by Enabling KV Cache and Parallel Decoding [51.711605076319216]
  Diffusion-based large language models (Diffusion LLMs) have shown promise for non-autoregressive text generation with parallel decoding capabilities.<n>We introduce a novel block-wise approximate KV Cache mechanism tailored for bidirectional diffusion models, enabling cache reuse with negligible performance drop.<n>We propose a confidence-aware parallel decoding strategy that selectively decodes tokens exceeding a confidence threshold, mitigating dependency violations and maintaining generation quality.
  arXiv  Detail & Related papers  (2025-05-28T17:39:15Z)
• Multimodal LLM-Guided Semantic Correction in Text-to-Image Diffusion [52.315729095824906]
  MLLM Semantic-Corrected Ping-Pong-Ahead Diffusion (PPAD) is a novel framework that introduces a Multimodal Large Language Model (MLLM) as a semantic observer during inference.<n>It performs real-time analysis on intermediate generations, identifies latent semantic inconsistencies, and translates feedback into controllable signals that actively guide the remaining denoising steps.<n>Extensive experiments demonstrate PPAD's significant improvements.
  arXiv  Detail & Related papers  (2025-05-26T14:42:35Z)
• TIDE : Temporal-Aware Sparse Autoencoders for Interpretable Diffusion Transformers in Image Generation [41.909091496502704]
  Diffusion Transformers (DiTs) are a powerful yet underexplored class of generative models.<n>We propose TIDE-Temporal-aware sparse autoencoders for Interpretable Diffusion transformErs framework.
  arXiv  Detail & Related papers  (2025-03-10T08:35:51Z)
• LF-Steering: Latent Feature Activation Steering for Enhancing Semantic Consistency in Large Language Models [16.37602070339033]
  Large Language Models (LLMs) often generate inconsistent responses when prompted with semantically equivalent paraphrased inputs.<n>We propose LF-Steering, a novel activation steering approach to precisely identify latent feature representations responsible for semantic inconsistency.<n>Our method maps the hidden states of the relevant transformer layer into a sparsely activated, high-dimensional feature space based on a sparse autoencoder.
  arXiv  Detail & Related papers  (2025-01-19T13:06:51Z)
• Multi-scale Transformer Network with Edge-aware Pre-training for Cross-Modality MR Image Synthesis [52.41439725865149]
  Cross-modality magnetic resonance (MR) image synthesis can be used to generate missing modalities from given ones. Existing (supervised learning) methods often require a large number of paired multi-modal data to train an effective synthesis model. We propose a Multi-scale Transformer Network (MT-Net) with edge-aware pre-training for cross-modality MR image synthesis.
  arXiv  Detail & Related papers  (2022-12-02T11:40:40Z)
• Towards Energy-Efficient, Low-Latency and Accurate Spiking LSTMs [1.7969777786551424]
  Spiking Neural Networks (SNNs) have emerged as an attractive-temporal computing paradigm vision for complex tasks. We propose an optimized spiking long short-term memory networks (LSTM) training framework that involves a novel. rev-to-SNN conversion framework, followed by SNN training. We evaluate our framework on sequential learning tasks including temporal M, Google Speech Commands (GSC) datasets, and UCI Smartphone on different LSTM architectures.
  arXiv  Detail & Related papers  (2022-10-23T04:10:27Z)
• Optical-Flow-Reuse-Based Bidirectional Recurrent Network for Space-Time Video Super-Resolution [52.899234731501075]
  Space-time video super-resolution (ST-VSR) simultaneously increases the spatial resolution and frame rate for a given video. Existing methods typically suffer from difficulties in how to efficiently leverage information from a large range of neighboring frames. We propose a coarse-to-fine bidirectional recurrent neural network instead of using ConvLSTM to leverage knowledge between adjacent frames.
  arXiv  Detail & Related papers  (2021-10-13T15:21:30Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.