Harnessing Intrinsic Subject-Aware Attention for Controllable Multi-Subject Video Generation

• URL: http://arxiv.org/abs/2609.11507v1
• Date: Thu, 10 Sep 2026 13:15:03 GMT
• Title: Harnessing Intrinsic Subject-Aware Attention for Controllable Multi-Subject Video Generation
• Abstract summary: We find that certain attention blocks naturally form an Intrinsic Spatial Grounding Map (ISGM) that precisely locates reference subjects.<n>In low-noise stages, we use ISGM to guide the attention mechanism, allowing precise control over fidelity strength during inference without retraining.<n>In high-noise stages, we use these same maps to automatically build preference pairs at no additional cost for Reinforcement Learning.
• Score: 23.97734604758057
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Multi-subject video generation faces two key challenges: uncontrollable fidelity strength and potential semantic drift. We address these by analyzing the internal mechanisms of Diffusion Transformers (DiTs). We found that certain attention blocks naturally form an Intrinsic Spatial Grounding Map (ISGM) that precisely locates reference subjects. Building on this insight, we propose Dual-phase Intrinsic Attention Leveraging (DIAL), a framework that uses these internal signals for both training and inference. In low-noise stages, we use ISGM to guide the attention mechanism, allowing precise control over fidelity strength during inference without retraining. In high-noise stages, we use these same maps to automatically build preference pairs at no additional cost for Reinforcement Learning (RL). This RL procedure effectively anchors the model's attention to reference subjects and mitigates semantic drift. Extensive experiments show that DIAL significantly outperforms baseline models on the OpenS2V-Eval benchmark, consistently improving identity consistency and enabling controllable fidelity strength.

Related papers

• DuoAD: Leveraging [CLS] Dual Characteristics for Training-Free Few-Shot Anomaly Detection [16.74080857973598]
  We propose a fully automated anomaly detection framework leveraging global context to remove manual tunings.<n>Our method achieves Image-AUC scores of 97.7%, 93.2%, and 84.5% on MVTec-AD, VisA, and Real-IAD datasets.
  arXiv  Detail & Related papers  (2026-07-27T01:31:48Z)
• Dual Consensus: Escaping from Spurious Majority in Unsupervised RLVR via Two-Stage Vote Mechanism [12.006669100411466]
  Current RLVR approaches rely heavily on accurate pseudo-label estimation.<n>We propose Dual Consensus Reinforcement Learning, which generates more reliable learning signals through a two-stage consensus mechanism.<n>We show that DCRL consistently improves Pass@1 over majority vote while yielding more stable training dynamics.
  arXiv  Detail & Related papers  (2026-03-17T07:57:23Z)
• Dual-Channel Attention Guidance for Training-Free Image Editing Control in Diffusion Transformers [11.772150619675527]
  Existing attention manipulation methods focus exclusively on the Key space to modulate attention routing.<n>We propose Dual-Channel Attention Guidance (DCAG) to simultaneously manipulate both the Key channel and the Value channel.<n>DCAG consistently outperforms Key-only guidance across all fidelity metrics.
  arXiv  Detail & Related papers  (2026-02-20T06:24:20Z)
• OmniVL-Guard: Towards Unified Vision-Language Forgery Detection and Grounding via Balanced RL [63.388513841293616]
  Existing forgery detection methods fail to handle the interleaved text, images, and videos prevalent in real-world misinformation.<n>To bridge this gap, this paper targets to develop a unified framework for omnibus vision-language forgery detection and grounding.<n>We propose textbf OmniVL-Guard, a balanced reinforcement learning framework for omnibus vision-language forgery detection and grounding.
  arXiv  Detail & Related papers  (2026-02-11T09:41:36Z)
• Learning Visually Interpretable Oscillator Networks for Soft Continuum Robots from Video [4.857795247230421]
  We introduce Attention Broadcast Decoder (ABCD), a plug-and-play module for autoencoder-based latent dynamics learning.<n>ABCD generates pixel-accurate attention maps localizing each latent dimension's contribution while filtering static backgrounds.<n>We validate our approach on single- and double-segment SCRs, demonstrating that ABCD-based models significantly improve multi-step prediction accuracy.
  arXiv  Detail & Related papers  (2025-11-23T07:27:39Z)
• DeLeaker: Dynamic Inference-Time Reweighting For Semantic Leakage Mitigation in Text-to-Image Models [55.30555646945055]
  Text-to-Image (T2I) models are vulnerable to semantic leakage.<n>We introduce DeLeaker, a lightweight approach that mitigates leakage by directly intervening on the model's attention maps.<n>SLIM is the first dataset dedicated to semantic leakage.
  arXiv  Detail & Related papers  (2025-10-16T17:39:21Z)
• ScaleWeaver: Weaving Efficient Controllable T2I Generation with Multi-Scale Reference Attention [86.93601565563954]
  ScaleWeaver is a framework designed to achieve high-fidelity, controllable generation upon advanced visual autoregressive( VAR) models.<n>The proposed Reference Attention module discards the unnecessary attention from image$rightarrow$condition, reducing computational cost.<n>Experiments show that ScaleWeaver delivers high-quality generation and precise control while attaining superior efficiency over diffusion-based methods.
  arXiv  Detail & Related papers  (2025-10-16T17:00:59Z)
• SG-XDEAT: Sparsity-Guided Cross-Dimensional and Cross-Encoding Attention with Target-Aware Conditioning in Tabular Learning [0.0]
  We propose SG-XDEAT, a novel framework for supervised learning on tabular data.<n>At its core, SG-XDEAT employs a dual-stream encoder that decomposes each input feature into two parallel representations.<n>These dual representations are then propagated through a hierarchical stack of attention-based modules.
  arXiv  Detail & Related papers  (2025-10-14T15:56:40Z)
• Sycophancy Mitigation Through Reinforcement Learning with Uncertainty-Aware Adaptive Reasoning Trajectories [58.988535279557546]
  We introduce textbf sycophancy Mitigation through Adaptive Reasoning Trajectories.<n>We show that SMART significantly reduces sycophantic behavior while preserving strong performance on out-of-distribution inputs.
  arXiv  Detail & Related papers  (2025-09-20T17:09:14Z)
• ERIS: An Energy-Guided Feature Disentanglement Framework for Out-of-Distribution Time Series Classification [51.07970070817353]
  An ideal time series classification (TSC) should be able to capture invariant representations.<n>Current methods are largely unguided, lacking the semantic direction required to isolate truly universal features.<n>We propose an end-to-end Energy-Regularized Information for Shift-Robustness framework to enable guided and reliable feature disentanglement.
  arXiv  Detail & Related papers  (2025-08-19T12:13:41Z)
• Tracking the Unstable: Appearance-Guided Motion Modeling for Robust Multi-Object Tracking in UAV-Captured Videos [58.156141601478794]
  Multi-object tracking (UAVT) aims to track multiple objects while maintaining consistent identities across frames of a given video.<n>Existing methods typically model motion cues and appearance separately, overlooking their interplay and resulting in suboptimal tracking performance.<n>We propose AMOT, which exploits appearance and motion cues through two key components: an Appearance-Motion Consistency (AMC) matrix and a Motion-aware Track Continuation (MTC) module.
  arXiv  Detail & Related papers  (2025-08-03T12:06:47Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.