StreamHOI: Interaction-aware Temporal Memory Adaptation for Streaming HOI Video Generation

• URL: http://arxiv.org/abs/2607.20174v1
• Date: Wed, 22 Jul 2026 14:06:39 GMT
• Title: StreamHOI: Interaction-aware Temporal Memory Adaptation for Streaming HOI Video Generation
• Abstract summary: We present emphStreamHOI, a low-latency streaming framework for long-duration HOI video generation.<n>StreamHOI achieves strong interaction plausibility, object fidelity, human quality and efficiency, reaching 17.6 FPS with 0.75s first-chunk latency.
• Score: 31.081864027419016
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Existing human--object interaction (HOI) video generation methods are largely limited to offline short-video generation with complex driving conditions, making them unsuitable for real-time interactive applications. We present \emph{StreamHOI}, a low-latency streaming framework for long-duration HOI video generation. Instead of converting heavily conditioned HOI pipelines into streaming systems, we study how an image-to-video streaming generator should organize historical memory to preserve interactions under bounded latency. We find that the standard sink-local memory design faces a trade-off in streaming HOI generation, and different transformer blocks show different historical-memory preferences for HOI regions and surrounding regions. To match memory composition with block behavior, StreamHOI performs offline HOI-aware block profiling and applies bias-guided memory-specialized training to adapt the generator to block-specific memory layouts. We further introduce a memory distance scaling module to strengthen long-range access to early interaction states. Extensive comparisons with both long-video baselines and recent HOI generation methods demonstrate that StreamHOI achieves strong interaction plausibility, object fidelity, human quality and efficiency, reaching 17.6 FPS with 0.75s first-chunk latency.

Related papers

• Beyond Retrieval: Progressive Latent Memory Evolution for Streaming Video Understanding [64.34948094316685]
  We introduce LatentStream, a progressive latent working memory framework that shifts streaming memory from store-and-retrieve to retrieve-and-internalize.<n>LatentStream achieves new state-of-the-art results on existing online and offline video benchmarks.
  arXiv  Detail & Related papers  (2026-09-03T17:28:14Z)
• Ripple: Real-Time Streaming Audio-Video Generation With Cross-Modal Recurrent Memory [20.92714383117306]
  Ripple is a real-time joint audio-video generation system with a cross-modal recurrent memory mechanism.<n>It achieves 28 FPS at 480P resolution, over faster than the teacher, while capable of coherent long-form generation.
  arXiv  Detail & Related papers  (2026-07-29T12:13:01Z)
• LiveStarPro: Proactive Streaming Video Understanding with Hierarchical Memory for Long-Horizon Streams [59.485485426790966]
  We introduce LiveStarPro, a live streaming assistant that is designed for proactive video understanding over long-horizon streams.<n>LiveStarPro consistently surpasses existing methods, attaining a 28.9% improvement in semantic correctness.<n>Its streaming key-value cache further yields a 1.58x inference speedup over the same model without caching.
  arXiv  Detail & Related papers  (2026-06-16T11:18:05Z)
• SlotMemory: Object-Centric KV Memory for Streaming Long-Video Generation [30.157534925157844]
  SlotMemory is an object-centric Key-Value memory mechanism for streaming video diffusion.<n>Our approach shifts the memory abstraction from "when" an event occurred to "what" is being represented by decomposing the transformer's key-value manifold.
  arXiv  Detail & Related papers  (2026-05-29T09:04:26Z)
• Attend Locally, Remember Linearly: Linear Attention as Cross-Frame Memory for Autoregressive Video Diffusion [61.57938553036056]
  ARL2 is a hybrid attention module that replaces quadratic cross-frame attention with a fixed-size recurrent state.<n>We decompose self-attention into two branches: an intra-frame softmax branch for spatial detail and local dependencies, and an inter-frame recurrent linear branch that maintains a fixed-size state for streaming context.<n>With 75% of layers replaced by hybrid linear attention, the model achieves up to 2.26 wall-clock speedup and 54% memory reduction, while maintaining comparable quality with improving temporal consistency.
  arXiv  Detail & Related papers  (2026-05-15T19:33:45Z)
• SWIFT: Prompt-Adaptive Memory for Efficient Interactive Long Video Generation [31.105007908298003]
  Streaming long-video generation faces a central challenge in continuous semantic switching.<n>Current approaches rely on cache rebuilding at prompt boundaries or fixed memory budgets.<n>We present SWIFT, a training-free framework for multi-prompt long-video generation.
  arXiv  Detail & Related papers  (2026-05-10T09:37:56Z)
• Long-Horizon Streaming Video Generation via Hybrid Attention with Decoupled Distillation [16.611062315689306]
  We propose Hybrid Forcing, which jointly optimize temporal information retention and computational efficiency.<n>Our model achieves real-time, unbounded 832x480 video generation at 29.5 FPS on a single NVIDIA H100 GPU without quantization or model compression.
  arXiv  Detail & Related papers  (2026-04-11T08:54:07Z)
• MemFlow: Flowing Adaptive Memory for Consistent and Efficient Long Video Narratives [54.07515675393396]
  Existing solutions maintain the memory by compressing historical frames with predefined strategies.<n>We propose MemFlow to address this problem.<n>MemFlow achieves outstanding long-context consistency with negligible burden.
  arXiv  Detail & Related papers  (2025-12-16T18:59:59Z)
• VideoSSM: Autoregressive Long Video Generation with Hybrid State-Space Memory [42.2374676860638]
  Autoregressive (AR) diffusion enables streaming, interactive long-video generation by producing frames causally.<n>Maintaining coherence over minute-scale horizons remains challenging due to accumulated errors, motion drift, and content repetition.<n>We propose VideoSSM, a Long Video Model that unifies AR diffusion with a hybrid state-space memory.
  arXiv  Detail & Related papers  (2025-12-04T07:06:02Z)
• RELIC: Interactive Video World Model with Long-Horizon Memory [74.81433479334821]
  A truly interactive world model requires real-time long-horizon streaming, consistent spatial memory, and precise user control.<n>We present RELIC, a unified framework that tackles these three challenges altogether.<n>Given a single image and a text description, RELIC enables memory-aware, long-duration exploration of arbitrary scenes in real time.
  arXiv  Detail & Related papers  (2025-12-03T18:29:20Z)
• Mixture of Contexts for Long Video Generation [72.96361488755986]
  We recast long-context video generation as an internal information retrieval task.<n>We propose a simple, learnable sparse attention routing module, Mixture of Contexts (MoC), as an effective long-term memory retrieval engine.<n>As we scale the data and gradually sparsify the routing, the model allocates compute to salient history, preserving identities, actions, and scenes over minutes of content.
  arXiv  Detail & Related papers  (2025-08-28T17:57:55Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.