MegaSlide-DiT: Memory-Centric Adaptation and Deformable Local Attention for Efficient Video Diffusion

• URL: http://arxiv.org/abs/2607.22696v1
• Date: Fri, 17 Jul 2026 19:30:05 GMT
• Title: MegaSlide-DiT: Memory-Centric Adaptation and Deformable Local Attention for Efficient Video Diffusion
• Abstract summary: MegaSlide-DiT is a prototype that demonstrates how a pre-trained 105B DiT can be adapted on a single H200 GPU with 1.5 TB of host RAM.<n>Our own insight is that not all persistent weights, master weights and moments remain in host memory, while only transient shards are streamed to the GPU on demand.
• Score: 3.7821833028870038
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: High-resolution video diffusion models built on Diffusion Transformers (DiTs) deliver strong fidelity but quickly exhaust the memory budget of a single workstation. A 100 billion-plus parameter DiT easily requires over a terabyte of persistent state, while naive spatiotemporal self-attention grows quadratically in sequence length. These two walls -- parameter memory and activation memory -- prevent researchers from adapting massive generative models without large GPU clusters. We revisit this problem from a systems perspective and introduce MegaSlide-DiT, a prototype that demonstrates how a pre-trained 105B DiT can be adapted on a single H200 GPU with 1.5 TB of host RAM. Our key insight is that the GPU need not own the model state: all persistent weights, master weights and optimizer moments remain in host memory, while only transient shards are streamed to the GPU on demand. Simultaneously, we replace quadratic global attention with 3D Deformable Slide Attention (3D-DSA), a motion-adaptive local attention operator that reduces both memory and computational complexity to linear in the sequence length. We report detailed memory accounting, execution traces and evaluation results to substantiate our design. MegaSlide-DiT does not claim to train a 105B model from scratch on a single GPU, nor does it magically solve bandwidth limits; rather, it offers a pragmatic path for full-parameter adaptation of massive video diffusion models on high-end workstations.

Related papers

• Mixture-of-Parallelisms: Towards Memory-Efficient Training Stack for Mixture-of-Experts Models [67.17268530365189]
  This paper showcases a memory-efficient training stack for Mixture-of-Experts (MoE) models.<n>It combines and specializes various existing and novel parallelism techniques at different layers and stages of the MoE model training pipeline.
  arXiv  Detail & Related papers  (2026-07-02T08:06:57Z)
• OOM-Free Alpamayo via CPU-GPU Memory Swapping for Vision-Language-Action Models [1.1011268090482575]
  Vision-Language-Action (VLA) models for autonomous driving unify perception, reasoning, and control in a single neural network.<n>We present a framework, which enables memory-efficient VLA inference on VRAM-constrained GPU.<n>Our work achieves up to 3.55x speedup over Accelerate offloading while maintaining full BF16 precision.
  arXiv  Detail & Related papers  (2026-05-12T07:37:10Z)
• FlashMem: Supporting Modern DNN Workloads on Mobile with GPU Memory Hierarchy Optimizations [10.92493656178845]
  FlashMem is a memory streaming framework designed to efficiently execute large-scale modern deep neural networks and multi-DNN workloads.<n>We show that FlashMem achieves 2.0x to 8.4x memory reduction and 1.7x to 75.0x speedup compared to existing frameworks.<n>Results on 11 models demonstrate that FlashMem achieves 2.0x to 8.4x memory reduction and 1.7x to 75.0x speedup compared to existing frameworks.
  arXiv  Detail & Related papers  (2026-02-17T06:23:40Z)
• Horizon-LM: A RAM-Centric Architecture for LLM Training [26.927410607740025]
  Horizon-LM is a memory-centric training system that redefines the roles of CPU and GPU for large-model optimization.<n>On a single H200 GPU with 1.5,TB host RAM, Horizon-LM reliably trains models up to 120B parameters.<n>On a standard single A100 machine, Horizon-LM achieves up to 12.2$times$ higher training throughput than DeepSpeed ZeRO-3 with CPU offloading.
  arXiv  Detail & Related papers  (2026-02-04T18:04:46Z)
• 70% Size, 100% Accuracy: Lossless LLM Compression for Efficient GPU Inference via Dynamic-Length Float [52.079202872069835]
  Large-scale AI models, such as Large Language Models (LLMs) and Diffusion Models (DMs) have grown rapidly in size.<n>We introduce Dynamic-Length Float (DFloat11), a compression framework that reduces LLM and DM size by 30% while preserving outputs that are bit-for-bit identical to the original model.
  arXiv  Detail & Related papers  (2025-04-15T22:38:38Z)
• Look Every Frame All at Once: Video-Ma$^2$mba for Efficient Long-form Video Understanding with Multi-Axis Gradient Checkpointing [52.050036778325094]
  Video-Ma$2$mba is a novel architecture that incorporates State Space Models (SSMs) within the Mamba-2 framework.<n>Our approach significantly reduces the memory footprint compared to standard gradient checkpointing.<n>By maintaining a detailed capture of temporal dynamics, our model improves the accuracy and relevance of responses in long video understanding tasks.
  arXiv  Detail & Related papers  (2024-11-29T04:12:13Z)
• LiVOS: Light Video Object Segmentation with Gated Linear Matching [116.58237547253935]
  LiVOS is a lightweight memory network that employs linear matching via linear attention. For longer and higher-resolution videos, it matched STM-based methods with 53% less GPU memory and supports 4096p inference on a 32G consumer-grade GPU.
  arXiv  Detail & Related papers  (2024-11-05T05:36:17Z)
• Adaptive Caching for Faster Video Generation with Diffusion Transformers [52.73348147077075]
  Diffusion Transformers (DiTs) rely on larger models and heavier attention mechanisms, resulting in slower inference speeds. We introduce a training-free method to accelerate video DiTs, termed Adaptive Caching (AdaCache) We also introduce a Motion Regularization (MoReg) scheme to utilize video information within AdaCache, controlling the compute allocation based on motion content.
  arXiv  Detail & Related papers  (2024-11-04T18:59:44Z)
• AI and Memory Wall [81.06494558184049]
  We show how memory bandwidth can become the dominant bottleneck for decoder models. We argue for a redesign in model architecture, training, and deployment strategies to overcome this memory limitation.
  arXiv  Detail & Related papers  (2024-03-21T04:31:59Z)
• DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scale [20.558091867632445]
  DeepSpeed Inference is a comprehensive system solution for transformer model inference. It reduces latency by up to 7.3X over the state-of-the-art for latency-oriented scenarios and increases throughput by over 1.5x for throughput-oriented scenarios. It can inference 25x larger models than with GPU-only solutions, while delivering a high throughput of 84 TFLOPS (over $50%$ of A6000 peak)
  arXiv  Detail & Related papers  (2022-06-30T18:01:08Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.