Context Forcing: Consistent Autoregressive Video Generation with Long Context

• URL: http://arxiv.org/abs/2602.06028v1
• Date: Thu, 05 Feb 2026 18:58:01 GMT
• Title: Context Forcing: Consistent Autoregressive Video Generation with Long Context
• Abstract summary: We propose textbfContext Forcing, a framework that trains a long-context student via a long-context teacher.<n>By ensuring the teacher is aware of the full generation history, we eliminate the supervision mismatch.<n>Our method enables effective context lengths exceeding 20 seconds -- 2 to 10 times longer than state-of-the-art methods.
• Score: 74.08361435348873
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Recent approaches to real-time long video generation typically employ streaming tuning strategies, attempting to train a long-context student using a short-context (memoryless) teacher. In these frameworks, the student performs long rollouts but receives supervision from a teacher limited to short 5-second windows. This structural discrepancy creates a critical \textbf{student-teacher mismatch}: the teacher's inability to access long-term history prevents it from guiding the student on global temporal dependencies, effectively capping the student's context length. To resolve this, we propose \textbf{Context Forcing}, a novel framework that trains a long-context student via a long-context teacher. By ensuring the teacher is aware of the full generation history, we eliminate the supervision mismatch, enabling the robust training of models capable of long-term consistency. To make this computationally feasible for extreme durations (e.g., 2 minutes), we introduce a context management system that transforms the linearly growing context into a \textbf{Slow-Fast Memory} architecture, significantly reducing visual redundancy. Extensive results demonstrate that our method enables effective context lengths exceeding 20 seconds -- 2 to 10 times longer than state-of-the-art methods like LongLive and Infinite-RoPE. By leveraging this extended context, Context Forcing preserves superior consistency across long durations, surpassing state-of-the-art baselines on various long video evaluation metrics.

Related papers

• Ring Forcing: Towards Precise Long-Term Memory for Autoregressive Video Diffusion [30.13108213701987]
  Ring Forcing is an autoregressive video diffusion framework designed to robustly construct and precisely utilize long-term memory.<n>Our ring-structured training strategy enforces retrieval from distant history, effectively reconciling the trade-off between strict historical adherence and generative diversity.<n>We present a sparse RoPE mechanism to enable flexible, scalable memory adaptation while fully exploiting pre-trained priors.
  arXiv  Detail & Related papers  (2026-08-27T08:25:32Z)
• IS-CoT: Breaking the Long-form Generation Collapse via Interleaved Structural Thinking [50.381430711141995]
  We introduce the Interleaved Structural Chain-of-Thought (IS-CoT) framework.<n>IS-CoT embeds a dynamic Plan-Write-Reflect cycle into the generation process.<n>We construct a high-quality dataset of interleaved reasoning traces via a multi-teacher pipeline and train IS-Writer-8B.
  arXiv  Detail & Related papers  (2026-06-08T16:31:00Z)
• An Efficient Long-Context Ranking Architecture With Calibrated LLM Distillation: Application to Person-Job Fit [0.6117371161379207]
  We propose a re-ranking model based on a new generation of late cross-attention architecture.<n>To mitigate historical data biases, we use a generative large language model (LLM) as a teacher.<n>The resulting model produces skill-fit scores that enable consistent and interpretable person-job matching.
  arXiv  Detail & Related papers  (2026-01-15T12:01:27Z)
• Beyond Length: Quantifying Long-Range Information for Long-Context LLM Pretraining Data [67.46386646195818]
  We introduce LongFilter, a framework for curating training data tailored to long-context pretraining.<n>LongFilter measures the information gain provided by extended context by contrasting model predictions under long-context versus short-context settings.<n>Experiments with LLaMA-3-8B, extending its context length from 8K to 64K, show that LongFilter efficiently selects high-quality data and yields substantial improvements on benchmarks such as HELMET, LongBench, and RULER.
  arXiv  Detail & Related papers  (2025-10-29T06:21:08Z)
• Self-Forcing++: Towards Minute-Scale High-Quality Video Generation [50.945885467651216]
  Diffusion models have revolutionized image and video generation, achieving unprecedented visual quality.<n>Recent work has explored autoregressive formulations for long video generation, typically by distilling from short-horizon bidirectional teachers.<n>We propose a simple yet effective approach to mitigate quality degradation in long-horizon video generation without requiring supervision from long-video teachers or retraining on long video datasets.
  arXiv  Detail & Related papers  (2025-10-02T17:55:42Z)
• Mixture of Contexts for Long Video Generation [72.96361488755986]
  We recast long-context video generation as an internal information retrieval task.<n>We propose a simple, learnable sparse attention routing module, Mixture of Contexts (MoC), as an effective long-term memory retrieval engine.<n>As we scale the data and gradually sparsify the routing, the model allocates compute to salient history, preserving identities, actions, and scenes over minutes of content.
  arXiv  Detail & Related papers  (2025-08-28T17:57:55Z)
• Scaling Instruction-Tuned LLMs to Million-Token Contexts via Hierarchical Synthetic Data Generation [15.975325252309554]
  We introduce a novel post-training synthetic data generation strategy designed to efficiently extend the context window of Large Language Models.<n>Our approach scalably extends to arbitrarily long context lengths, unconstrained by the length of available real-world data.<n>We demonstrate that our model, with a context length of up to 1M tokens, performs well on the RULER benchmark and InfiniteBench.
  arXiv  Detail & Related papers  (2025-04-17T04:46:57Z)
• Long-Context Autoregressive Video Modeling with Next-Frame Prediction [17.710915002557996]
  Long-context video modeling is essential for enabling generative models to function as world simulators.<n>While training directly on long videos is a natural solution, the rapid growth of vision tokens makes it computationally prohibitive.<n>We propose Frame AutoRegressive (FAR) models temporal dependencies between continuous frames, converges faster than video diffusion transformers, and outperforms token-level autoregressive models.
  arXiv  Detail & Related papers  (2025-03-25T03:38:06Z)
• Generalizing From Short to Long: Effective Data Synthesis for Long-Context Instruction Tuning [103.65680870130839]
  We investigate how to design instruction data for the post-training phase of a long context pre-trained model.<n>Our controlled study reveals that models instruction-tuned on short contexts can effectively generalize to longer ones.<n>Based on these findings, we propose context synthesis, a novel data synthesis framework.
  arXiv  Detail & Related papers  (2025-02-21T17:02:40Z)
• LongReD: Mitigating Short-Text Degradation of Long-Context Large Language Models via Restoration Distillation [79.90766312484489]
  Long Context Pre-training with Restoration Distillation (LongReD)<n>LongReD distills the hidden state of selected layers from the original model on short texts.<n>Experiments on common text benchmarks demonstrate that LongReD effectively preserves the model's short-text performance.
  arXiv  Detail & Related papers  (2025-02-11T08:37:16Z)
• NExtLong: Toward Effective Long-Context Training without Long Documents [28.002824369635768]
  We propose NExtLong, a novel framework for long-context data through Negative document Extension.<n> NExtLong decomposes a document into multiple meta-chunks and extends the context by interleaving hard negative distractors retrieved from pretraining corpora.<n>Extensive experiments demonstrate that NExtLong achieves significant performance improvements compared to existing long-context synthesis approaches.
  arXiv  Detail & Related papers  (2025-01-22T10:01:54Z)
• LIFT: Improving Long Context Understanding Through Long Input Fine-Tuning [35.31849814789343]
  This paper introduces Long Input Fine-Tuning (LIFT) for long context modeling.<n>LIFT enables efficient processing of lengthy inputs without the computational burden of offline long-context adaptation.<n>The framework is further enhanced by integrating in-context learning and pre-LIFT supervised fine-tuning.
  arXiv  Detail & Related papers  (2024-12-18T09:04:55Z)
• Effective Long-Context Scaling of Foundation Models [90.57254298730923]
  We present a series of long-context LLMs that support effective context windows of up to 32,768 tokens. Our models achieve consistent improvements on most regular tasks and significant improvements on long-context tasks over Llama 2.
  arXiv  Detail & Related papers  (2023-09-27T21:41:49Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.