SUNTA: Hierarchical Video Prediction with Surprise-based Chunking

• URL: http://arxiv.org/abs/2607.02087v1
• Date: Thu, 02 Jul 2026 12:27:54 GMT
• Title: SUNTA: Hierarchical Video Prediction with Surprise-based Chunking
• Abstract summary: We argue that chunking should instead be driven by prediction errors, which more directly indicate when longer-range context becomes necessary.<n>We propose Surprise-based Nested Temporal Abstraction (SUNTA), a method that employs a decoupled training strategy to preserve surprise signals.<n>Experiments on video prediction tasks in 2D and 3D environments demonstrate that SUNTA outperforms baselines.
• Score: 26.31264097587371
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Hierarchical state-space models (HSSMs) offer a promising approach to long-horizon prediction by segmenting sequences into temporal chunks. However, their performance hinges on how chunk boundaries are determined. While prior HSSMs typically rely on fixed-length chunking or similarity-based boundary detection, these methods often misalign with the intrinsic temporal structure of the data. We argue that chunking should instead be driven by prediction errors, which more directly indicate when longer-range context becomes necessary. Nevertheless, integrating surprise-based chunking into HSSMs introduces critical challenges, including hierarchical collapse during end-to-end training and the absence of surprise signals during open-loop prediction. To address these issues, we propose Surprise-based Nested Temporal Abstraction (SUNTA), a method that employs a decoupled training strategy to preserve surprise signals and uses internal inconsistency as a top-down surprise metric to determine chunk boundaries within imagined rollouts. Experiments on video prediction tasks in 2D and 3D environments demonstrate that SUNTA outperforms baselines, uniquely maintaining accurate predictions over 250 timesteps, whereas all baselines degrade within the first 10 timesteps.

Related papers

• Surprise Forcing: What to Remember, When to Skip in Long Video Generation [65.0022589149937]
  Streaming autoregressive diffusion makes minute-scale video synthesis practical, but its bounded context and fixed denoising schedule allocate resources uniformly.<n>We introduce Surprise Forcing, a training-free framework that treats both limitations as online resource-allocation problems.<n> Experiments on VBench, VBench-Long, and VBench-2.0 show that the proposed allocation strategy improves long-horizon consistency and visual quality while retaining real-time streaming throughput.
  arXiv  Detail & Related papers  (2026-07-20T18:37:58Z)
• ChunkFlow: Towards Continuity-Consistent Chunked Policy Learning [5.727608625370017]
  ChunkFlow is a seam-aware training-and-execution framework for chunked policies.<n>It partitions each chunk into frozen, editable, and future zones, applies deterministic overlap blending at execution, and trains raw predictions with seam and first- and second-order continuity losses.<n> Experiments on CALVIN, LIBERO, and real robots show an improved success-stability trade-off with low-latency inference.
  arXiv  Detail & Related papers  (2026-07-14T17:43:25Z)
• Surprised by Attention: Predictable Query Dynamics for Time Series Anomaly Detection [1.7751300245073598]
  AxonAD is an unsupervised detector that treats multi-head attention query evolution as a short horizon predictable process.<n>A tail-aggregated query mismatch score measures cosine deviation between predicted and target queries on recent timesteps.<n>On in-vehicle telemetry with interval annotations, AxonAD improves ranking quality and temporal localization over strong baselines.
  arXiv  Detail & Related papers  (2026-03-13T11:40:51Z)
• Generative Regression Based Watch Time Prediction for Short-Video Recommendation [36.95095097454143]
  Watch time prediction has emerged as a pivotal task in short video recommendation systems.<n>Recent studies have attempted to address these issues by converting the continuous watch time estimation into an ordinal regression task.<n>We propose a novel Generative Regression (GR) framework that reformulates WTP as a sequence generation task.
  arXiv  Detail & Related papers  (2024-12-28T16:48:55Z)
• Spatial Decomposition and Temporal Fusion based Inter Prediction for Learned Video Compression [59.632286735304156]
  We propose a spatial decomposition and temporal fusion based inter prediction for learned video compression. With the SDD-based motion model and long short-term temporal fusion, our proposed learned video can obtain more accurate inter prediction contexts.
  arXiv  Detail & Related papers  (2024-01-29T03:30:21Z)
• Streaming Motion Forecasting for Autonomous Driving [71.7468645504988]
  We introduce a benchmark that queries future trajectories on streaming data and we refer to it as "streaming forecasting" Our benchmark inherently captures the disappearance and re-appearance of agents, which is a safety-critical problem yet overlooked by snapshot-based benchmarks. We propose a plug-and-play meta-algorithm called "Predictive Streamer" that can adapt any snapshot-based forecaster into a streaming forecaster.
  arXiv  Detail & Related papers  (2023-10-02T17:13:16Z)
• Temporal Action Localization with Enhanced Instant Discriminability [66.76095239972094]
  Temporal action detection (TAD) aims to detect all action boundaries and their corresponding categories in an untrimmed video. We propose a one-stage framework named TriDet to resolve imprecise predictions of action boundaries by existing methods. Experimental results demonstrate the robustness of TriDet and its state-of-the-art performance on multiple TAD datasets.
  arXiv  Detail & Related papers  (2023-09-11T16:17:50Z)
• Uncovering the Missing Pattern: Unified Framework Towards Trajectory Imputation and Prediction [60.60223171143206]
  Trajectory prediction is a crucial undertaking in understanding entity movement or human behavior from observed sequences. Current methods often assume that the observed sequences are complete while ignoring the potential for missing values. This paper presents a unified framework, the Graph-based Conditional Variational Recurrent Neural Network (GC-VRNN), which can perform trajectory imputation and prediction simultaneously.
  arXiv  Detail & Related papers  (2023-03-28T14:27:27Z)
• Semi-Supervised Temporal Action Detection with Proposal-Free Masking [134.26292288193298]
  We propose a novel Semi-supervised Temporal action detection model based on PropOsal-free Temporal mask (SPOT) SPOT outperforms state-of-the-art alternatives, often by a large margin.
  arXiv  Detail & Related papers  (2022-07-14T16:58:47Z)
• Interpretable Deep Feature Propagation for Early Action Recognition [39.966828592322315]
  In this study, we address action prediction by investigating how action patterns evolve over time in a spatial feature space. We work with intermediate-layer ConvNet features, which allow for abstraction from raw data, while retaining spatial layout. We employ a Kalman filter to combat error build-up and unify across prediction start times.
  arXiv  Detail & Related papers  (2021-07-11T19:40:19Z)
• Revisiting Hierarchical Approach for Persistent Long-Term Video Prediction [55.4498466252522]
  We set a new standard of video prediction with orders of magnitude longer prediction time than existing approaches. Our method predicts future frames by first estimating a sequence of semantic structures and subsequently translating the structures to pixels by video-to-video translation. We evaluate our method on three challenging datasets involving car driving and human dancing, and demonstrate that it can generate complicated scene structures and motions over a very long time horizon.
  arXiv  Detail & Related papers  (2021-04-14T08:39:38Z)
• A PAC-Bayesian Perspective on Structured Prediction with Implicit Loss Embeddings [11.711761707845865]
  PAC-Bayes has gained interest recently for its capacity of producing tight risk bounds for predictor distributions. We present two generalization bounds, on the risk and excess risk, which yield insights into the behavior of ILE predictors. Two learning algorithms are derived from these bounds.
  arXiv  Detail & Related papers  (2020-12-07T15:19:43Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.