Temporal Self-Distillation: Learning Visual State Tracking in Videos Without Supervision

• URL: http://arxiv.org/abs/2609.04203v1
• Date: Thu, 03 Sep 2026 17:59:55 GMT
• Title: Temporal Self-Distillation: Learning Visual State Tracking in Videos Without Supervision
• Abstract summary: S$3$T (Self-Supervised Self-Distillation over Time) is the first fully self-contained framework for continuous video state tracking.<n>On LLaVA-OneVision-2-8B, S$3$T improves VSTAT accuracy by $+1.74$ as a single model, $+2.38$ with souping, and $+2.70$ with additional vision-encoder adaptation.<n>The capability learned from unlabeled synthetic clips transfers to real videos, improving performance by $+7.95$ on VSTAT-YouTube state-tracking questions and $+4.50
• Score: 22.058460912910125
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We introduce S$^3$T (Self-Supervised Self-Distillation over Time), which, to the best of our knowledge, is the first fully self-contained framework for continuous video state tracking. Our method treats temporal sampling density as privileged information, based on the hypothesis that a denser view of the same clip recovers the running state more accurately. This view serves as the teacher, while a sparse-view student with the same weights learns to match its next-token distribution. The model generates its own target, so training requires no labels, separate teacher, or reward signal, and adds no inference cost. On LLaVA-OneVision-2-8B, S$^3$T improves VSTAT accuracy by $+1.74$ as a single model, $+2.38$ with souping, and $+2.70$ with additional vision-encoder adaptation, while prior self-evolving methods leave state tracking largely unchanged. The capability learned from unlabeled synthetic clips transfers to real videos, improving performance by $+7.95$ on VSTAT-YouTube state-tracking questions and $+4.50$ on MVBench Action Count.

Related papers

• StreamOPD: A Post-Training Recipe with Spatio-Temporal Cue Gating for Streaming Video Understanding [16.629848186775718]
  Reinforcement learning with verifiable rewards fits this regime poorly, encouraging long think-then-answer'' generations.<n>On-policy distillation supplies dense token-level teacher supervision on student trajectories but is stable only when both models train in thinking mode.<n>We provide a transparent and reproducible reference for open-source streaming-video research.
  arXiv  Detail & Related papers  (2026-08-17T09:25:36Z)
• Visual-OPSD: Cross-Modal On-Policy Self-Distillation for Efficient Unified Multimodal Reasoning [30.851590309402436]
  Unified multimodal models (UMMs) interleave generated ''visual thoughts'' (VTs) with text reasoning to improve spatial tasks.<n>This incurs roughly an order-of-magnitude inference cost from multi-step diffusion.<n>We propose Visual On-Policy Self-Distillation(Visual-OPSD) to solve this problem.
  arXiv  Detail & Related papers  (2026-06-17T11:59:15Z)
• MEMENTO: Teaching LLMs to Manage Their Own Context [50.3558738319336]
  Reasoning models think in long, unstructured streams with no mechanism for compressing or organizing their own intermediate state.<n>We introduce MEMENTO: a method that teaches models to segment reasoning into blocks, compress each block into a memento, and reason forward by attending only to mementos.<n>We show that a two-stage SFT recipe on OpenMementos is effective across different model families and scales.
  arXiv  Detail & Related papers  (2026-04-10T19:30:29Z)
• Learning from Streaming Video with Orthogonal Gradients [62.51504086522027]
  We address the challenge of representation learning from a continuous stream of video as input, in a self-supervised manner.<n>This differs from the standard approaches to video learning where videos are chopped and shuffled during training in order to create a non-redundant batch.<n>We demonstrate the drop in performance when moving from shuffled to sequential learning on three tasks.
  arXiv  Detail & Related papers  (2025-04-02T17:59:57Z)
• CoTracker3: Simpler and Better Point Tracking by Pseudo-Labelling Real Videos [63.90674869153876]
  We introduce CoTracker3, comprising a new tracking model and a new semi-supervised training recipe. This allows real videos without annotations to be used during training by generating pseudo-labels using off-the-shelf teachers. The model is available in online and offline variants and reliably tracks visible and occluded points.
  arXiv  Detail & Related papers  (2024-10-15T17:56:32Z)
• Refining Pre-Trained Motion Models [56.18044168821188]
  We take on the challenge of improving state-of-the-art supervised models with self-supervised training. We focus on obtaining a "clean" training signal from real-world unlabelled video. We show that our method yields reliable gains over fully-supervised methods in real videos.
  arXiv  Detail & Related papers  (2024-01-01T18:59:33Z)
• Look for the Change: Learning Object States and State-Modifying Actions from Untrimmed Web Videos [55.60442251060871]
  Human actions often induce changes of object states such as "cutting an apple" or "pouring coffee" We develop a self-supervised model for jointly learning state-modifying actions together with the corresponding object states. To cope with noisy uncurated training data, our model incorporates a noise adaptive weighting module supervised by a small number of annotated still images.
  arXiv  Detail & Related papers  (2022-03-22T11:45:10Z)
• ASCNet: Self-supervised Video Representation Learning with Appearance-Speed Consistency [62.38914747727636]
  We study self-supervised video representation learning, which is a challenging task due to 1) a lack of labels for explicit supervision and 2) unstructured and noisy visual information. Existing methods mainly use contrastive loss with video clips as the instances and learn visual representation by discriminating instances from each other. In this paper, we observe that the consistency between positive samples is the key to learn robust video representations.
  arXiv  Detail & Related papers  (2021-06-04T08:44:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.