LeVJEPA: Efficient & Scalable Video Pretraining without the Heuristics

• URL: http://arxiv.org/abs/2608.27395v1
• Date: Thu, 27 Aug 2026 17:26:24 GMT
• Title: LeVJEPA: Efficient & Scalable Video Pretraining without the Heuristics
• Abstract summary: We introduce LeVJEPA, the first video encoder trained under LeJEPA's collapse-free objective.<n>A single encoder is trained with an invariance loss over global and local views of a clip.<n>LeVJEPA matches or surpasses V-JEPA 2 across ViT-S/B/L at 5.6 to 20.8x less pretraining compute.
• Score: 50.45297537821907
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Video carries the temporal structure of the physical world, yet learning representations from it has remained computationally expensive: prevailing self-supervised methods either prevent representation collapse through architectural asymmetries, coupling an exponential-moving-average target encoder, a stop-gradient, and a capacity-limited predictor, or circumvent it by reconstructing masked content in pixel space. We introduce LeVJEPA, the first video encoder trained under LeJEPA's collapse-free objective, which dispenses with both. A single encoder is trained with an invariance loss over global and local views of a clip, regularized by SIGReg, which excludes collapse with a provable guarantee. The architecture reduces to an encoder and a projector, and the objective to a single hyperparameter. This formulation admits two properties. First, the cost of pretraining is governed by the number of tokens the encoder observes; uniform random token dropping renders this number small while simultaneously improving downstream accuracy. At matched epochs on identical data, LeVJEPA matches or surpasses V-JEPA 2 across ViT-S/B/L at 5.6 to 20.8x less pretraining compute, and at matched total FLOPs it exceeds the strongest video baseline by 7.6 points on ImageNet-1K while remaining competitive on motion-centric benchmarks. Second, since no asymmetry between branches is required, the encoder can be trained with block-causal attention at no measurable accuracy cost: temporal ordering becomes a property of the encoder itself. Against a compute-matched DINOv2 trained on frames of the same videos, LeVJEPA approaches the image-pretrained encoder on appearance-centric evaluation while nearly doubling its motion-centric accuracy. These results indicate that, once its computational overhead is removed, video becomes a viable and in several respects preferable substrate for general-purpose visual pretraining.

Related papers

• PACE: A Unified Condense-and-Extract Paradigm for Fast VLM Inference [15.233755808671871]
  Existing visual token pruning methods exhibit two fundamental limitations.<n>Most approaches operate exclusively post-vision encoder, leaving the substantial latency of the visual encoding phase unoptimized.<n>We propose PACE, a framework that accelerates both the vision encoder and the Large Language Model.
  arXiv  Detail & Related papers  (2026-08-27T14:52:09Z)
• SPLIT: Training-Free AI-Generated and Partially Edited Video Detection via Spatial Patch-Level Incoherence and Temporal Roughness [4.774658730084455]
  We introduce a training-free detector that operates on patch tokens from a frozen vision encoder to detect both fully generated and partially edited videos.<n>SPLIT computes two complementary signals: Two-step Temporal Roughness (TTR) and Local Spatial Motion Incoherence (LSMI)<n>We propose a service-aligned evaluation protocol based on Fake Recall at fixed FPR with real-only threshold calibration and cross-real threshold transfer.
  arXiv  Detail & Related papers  (2026-07-03T02:34:50Z)
• Beyond Encoder Accumulation: Measuring Encoder Roles in Multi-Encoder VLMs [50.3757247538081]
  We retrain and evaluate all 31 non-empty subsets of five common vision encoders under a unified pipeline.<n>We decompose each encoder's contribution into two axes, Capacity and Necessity.<n>Per-encoder pre-projector effective rank explains the residual score variation.
  arXiv  Detail & Related papers  (2026-06-02T16:46:42Z)
• EVATok: Adaptive Length Video Tokenization for Efficient Visual Autoregressive Generation [80.13014959623452]
  EVATok is a framework to produce $textbfE$fficient $textbfV$ideo $textbfA$daptive $textbfTok$enizers.<n>We show that EVATok achieves superior reconstruction and state-of-the-art class-to-video generation on UCF-101.
  arXiv  Detail & Related papers  (2026-03-12T17:59:59Z)
• PIO-FVLM: Rethinking Training-Free Visual Token Reduction for VLM Acceleration from an Inference-Objective Perspective [59.24570811503256]
  We propose PIO-FVLM to reduce redundant visual tokens in vision-models (VLMs) to accelerate inference.<n>The proposed PIO-FVLM is training-free and compatible with FlashAttention, friendly to practical application and deployment.<n>On LLaVA-Next-7B, PIO-FVLM retains just 11.1% of visual tokens but maintains 97.2% of the original performance.
  arXiv  Detail & Related papers  (2026-02-04T15:33:10Z)
• SpecVLM: Fast Speculative Decoding in Vision-Language Models [14.243294546325714]
  Speculative decoding is a powerful way to accelerate autoregressive large language models (LLMs)<n>We study speculative decoding for vision-language models (VLMs)<n>We introduce SpecVLM, a practical system that delivers 1.5--2.3x end-to-end speedups over full autoregressive inference.
  arXiv  Detail & Related papers  (2025-09-15T11:53:56Z)
• SparseTem: Boosting the Efficiency of CNN-Based Video Encoders by Exploiting Temporal Continuity [19.900719882624028]
  We propose a memory-efficient scheduling method to eliminate memory overhead and an online adjustment mechanism to minimize accuracy degradation.<n>SparseTem achieves speedup of 1.79x for EfficientDet and 4.72x for CRNN, with minimal accuracy drop and no additional memory overhead.
  arXiv  Detail & Related papers  (2024-10-28T07:13:25Z)
• Progressive Token Length Scaling in Transformer Encoders for Efficient Universal Segmentation [67.85309547416155]
  A powerful architecture for universal segmentation relies on transformers that encode multi-scale image features and decode object queries into mask predictions.<n>With efficiency being a high priority for scaling such models, we observed that the state-of-the-art method Mask2Former uses 50% of its compute only on the transformer encoder.<n>This is due to the retention of a full-length token-level representation of all backbone feature scales at each encoder layer.
  arXiv  Detail & Related papers  (2024-04-23T01:34:20Z)
• It Takes Two: Masked Appearance-Motion Modeling for Self-supervised Video Transformer Pre-training [76.69480467101143]
  Self-supervised video transformer pre-training has recently benefited from the mask-and-predict pipeline. We explicitly investigate motion cues in videos as extra prediction target and propose our Masked Appearance-Motion Modeling framework. Our method learns generalized video representations and achieves 82.3% on Kinects-400, 71.3% on Something-Something V2, 91.5% on UCF101, and 62.5% on HMDB51.
  arXiv  Detail & Related papers  (2022-10-11T08:05:18Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.