ThinkJEPA: Empowering Latent World Models with Large Vision-Language Reasoning Model

• URL: http://arxiv.org/abs/2603.22281v1
• Date: Mon, 23 Mar 2026 17:59:42 GMT
• Title: ThinkJEPA: Empowering Latent World Models with Large Vision-Language Reasoning Model
• Abstract summary: Vision-language models (VLMs) provide strong semantic grounding and general knowledge by reasoning over uniformly sampled frames.<n>We propose a VLM-guided JEPA-style latent world modeling framework that combines dense-frame dynamics modeling with long-horizon semantic guidance.
• Score: 53.15040805435013
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Recent progress in latent world models (e.g., V-JEPA2) has shown promising capability in forecasting future world states from video observations. Nevertheless, dense prediction from a short observation window limits temporal context and can bias predictors toward local, low-level extrapolation, making it difficult to capture long-horizon semantics and reducing downstream utility. Vision--language models (VLMs), in contrast, provide strong semantic grounding and general knowledge by reasoning over uniformly sampled frames, but they are not ideal as standalone dense predictors due to compute-driven sparse sampling, a language-output bottleneck that compresses fine-grained interaction states into text-oriented representations, and a data-regime mismatch when adapting to small action-conditioned datasets. We propose a VLM-guided JEPA-style latent world modeling framework that combines dense-frame dynamics modeling with long-horizon semantic guidance via a dual-temporal pathway: a dense JEPA branch for fine-grained motion and interaction cues, and a uniformly sampled VLM \emph{thinker} branch with a larger temporal stride for knowledge-rich guidance. To transfer the VLM's progressive reasoning signals effectively, we introduce a hierarchical pyramid representation extraction module that aggregates multi-layer VLM representations into guidance features compatible with latent prediction. Experiments on hand-manipulation trajectory prediction show that our method outperforms both a strong VLM-only baseline and a JEPA-predictor baseline, and yields more robust long-horizon rollout behavior.

Related papers

• MMaDA-VLA: Large Diffusion Vision-Language-Action Model with Unified Multi-Modal Instruction and Generation [47.2088748094605]
  Vision-Language-Action (VLA) models aim to control robots for manipulation from visual observations and natural-language instructions.<n>We present MMaDA-VLA, a fully native pre-trained large diffusion VLA model that unifies multi-modal understanding and generation in a single framework.
  arXiv  Detail & Related papers  (2026-03-26T12:55:51Z)
• Latent-WAM: Latent World Action Modeling for End-to-End Autonomous Driving [40.17041348571413]
  Latent-WAM is an efficient end-to-end autonomous driving framework.<n>It achieves strong trajectory planning through spatially-aware and dynamics-informed latent world representations.
  arXiv  Detail & Related papers  (2026-03-25T17:56:07Z)
• Insight-V++: Towards Advanced Long-Chain Visual Reasoning with Multimodal Large Language Models [65.4947731385794]
  We present a unified multi-agent visual reasoning framework that evolves from our foundational image-centric model, Insight-V.<n>We introduce two novel algorithms, ST-GRPO and J-GRPO, which enhance spatial-temporal reasoning and improve evaluative robustness.<n>Experiments on base models like LLaVA-NeXT and Qwen2.5-VL demonstrate significant performance gains across challenging image and video reasoning benchmarks.
  arXiv  Detail & Related papers  (2026-03-18T15:28:07Z)
• AutoTraces: Autoregressive Trajectory Forecasting via Multimodal Large Language Models [5.085937845977223]
  AutoTraces is an autoregressive vision-trajectory model for robot trajectory forecasting in humam-language environments.<n>Key innovation lies in a trajectory tokenization scheme, which represents waypoints with point tokens as categorical markers while encoding waypoint numerical values as corresponding point embeddings.
  arXiv  Detail & Related papers  (2026-03-09T05:53:06Z)
• SGHA-Attack: Semantic-Guided Hierarchical Alignment for Transferable Targeted Attacks on Vision-Language Models [73.19044613922911]
  Large vision-language models (VLMs) are vulnerable to transfer-based adversarial perturbations.<n>We propose SGHA-Attack, a framework that adopts multiple target references and enforces intermediate-layer consistency.<n>Experiments on open-source and commercial black-box VLMs show that SGHA-Attack achieves stronger targeted transferability than prior methods.
  arXiv  Detail & Related papers  (2026-02-02T03:10:41Z)
• VFMF: World Modeling by Forecasting Vision Foundation Model Features [67.09340259579761]
  We introduce a generative forecaster that performs autoregressive flow matching in vision foundation models feature space.<n>We show that this latent information more effectively than previously used PCA-based alternatives, both for forecasting and other applications.<n>With matched architecture and compute, our method produces sharper and more accurate predictions than regression across all modalities.
  arXiv  Detail & Related papers  (2025-12-12T02:10:05Z)
• Scaling Up Occupancy-centric Driving Scene Generation: Dataset and Method [54.461213497603154]
  Occupancy-centric methods have recently achieved state-of-the-art results by offering consistent conditioning across frames and modalities.<n>Nuplan-Occ is the largest occupancy dataset to date, constructed from the widely used Nuplan benchmark.<n>We develop a unified framework that jointly synthesizes high-quality occupancy, multi-view videos, and LiDAR point clouds.
  arXiv  Detail & Related papers  (2025-10-27T03:52:45Z)
• LLM-Integrated Bayesian State Space Models for Multimodal Time-Series Forecasting [20.735693014588236]
  We introduce Bayesian State space models (LBS), a novel framework for multimodal temporal forecasting.<n>At a high level, LBS consists of two components: (1) a state space model (SSM) backbone that captures the temporal dynamics of latent states from which both numerical and textual observations are generated.<n> Experiments on the TextTimeCorpus benchmark demonstrate that LBS improves the previous state-of-the-art by 13.20% while providing human-readable summaries of each forecast.
  arXiv  Detail & Related papers  (2025-10-23T19:28:26Z)
• Vision-LLMs for Spatiotemporal Traffic Forecasting [14.700408329373998]
  Large Language Models (LLMs) inherently struggle to model the complex spatial dependencies of grid-based traffic data.<n>We propose ST-Vision-LLM, a novel framework reframe thatstemporal forecasting as a vision-language fusion problem.<n>We show that ST-Vision-LLM outperforms existing methods by 15.6% in long-term prediction accuracy and exceeds the second-best baseline by over 30.04% in cross-domain scenarios.
  arXiv  Detail & Related papers  (2025-10-13T11:15:56Z)
• SG-LDM: Semantic-Guided LiDAR Generation via Latent-Aligned Diffusion [0.11184789007828977]
  Lidar point cloud synthesis offers a promising solution to augment deep learning pipelines.<n>Existing methods focus on unconditional lidar point cloud generation, overlooking their potential for real-world applications.<n>We propose SG-LDM, a Semantic-Guided Lidar Diffusion Model that employs latent alignment to enable robust semantic-to-lidar synthesis.
  arXiv  Detail & Related papers  (2025-06-30T08:13:04Z)
• LaDi-WM: A Latent Diffusion-based World Model for Predictive Manipulation [45.02469804709771]
  We propose LaDi-WM, a world model that predicts the latent space of future states using diffusion modeling.<n>We show that LaDi-WM significantly enhances policy performance by 27.9% on the LIBERO-LONG benchmark and 20% on the real-world scenario.
  arXiv  Detail & Related papers  (2025-05-13T04:42:14Z)
• BEVWorld: A Multimodal World Simulator for Autonomous Driving via Scene-Level BEV Latents [56.33989853438012]
  We propose BEVWorld, a framework that transforms multimodal sensor inputs into a unified and compact Bird's Eye View latent space for holistic environment modeling.<n>The proposed world model consists of two main components: a multi-modal tokenizer and a latent BEV sequence diffusion model.
  arXiv  Detail & Related papers  (2024-07-08T07:26:08Z)
• MUSE-VAE: Multi-Scale VAE for Environment-Aware Long Term Trajectory Prediction [28.438787700968703]
  Conditional MUSE offers diverse and simultaneously more accurate predictions compared to the current state-of-the-art. We demonstrate these assertions through a comprehensive set of experiments on nuScenes and SDD benchmarks as well as PFSD, a new synthetic dataset.
  arXiv  Detail & Related papers  (2022-01-18T18:40:03Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.