FactorJEPA: Factorizing Monolithic Futures into Layout-Agent-Interaction Channels for Crowded and Chaotic Global South Urban Worlds

• URL: http://arxiv.org/abs/2608.01049v1
• Date: Sun, 02 Aug 2026 07:32:46 GMT
• Title: FactorJEPA: Factorizing Monolithic Futures into Layout-Agent-Interaction Channels for Crowded and Chaotic Global South Urban Worlds
• Abstract summary: We study a largely unexplored regime: populous, crowded, and chaotic Global South urban environments.<n>Unlike the lower-density, lane-structured settings that dominate existing evaluations, these scenes exhibit soft spatial boundaries.<n>We introduce the first large-scale dataset for this regime: 1,000 hours of drive-through, walk-through, and aerial video across 22 cities.
• Score: 20.90457097908955
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: World models have attracted significant attention for their ability to capture and predict the structure and dynamics of the physical world. In this emerging landscape, Joint Embedding Predictive Architectures (JEPA) offer a particularly compelling direction. We study a largely unexplored regime: populous, crowded, and chaotic Global South urban environments, which we call DENSEWORLD. Unlike the lower-density, lane-structured settings that dominate existing evaluations, these scenes exhibit soft spatial boundaries, extreme agent heterogeneity, persistent occlusion, and rapid social negotiation under mixed traffic. We introduce the first large-scale dataset for this regime: 1,000 hours of drive-through, walk-through, and aerial video across 22 cities. Existing JEPA formulations struggle to preserve dense interaction dynamics under heterogeneity and partial observability. We introduce FactorJEPA, which makes world structure a first-class predictive primitive. Rather than encoding the future in a monolithic latent, it composes layout, entities, and interactions, using a visibility gate and separated subspaces to preserve partially observed agents and discourage cross-factor shortcuts. FactorJEPA improves (i) future-latent accuracy (Future-frame L1), (ii) intervention-sensitive prediction (Causal L1), and (iii) robustness to reduced visual evidence (Mask-ratio slope), while exposing (iv) a reproducible motion-information trade-off (Motion cosine). Method rankings replicate across 2B and 1B V-JEPA 2.1 backbones, with rho = 0.895 to 0.978. We publicly release the DENSEWORLD-115k dataset (https://huggingface.co/datasets/anonymousML123/denseworld-115k) and the surgery-trained FactorJEPA checkpoints (https://huggingface.co/datasets/anonymousML123/factorjepa-outputs/tree/main/outputs/full/vjepa_2_1_ vitg_1B/train/m09c_surgery_3stage_DI_diheavy_encoder).

Related papers

• Orthogonal JEPA: Factorized Predictive States for Latent World Models [95.20158869896393]
  We introduce method, a latent world-modeling framework based on predictive factorization.<n>We show that method can allocate redundant capacity to dominant signals while providing weak or conflicting gradients to less dominant predictive structure.<n>Experiments on controlled vision, single-cell transcriptomics, longitudinal health records, continuous control, and molecular dynamics evaluate representation quality, forecasting, planning, and long-horizon stability.
  arXiv  Detail & Related papers  (2026-08-20T13:59:57Z)
• UniJEPA: A Unified Joint-Embedding Predictive Architecture for Task-Agnostic Visual World Modeling [24.745715007832953]
  Joint-Embedding Predictive Architectures (JEPAs) have emerged as a principled framework for self-supervised learning of world models in compact latent spaces.<n>We present UniJEPA, a unified JEPA that jointly learns photometric prediction (image-level transformations) and temporal prediction (video-level next-state dynamics) in one shared latent space.
  arXiv  Detail & Related papers  (2026-08-07T16:55:58Z)
• UWM-JEPA: Predictive World Models That Imagine in Belief Space [0.2864713389096699]
  We introduce the Unitary World Model JEPA, a JEPA world model with a density-matrix latent on a joint system-environment space and a learned unitary predictor.<n>The construction preserves the joint-state spectrum exactly during rollout, so the predictor itself cannot dissipate the represented uncertainty.<n>For JEPA world models to imagine under partial observability, latent geometry and predictor dynamics matter, not frozen context-encoding capacity alone.
  arXiv  Detail & Related papers  (2026-05-25T00:28:51Z)
• Entity-Centric World Models: Interaction-Aware Masking for Causal Video Prediction [0.0]
  Joint Embedding Predictive Architectures (JEPA) have set new benchmarks in semantic classification.<n>We propose Interaction-Aware JEPA (IA-JEPA), which utilizes a self-supervised motion-centric masking strategy.<n>IA-JEPA achieves 14.26% accuracy on causal reasoning tasks, a significant lead over the 3.22% achieved by standard patch-masked baselines.
  arXiv  Detail & Related papers  (2026-05-14T23:10:04Z)
• ThinkJEPA: Empowering Latent World Models with Large Vision-Language Reasoning Model [53.15040805435013]
  Vision-language models (VLMs) provide strong semantic grounding and general knowledge by reasoning over uniformly sampled frames.<n>We propose a VLM-guided JEPA-style latent world modeling framework that combines dense-frame dynamics modeling with long-horizon semantic guidance.
  arXiv  Detail & Related papers  (2026-03-23T17:59:42Z)
• Probing the Latent World: Emergent Discrete Symbols and Physical Structure in Latent Representations [0.0]
  Video world models trained with Joint Embedding Predictive Architectures (JEPA) acquire rich representations by predicting masked regions in latent space rather than reconstructing pixels.<n>This pathway removes the visual verification of generative models, creating a structural interpretability gap: the encoder has learned physical structure inaccessible in any inspectable form.<n>We propose the AI Mother Tongue (AIM) framework as a passive quantization probe: a lightweight, vocabulary-free probe that converts V-JEPA 2 continuous latent vectors into discrete symbol sequences without task-specific supervision or modifying the encoder.
  arXiv  Detail & Related papers  (2026-03-20T01:30:59Z)
• LeWorldModel: Stable End-to-End Joint-Embedding Predictive Architecture from Pixels [49.35636088613484]
  Joint Embedding Predictive Architectures (JEPAs) offer a compelling framework for learning world models in compact latent spaces.<n>We introduce LeModelWorld, the first JEPA that trains stably end-to-end from raw pixels using only two loss terms.<n>With 15M parameters trainable on a single GPU in a few hours, LeWM plans up to 48x faster than foundation-model-based world models.
  arXiv  Detail & Related papers  (2026-03-13T19:48:14Z)
• BiJEPA: Bi-directional Joint Embedding Predictive Architecture for Symmetric Representation Learning [0.0]
  BiJEPA enforces cycle-consistent predictability between data segments.<n>We evaluate BiJEPA on three distinct modalities: synthetic periodic signals, chaotic Lorenz attractor trajectories, and high-dimensional image data.
  arXiv  Detail & Related papers  (2026-02-10T01:16:20Z)
• UniSH: Unifying Scene and Human Reconstruction in a Feed-Forward Pass [83.7071371474926]
  UniSH is a unified, feed-forward framework for joint metric-scale 3D scene and human reconstruction.<n>Our framework bridges strong, disparate priors from scene reconstruction and HMR.<n>Our model achieves state-of-the-art performance on human-centric scene reconstruction.
  arXiv  Detail & Related papers  (2026-01-03T16:06:27Z)
• LeJEPA: Provable and Scalable Self-Supervised Learning Without the Heuristics [53.247652209132376]
  Joint-Embedding Predictive Architectures (JEPAs) offer a promising blueprint, but lack of practical guidance and theory has led to ad-hoc R&D.<n>We present a comprehensive theory of JEPAs and instantiate it in bf LeJEPA, a lean, scalable, and theoretically grounded training objective.
  arXiv  Detail & Related papers  (2025-11-11T18:21:55Z)
• Trajectory World Models for Heterogeneous Environments [67.27233466954814]
  Heterogeneity in sensors and actuators across environments poses a significant challenge to building large-scale pre-trained world models.<n>We introduce UniTraj, a unified dataset comprising over one million trajectories from 80 environments, designed to scale data while preserving critical diversity.<n>We also propose TrajWorld, a novel architecture capable of handling varying sensor and actuator information and capturing environment dynamics in-context.
  arXiv  Detail & Related papers  (2025-02-03T13:59:08Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.