Orca: The World is in Your Mind

• URL: http://arxiv.org/abs/2606.30534v2
• Date: Tue, 30 Jun 2026 13:57:56 GMT
• Title: Orca: The World is in Your Mind
• Abstract summary: We introduce Orca, an initial instantiation of a general world foundation model.<n>Orca learns a unified world latent space from multimodal world signals and exposes it through multimodal readout interfaces.
• Score: 82.01041475853545
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We introduce Orca, an initial instantiation of a general world foundation model. Orca learns a unified world latent space from multimodal world signals and exposes it through multimodal readout interfaces. Rather than optimizing isolated next-token, next-frame, or next-action prediction, we are centered on Next-State-Prediction modeling, offering a unified state-transition modeling route toward understanding, predicting, and acting upon the world. Orca learns through two complementary paradigms: unconscious learning captures dense natural state transitions from continuous videos, and conscious learning models sparse meaningful state transitions by language-described events and VQA supervision. For pre-training, we construct a large-scale world-learning inventory data, including 125K hours of video data and 160M event annotations. After pre-training, Orca learns a unified world latent space. To examine whether the learned latent supports downstream, we evaluate it by three representative downstream readouts: text generation, image prediction, and embodied action generation. Orca's backbone is frozen, and only the lightweight modality-specific decoders are trainable. Experiments show the scalability of the proposed paradigm and verify that stronger world latent enables stronger downstream readouts. Orca outperforms similar-sized specialized baselines. These results show that Orca, as a general world foundation model, presents a promising approach to understanding, predicting, and acting upon the world. Finally, we discuss the current limitations, aiming to provide useful insights and inspiration for the community.

Related papers

• Code World Model: Coding Agent as World Brain [51.31807545747191]
  We introduce Code World Model, a framework that separates world evolution from visual realization.<n>A coding agent serves as the world brain, reasoning about events and their consequences.<n>To connect executable state with visual generation, we introduce a proxy representation that encodes frame-wisetemporal constraints into a proxy video.
  arXiv  Detail & Related papers  (2026-08-26T15:37:33Z)
• PhiZero: A World Model Built Around Physical Language [50.268271884186845]
  PhiZero is a physical world model built around physical language.<n>It infers future world evolution as a physical-language sequence and then renders the inferred transitions into videos.<n>We show its potential for realistic and interactive world modeling, fine-grained action-conditioned simulation, and zero-shot motion transfer.
  arXiv  Detail & Related papers  (2026-07-30T17:59:46Z)
• X-Foresight: A Joint Vision-Action Causal Forecasting Network via Predictive World Modeling [47.54820149491433]
  We introduce X-Foresight, a predictive world model integrated directly into the Vision-Language-Action architecture.<n>By predicting semantically distant chunks rather than adjacent frames, X-Foresight escapes trivial extrapolation.<n> Comprehensive experiments demonstrate that X-Foresight significantly outperforms VLA baselines in planning performance.
  arXiv  Detail & Related papers  (2026-05-24T06:37:04Z)
• DreamWorld: Unified World Modeling in Video Generation [32.857497363728584]
  We introduce textbfDreamWorld, a unified framework that integrates complementary world knowledge into video generators.<n>We show that DreamWorld improves world consistency, outperforming Wan2.1 by 2.26 points on VBench.
  arXiv  Detail & Related papers  (2026-02-28T05:02:39Z)
• Can World Models Benefit VLMs for World Dynamics? [59.73433292793044]
  We investigate the capabilities when world model priors are transferred into Vision-Language Models.<n>We name our best-performing variant Dynamic Vision Aligner (DyVA)<n>We find DyVA to surpass both open-source and proprietary baselines, achieving state-of-the-art or comparable performance.
  arXiv  Detail & Related papers  (2025-10-01T13:07:05Z)
• Back to the Features: DINO as a Foundation for Video World Models [29.116268020812072]
  We present DINO-world, a powerful generalist video world model trained to predict future frames in the latent space of DINOv2.<n>By leveraging a pre-trained image encoder and training a future predictor on a large-scale uncurated video dataset, DINO-world learns the temporal dynamics of diverse scenes.
  arXiv  Detail & Related papers  (2025-07-25T17:54:10Z)
• Pandora: Towards General World Model with Natural Language Actions and Video States [61.30962762314734]
  Pandora is a hybrid autoregressive-diffusion model that simulates world states by generating videos and allows real-time control with free-text actions. Pandora achieves domain generality, video consistency, and controllability through large-scale pretraining and instruction tuning.
  arXiv  Detail & Related papers  (2024-06-12T18:55:51Z)
• Predicting Long-horizon Futures by Conditioning on Geometry and Time [49.86180975196375]
  We explore the task of generating future sensor observations conditioned on the past. We leverage the large-scale pretraining of image diffusion models which can handle multi-modality. We create a benchmark for video prediction on a diverse set of videos spanning indoor and outdoor scenes.
  arXiv  Detail & Related papers  (2024-04-17T16:56:31Z)
• ProxyCap: Real-time Monocular Full-body Capture in World Space via Human-Centric Proxy-to-Motion Learning [47.228139909747235]
  We introduce ProxyCap, a human-centric proxy-to-motion learning scheme to learn world-space motions. Our network is designed to learn human motions from a human-centric perspective. We demonstrate the first real-time monocular full-body capture system with plausible foot-ground contact in world space.
  arXiv  Detail & Related papers  (2023-07-03T17:59:45Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.