Cosmos 3: Omnimodal World Models for Physical AI

• URL: http://arxiv.org/abs/2606.02800v2
• Date: Fri, 05 Jun 2026 16:34:56 GMT
• Title: Cosmos 3: Omnimodal World Models for Physical AI
• Abstract summary: We introduce Cosmos 3, a family of omnimodal world models designed to jointly process and generate language, image, video, audio, and action sequences.<n>Our evaluation demonstrates that Cosmos 3 unifies a new state-of-the-art across a diverse suite of understanding and generation tasks.
• Score: 321.3056180868173
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We introduce Cosmos 3, a family of omnimodal world models designed to jointly process and generate language, image, video, audio, and action sequences within a unified mixture-of-transformers architecture. By supporting highly flexible input-output configurations, Cosmos 3 seamlessly unifies critical modalities for Physical AI -- effectively subsuming vision-language models, video generators, world simulators, and world-action models into a single framework. Our evaluation demonstrates that Cosmos 3 establishes a new state-of-the-art across a diverse suite of understanding and generation tasks, demonstrating omnimodal world models as scalable, general-purpose backbones for embodied agents. Our post-trained Cosmos 3 models were ranked as the best open-source Text-to-Image and Image-to-Video models by Artificial Analysis, and the best policy model by RoboArena at the time the technical report was written. To accelerate open research and deployment in Physical AI, we make our code, model checkpoints, curated synthetic datasets, and evaluation benchmark available under the Linux Foundation's OpenMDW-1.1 License at https://github.com/nvidia/cosmos and https://huggingface.co/collections/nvidia/cosmos3. The project website is available at https://research.nvidia.com/labs/cosmos-lab/cosmos3.

Related papers

• WorldAgents: Can Foundation Image Models be Agents for 3D World Models? [82.83725150353915]
  We demonstrate that 2D models do indeed encapsulate a grasp of 3D worlds.<n>By exploiting this understanding, our method successfully synthesizes expansive, realistic, and 3D-consistent worlds.
  arXiv  Detail & Related papers  (2026-03-20T07:22:41Z)
• Latent Particle World Models: Self-supervised Object-centric Stochastic Dynamics Modeling [51.40150411616207]
  We introduce Latent Particle World Model (LPWM), a self-supervised object-centric world model scaled to real-world multi-object datasets.<n>LPWM autonomously discovers keypoints, bounding boxes, and object masks directly from video data.<n>Our architecture is trained end-to-end purely from videos and supports flexible conditioning on actions, language, and image goals.
  arXiv  Detail & Related papers  (2026-03-04T19:36:08Z)
• World Simulation with Video Foundation Models for Physical AI [181.78265691303315]
  We release [Cosmos-Predict2.5] and [Cosmos-Transfer2.5] as versatile tools for scaling embodied intelligence.<n>We release source code, pretrained checkpoints, and curated benchmarks under the NVIDIA Open Model License.
  arXiv  Detail & Related papers  (2025-10-28T22:44:13Z)
• Yume: An Interactive World Generation Model [38.818537395166835]
  Yume aims to use images, text, or videos to create an interactive, realistic, and dynamic world.<n>Method creates a dynamic world from an input image and allows exploration of the world using keyboard actions.
  arXiv  Detail & Related papers  (2025-07-23T17:57:09Z)
• PoE-World: Compositional World Modeling with Products of Programmatic Experts [50.35012247866856]
  Learning how the world works is central to building AI agents that can adapt to complex environments.<n>Recent advances in program synthesis using Large Language Models (LLMs) give an alternate approach which learns world models represented as source code.<n>We show that this approach can learn complex world models from just a few observations. We evaluate the learned world models by embedding them in a model-based planning agent, demonstrating efficient performance and generalization to unseen levels on Atari's Pong and Montezuma's Revenge.
  arXiv  Detail & Related papers  (2025-05-16T03:28:42Z)
• Cosmos-Transfer1: Conditional World Generation with Adaptive Multimodal Control [97.98560001760126]
  We introduce Cosmos-Transfer, a conditional world generation model that can generate world simulations based on multiple spatial control inputs.<n>We conduct evaluations to analyze the proposed model and demonstrate its applications for Physical AI, including robotics2Real and autonomous vehicle data enrichment.
  arXiv  Detail & Related papers  (2025-03-18T17:57:54Z)
• Cosmos World Foundation Model Platform for Physical AI [136.1002343616157]
  We present the Cosmos World Foundation Model Platform to help developers build customized world models for their Physical AI setups.<n>Our platform covers a video curation pipeline, pre-trained world foundation models, examples of post-training of pre-trained world foundation models, and video tokenizers.
  arXiv  Detail & Related papers  (2025-01-07T06:55:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.