Puffin-World: Scaling a Unified Multimodal Model with Native 3D World States
- URL: http://arxiv.org/abs/2609.04196v1
- Date: Thu, 03 Sep 2026 17:59:13 GMT
- Title: Puffin-World: Scaling a Unified Multimodal Model with Native 3D World States
- Abstract summary: Puffin-World integrates physical understanding, spatial simulation, and 3D world generation and reconstruction without relying on external offline modules.<n>Our framework jointly models three native world states: physics (gravity field and latitude), geometry (depth), and appearance (image)<n>By grounding absolute camera properties in the real world, Puffin-World enables physically consistent and visually stable world generation.
- Score: 106.40454498751295
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: We propose Puffin-World, a unified multimodal architecture that integrates physical understanding, spatial simulation, and 3D world generation and reconstruction without relying on external offline modules. To reliably construct and interact with 3D worlds, our framework jointly models three native world states: physics (gravity field and latitude), geometry (depth), and appearance (image), together with a unified Omni-Camera representation that supports diverse tasks and flexible motions. Beyond modeling these states, we introduce a strategy for propagating physical dynamics across future frames. By grounding absolute camera properties in the real world, Puffin-World enables physically consistent and visually stable world generation. We further couple appearance and geometry within a single generative process, jointly synthesizing each future view and reconstructing its underlying geometry. This unified paradigm enables interleaved closed-loop applications requiring synergy across multiple tasks, including mimic and self-calibrated world exploration. To scale Puffin-World to complex scenarios, we construct Puffin-16M, comprising 15 million vision-language-camera triplets and 1 million trajectories featuring various and challenging motions. To foster further research in this area, we released the code, models, and datasets.
Related papers
- PAIWorld: A 3D-Consistent World Foundation Model for Robotic Manipulation [51.385664546670284]
We present PAIWorld, a framework that augments diffusion-transformer world models via three core components.<n>It achieves state-of-the-art multi-view 3D consistency on robotic benchmarks, ranking 1st on the WorldArena leaderboard and 2nd on the AgiBot-Challenge2026 leaderboard.
arXiv Detail & Related papers (2026-06-16T18:23:23Z) - MetaWorld: Scaling Multi-Agent Video World Model from Single-view Video Data [125.43597497646444]
MetaWorld is a novel framework that scales multi-agent video world models to open-domain environments directly from single-view videos.<n>It achieves superior cross-view consistency and identity fidelity, establishing a highly scalable, physics-driven paradigm for multi-agent video world modeling.
arXiv Detail & Related papers (2026-06-01T18:20:20Z) - HY-World 2.0: A Multi-Modal World Model for Reconstructing, Generating, and Simulating 3D Worlds [50.80574129942699]
HY-World 2.0 is a multi-modal world model framework that advances our prior project HYWorld 1.0.<n>We introduce key innovations to enhance panorama fidelity, enable 3D scene understanding and planning, and upgrade WorldStereo.<n>Also, we introduce WorldLens, a high-performance 3DGS rendering platform featuring a flexible engine-agnostic architecture.
arXiv Detail & Related papers (2026-04-15T17:59:17Z) - Beyond Pixel Histories: World Models with Persistent 3D State [50.4601060508243]
PERSIST is a new paradigm of world model which simulates the evolution of a latent 3D scene.<n>We show substantial improvements in spatial memory, 3D consistency, and long-horizon stability over existing methods.
arXiv Detail & Related papers (2026-03-03T19:58:31Z) - DynamicVerse: A Physically-Aware Multimodal Framework for 4D World Modeling [67.95038177144554]
We introduce DynamicVerse, a physical-scale, multimodal 4D world modeling framework for dynamic real-world video.<n>We employ vision, geometric, and multimodal models to interpret metric-scale static geometry, real-world dynamic motion, instance-level masks, and holistic captions.<n> DynamicVerse delivers a large-scale dataset consisting of 100K+ videos with 800K+ annotated masks and 10M+ frames from internet videos.
arXiv Detail & Related papers (2025-12-02T18:24:27Z) - Simulating the Visual World with Artificial Intelligence: A Roadmap [48.64639618440864]
Video generation is shifting from generating visually appealing clips to building virtual environments that support interaction and maintain physical plausibility.<n>This survey provides a systematic overview of this evolution, conceptualizing modern video foundation models as the combination of two core components.<n>We trace the progression of video generation through four generations, culminating in a video generation model that embodies intrinsic physical plausibility.
arXiv Detail & Related papers (2025-11-11T18:59:50Z) - HunyuanWorld 1.0: Generating Immersive, Explorable, and Interactive 3D Worlds from Words or Pixels [30.986527559921335]
HunyuanWorld 1.0 is a novel framework that combines the best of both worlds for generating immersive, explorable, and interactive 3D scenes from text and image conditions.<n>Our approach features three key advantages: 1) 360deg immersive experiences via panoramic world proxies; 2) mesh export capabilities for seamless compatibility with existing computer graphics pipelines; 3) disentangled object representations for augmented interactivity.
arXiv Detail & Related papers (2025-07-29T13:43:35Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.