Render, Don't Decode: Weight-Space World Models with Latent Structural Disentanglement

• URL: http://arxiv.org/abs/2605.06298v2
• Date: Fri, 08 May 2026 08:40:29 GMT
• Title: Render, Don't Decode: Weight-Space World Models with Latent Structural Disentanglement
• Abstract summary: We introduce NOVA, a world modelling framework that represents the system state as the weights and biases of an auxiliary coordinate-based implicit neural representation (INR)<n>This structured representation is analytically rendered, which eliminates the decoder bottleneck while conferring compactness, portability, and zero-shot super-resolution.<n>We validate our framework on several challenging datasets, achieving strong controllable forecasting while operating on a single consumer GPU at $sim$40M parameters.
• Score: 2.931556701508956
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Training world models on vast quantities of unlabelled videos is a critical step toward fully autonomous intelligence. However, the prevailing paradigm of encoding raw pixels into opaque latent spaces and relying on heavy decoders for reconstruction leaves these models computationally expensive and uninterpretable. We address this problem by introducing NOVA, a world modelling framework that represents the system state as the weights and biases of an auxiliary coordinate-based implicit neural representation (INR). This structured representation is analytically rendered, which eliminates the decoder bottleneck while conferring compactness, portability, and zero-shot super-resolution. Furthermore, like most latent action models, NOVA can be distilled into a context-dependent video generator via an action-matching objective. Surprisingly, without resorting to auxiliary losses or adversarial objectives, NOVA can disentangle structural scene components such as background, foreground, and inter-frame motion, enabling users to edit either content or dynamics without compromising the other. We validate our framework on several challenging datasets, achieving strong controllable forecasting while operating on a single consumer GPU at $\sim$40M parameters. Ultimately, structured representations like INRs not only enhance our understanding of latent dynamics but also pave the way for immersive and customisable virtual experiences.

Related papers

• Dynamic-Robust Photometric-Semantic Reconstruction for Open-Vocabulary 3D Scene Understanding [63.599342750566485]
  SPAR is a novel joint semantic-geometric encoding architecture that explicitly isolates transient dynamic noise prior to latent space aggregation.<n>Our end-to-end approach achieves exceptional novel view synthesis quality, yielding a PSNR of 22.15 dB and 23.33 dB given only 3 and 4 input views respectively.
  arXiv  Detail & Related papers  (2026-08-29T09:58:24Z)
• An active-learning framework for real-time depth perception from monocular vision streams [24.813023779263645]
  An Online Active Learning mechanism is introduced to endow compact neural networks with the capability to adapt continuously during operation.<n>A closed-loop Predict-Evaluate-Correct learning paradigm is established to actively select high-confidence, information-rich signals from streaming visual input.<n>The proposed system achieves approximately a 75% reduction in computational cost while maintaining competitive depth estimation accuracy.
  arXiv  Detail & Related papers  (2026-08-05T14:45:16Z)
• RayDer: Scalable Self-Supervised Novel View Synthesis from Real-World Video [24.21520175195074]
  RayDer is a feed-forward transformer that consolidates camera estimation, scene reconstruction, and rendering into a single backbone.<n>A minimal dynamic state, treated as a nuisance factor, absorbs time-varying content and enables stable training on real-world video.<n>RayDer exhibits clean power-law scaling with data and compute, and outperforms static-scene data mixtures.
  arXiv  Detail & Related papers  (2026-05-29T16:50:27Z)
• QVAD: A Question-Centric Agentic Framework for Efficient and Training-Free Video Anomaly Detection [11.652235163237117]
  Video Anomaly Detection (VAD) is a fundamental challenge in computer vision.<n>We propose QVAD, a question-centric agentic framework that treats VLM-LLM interaction as a dynamic dialogue.<n>QVAD achieves high inference speeds with minimal memory footprints, making advanced VAD capabilities deployable on resource-constrained edge devices.
  arXiv  Detail & Related papers  (2026-04-03T13:48:34Z)
• Motion Forcing: A Decoupled Framework for Robust Video Generation in Motion Dynamics [37.22501359080204]
  We introduce textbfMotion Forcing, a framework designed to stabilize a video generation equilibrium.<n>Our key insight is to explicitly decouple physical reasoning from visual synthesis.<n>Experiments on autonomous driving benchmarks show that Motion Forcing significantly outperforms state-of-the-art baselines.
  arXiv  Detail & Related papers  (2026-03-11T04:44:46Z)
• Infinite-World: Scaling Interactive World Models to 1000-Frame Horizons via Pose-Free Hierarchical Memory [101.2076718776139]
  We propose a robust interactive world model capable of maintaining coherent visual memory over 1000+ frames in complex real-world environments.<n>We introduce a Pose-free Memory (HPMC) that distills historical latents into a fixed-budget geometric representation.<n>We also propose an Uncertainty-aware Action Labeling module that discretizes continuous motion into a tri-state logic.
  arXiv  Detail & Related papers  (2026-02-02T17:52:56Z)
• TriFusion-AE: Language-Guided Depth and LiDAR Fusion for Robust Point Cloud Processing [0.0]
  Autoencoders offer a natural framework for denoising and reconstruction, but their performance degrades under challenging real-world conditions.<n>We propose TriFusion-AE, a cross-attention autoencoder that integrates textual priors, monocular depth maps from multi-view images, and LiDAR point clouds to improve robustness.<n>Our model achieves significantly more robust reconstruction under strong adversarial attacks and heavy noise, where CNN-based autoencoders collapse.
  arXiv  Detail & Related papers  (2025-09-23T07:37:28Z)
• HoliGS: Holistic Gaussian Splatting for Embodied View Synthesis [59.25751939710903]
  We propose a novel deformable Gaussian splatting framework that addresses embodied view synthesis from long monocular RGB videos.<n>Our method leverages invertible Gaussian Splatting deformation networks to reconstruct large-scale, dynamic environments accurately.<n>Results highlight a practical and scalable solution for EVS in real-world scenarios.
  arXiv  Detail & Related papers  (2025-06-24T03:54:40Z)
• Sparse identification of nonlinear dynamics and Koopman operators with Shallow Recurrent Decoder Networks [3.1484174280822845]
  We present a method to jointly solve the sensing and model identification problems with simple implementation, efficient, and robust performance.<n>SINDy-SHRED uses Gated Recurrent Units to model sparse sensor measurements along with a shallow network decoder to reconstruct the full-temporal field from the latent state space.<n>We conduct systematic experimental studies on PDE data such as turbulent flows, real-world sensor measurements for sea surface temperature, and direct video data.
  arXiv  Detail & Related papers  (2025-01-23T02:18:13Z)
• Distance Weighted Trans Network for Image Completion [52.318730994423106]
  We propose a new architecture that relies on Distance-based Weighted Transformer (DWT) to better understand the relationships between an image's components. CNNs are used to augment the local texture information of coarse priors. DWT blocks are used to recover certain coarse textures and coherent visual structures.
  arXiv  Detail & Related papers  (2023-10-11T12:46:11Z)
• DynIBaR: Neural Dynamic Image-Based Rendering [79.44655794967741]
  We address the problem of synthesizing novel views from a monocular video depicting a complex dynamic scene. We adopt a volumetric image-based rendering framework that synthesizes new viewpoints by aggregating features from nearby views. We demonstrate significant improvements over state-of-the-art methods on dynamic scene datasets.
  arXiv  Detail & Related papers  (2022-11-20T20:57:02Z)
• RISP: Rendering-Invariant State Predictor with Differentiable Simulation and Rendering for Cross-Domain Parameter Estimation [110.4255414234771]
  Existing solutions require massive training data or lack generalizability to unknown rendering configurations. We propose a novel approach that marries domain randomization and differentiable rendering gradients to address this problem. Our approach achieves significantly lower reconstruction errors and has better generalizability among unknown rendering configurations.
  arXiv  Detail & Related papers  (2022-05-11T17:59:51Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.