Rays as Pixels: Learning A Joint Distribution of Videos and Camera Trajectories

• URL: http://arxiv.org/abs/2604.09429v1
• Date: Fri, 10 Apr 2026 15:47:23 GMT
• Title: Rays as Pixels: Learning A Joint Distribution of Videos and Camera Trajectories
• Abstract summary: Rays as Pixels is a Video Diffusion Model that learns a joint distribution over videos and camera trajectories.<n>A single trained model handles three tasks: predicting camera trajectories from video, jointly generating video and camera trajectory from input images, and generating video from input images along a target camera trajectory.<n>We report results on pose estimation and camera-controlled video generation.
• Score: 25.30154413418057
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Recovering camera parameters from images and rendering scenes from novel viewpoints have long been treated as separate tasks in computer vision and graphics. This separation breaks down when image coverage is sparse or poses are ambiguous, since each task needs what the other produces. We propose Rays as Pixels, a Video Diffusion Model (VDM) that learns a joint distribution over videos and camera trajectories. We represent each camera as dense ray pixels (raxels) and denoise them jointly with video frames through Decoupled Self-Cross Attention mechanism. A single trained model handles three tasks: predicting camera trajectories from video, jointly generating video and camera trajectory from input images, and generating video from input images along a target camera trajectory. Because the model can both predict trajectories from a video and generate views conditioned on its own predictions, we evaluate it through a closed-loop self-consistency test, demonstrating that its forward and inverse predictions agree. Notably, trajectory prediction requires far fewer denoising steps than video generation, even a few denoising steps suffice for self-consistency. We report results on pose estimation and camera-controlled video generation.

Related papers

• Track2View: 4D-Consistent Camera-Controlled Video Generation via Paired 3D Point Tracks [18.57720960197565]
  Existing methods rely on per-frame embeddings, noisy point-cloud renderings, or implicit learned correspondences.<n>We propose Track2View, which conditions a video diffusion transformer on paired 3D point tracks.<n>At the core of Track2View is a dual-view track conditioner that transfers visual context from source to target view.
  arXiv  Detail & Related papers  (2026-06-14T01:23:50Z)
• Geo-Align: Video Generation Alignment via Metric Geometry Reward [53.65904111864641]
  Geo-Align is the first Reinforcement Learning framework specifically designed for camera-controlled video re-rendering.<n>Built upon a pretrained model, we optimize the model through a scale-aware perceptual reward mechanism.<n>Experiments demonstrate that Geo-Align consistently outperforms existing supervised learning baselines in both precise camera controllability and visual fidelity.
  arXiv  Detail & Related papers  (2026-05-22T17:59:43Z)
• Infinite-Homography as Robust Conditioning for Camera-Controlled Video Generation [49.12018869332346]
  InfCam is a camera-controlled video-to-video generation framework with high pose fidelity.<n>The framework integrates two key components: (1) infinite homography warping, which encodes 3D camera rotations directly within the 2D latent space of a video diffusion model.
  arXiv  Detail & Related papers  (2025-12-18T20:03:05Z)
• Direct Motion Models for Assessing Generated Videos [38.04485796547767]
  A current limitation of video generative video models is that they generate plausible looking frames, but poor motion.<n>Here we go beyond FVD by developing a metric which better measures plausible object interactions and motion.<n>We show that using point tracks instead of pixel reconstruction or action recognition features results in a metric which is markedly more sensitive to temporal distortions in synthetic data.
  arXiv  Detail & Related papers  (2025-04-30T22:34:52Z)
• ReCamMaster: Camera-Controlled Generative Rendering from A Single Video [72.42376733537925]
  ReCamMaster is a camera-controlled generative video re-rendering framework.<n>It reproduces the dynamic scene of an input video at novel camera trajectories.<n>Our method also finds promising applications in video stabilization, super-resolution, and outpainting.
  arXiv  Detail & Related papers  (2025-03-14T17:59:31Z)
• Latent-Reframe: Enabling Camera Control for Video Diffusion Model without Training [51.851390459940646]
  We introduce Latent-Reframe, which enables camera control in a pre-trained video diffusion model without fine-tuning.<n>Latent-Reframe operates during the sampling stage, maintaining efficiency while preserving the original model distribution.<n>Our approach reframes the latent code of video frames to align with the input camera trajectory through time-aware point clouds.
  arXiv  Detail & Related papers  (2024-12-08T18:59:54Z)
• Generating 3D-Consistent Videos from Unposed Internet Photos [68.944029293283]
  We train a scalable, 3D-aware video model without any 3D annotations such as camera parameters. Our results suggest that we can scale up scene-level 3D learning using only 2D data such as videos and multiview internet photos.
  arXiv  Detail & Related papers  (2024-11-20T18:58:31Z)
• A Semi-Self-Supervised Approach for Dense-Pattern Video Object Segmentation [6.092973123903838]
  We propose a semi-self-supervised approach for dense-VOS (DVOS) using a diffusion-based method through multi-task learning.<n>We evaluate our DVOS method for wheat head segmentation from a diverse set of videos, spanning from Boot-stage to Wheat-mature and Harvest-ready.<n>While our method was evaluated on wheat head segmentation, it can be extended to other crops and domains, such as crowd analysis or microscopic image analysis.
  arXiv  Detail & Related papers  (2024-06-07T17:58:36Z)
• Collaborative Video Diffusion: Consistent Multi-video Generation with Camera Control [70.17137528953953]
  Collaborative video diffusion (CVD) is trained on top of a state-of-the-art camera-control module for video generation. CVD generates multiple videos rendered from different camera trajectories with significantly better consistency than baselines.
  arXiv  Detail & Related papers  (2024-05-27T17:58:01Z)
• Lift, Splat, Shoot: Encoding Images From Arbitrary Camera Rigs by Implicitly Unprojecting to 3D [100.93808824091258]
  We propose a new end-to-end architecture that directly extracts a bird's-eye-view representation of a scene given image data from an arbitrary number of cameras. Our approach is to "lift" each image individually into a frustum of features for each camera, then "splat" all frustums into a bird's-eye-view grid. We show that the representations inferred by our model enable interpretable end-to-end motion planning by "shooting" template trajectories into a bird's-eye-view cost map output by our network.
  arXiv  Detail & Related papers  (2020-08-13T06:29:01Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.