Do You Know Where Your Camera Is? View-Invariant Policy Learning with Camera Conditioning

• URL: http://arxiv.org/abs/2510.02268v1
• Date: Thu, 02 Oct 2025 17:47:06 GMT
• Title: Do You Know Where Your Camera Is? View-Invariant Policy Learning with Camera Conditioning
• Abstract summary: We study view-invariant imitation learning by explicitly conditioning policies on camera extrinsics.<n>We show that conditioning on extrinsics significantly improves generalization across viewpoints for standard behavior cloning policies.
• Score: 22.045823914539408
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We study view-invariant imitation learning by explicitly conditioning policies on camera extrinsics. Using Plucker embeddings of per-pixel rays, we show that conditioning on extrinsics significantly improves generalization across viewpoints for standard behavior cloning policies, including ACT, Diffusion Policy, and SmolVLA. To evaluate policy robustness under realistic viewpoint shifts, we introduce six manipulation tasks in RoboSuite and ManiSkill that pair "fixed" and "randomized" scene variants, decoupling background cues from camera pose. Our analysis reveals that policies without extrinsics often infer camera pose using visual cues from static backgrounds in fixed scenes; this shortcut collapses when workspace geometry or camera placement shifts. Conditioning on extrinsics restores performance and yields robust RGB-only control without depth. We release the tasks, demonstrations, and code at https://ripl.github.io/know_your_camera/ .

Related papers

• From Fixed to Free Cameras: Calibration-Free View-Robust Vision-Language-Action Model [72.8641426724502]
  Camera-Centric VLA (CamVLA) is a new VLA model that decouples manipulation controls from camera geometry.<n>CamVLA consistently improves success rates across diverse unseen viewpoints.
  arXiv  Detail & Related papers  (2026-07-06T17:59:59Z)
• CameraNoise: Enabling Faithful Camera Control in Video Diffusion through Geometry-Flow-Guided Noise Warping [81.58057192696843]
  Existing methods that directly inject numerical camera parameters into the diffusion backbone often fail to bridge the gap between abstract coordinates and visual content.<n>We propose CameraNoise, a flow-to-noise warping method that encodes camera motion into a temporally coherent representation.<n>By integrating CameraNoise into the diffusion process, our framework delivers stable, high-fidelity videos.
  arXiv  Detail & Related papers  (2026-05-29T03:02:50Z)
• Choose What to Observe: Task-Aware Semantic-Geometric Representations for Visuomotor Policy [66.60668908340429]
  Visuomotor policies learn from demonstrations often overfit to nuisance visual factors in raw RGB observations.<n>We propose a task-aware observation interface that canonicalizes visual input into a shared representation.<n>We evaluate on RoboMimic (Lift), ManiSkill YCB grasping under clutter, four RLBench tasks under controlled appearance shifts, and two real-world Franka tasks.
  arXiv  Detail & Related papers  (2026-03-09T01:21:26Z)
• AnyCamVLA: Zero-Shot Camera Adaptation for Viewpoint Robust Vision-Language-Action Models [33.20237572326132]
  We propose a zero-shot camera adaptation framework without additional demonstration data, policy fine-tuning, or architectural modification.<n>Our key idea is to virtually adjust test-time camera observations to match the training camera configuration in real-time.<n>This approach consistently outperforms baselines that use data augmentation for policy fine-tuning or additional 3D-aware features for visual input.
  arXiv  Detail & Related papers  (2026-03-06T03:44:23Z)
• ReRoPE: Repurposing RoPE for Relative Camera Control [36.225344172088235]
  We introduce ReRoPE, a plug-and-play framework that incorporates relative camera information into pre-trained video diffusion models.<n>We evaluate our method on both image-to-video (I2V) and video-to-video (V2V) tasks in terms of camera control accuracy and visual fidelity.
  arXiv  Detail & Related papers  (2026-02-08T17:49:10Z)
• CamPilot: Improving Camera Control in Video Diffusion Model with Efficient Camera Reward Feedback [43.174121093566264]
  We build upon Reward Feedback Learning and aim to further improve camera controllability.<n>Current reward models lack the capacity to assess video-camera alignment.<n>We introduce an efficient camera-aware 3D decoder that decodes video latent into 3D representations for reward quantization.
  arXiv  Detail & Related papers  (2026-01-22T18:59:56Z)
• OmniVCus: Feedforward Subject-driven Video Customization with Multimodal Control Conditions [77.04071342405055]
  We develop an Image-Video Transfer Mixed (IVTM) training with image editing data to enable instructive editing for the subject in the customized video.<n>We also propose a diffusion Transformer framework, OmniVCus, with two embedding mechanisms, Lottery Embedding (LE) and Temporally Aligned Embedding (TAE)<n>Our method significantly surpasses state-of-the-art methods in both quantitative and qualitative evaluations.
  arXiv  Detail & Related papers  (2025-06-29T18:43:00Z)
• ReCamMaster: Camera-Controlled Generative Rendering from A Single Video [72.42376733537925]
  ReCamMaster is a camera-controlled generative video re-rendering framework.<n>It reproduces the dynamic scene of an input video at novel camera trajectories.<n>Our method also finds promising applications in video stabilization, super-resolution, and outpainting.
  arXiv  Detail & Related papers  (2025-03-14T17:59:31Z)
• CamI2V: Camera-Controlled Image-to-Video Diffusion Model [11.762824216082508]
  Integrated camera pose is a user-friendly and physics-informed condition in video diffusion models, enabling precise camera control.<n>We identify one of the key challenges as effectively modeling noisy cross-frame interactions to enhance geometry consistency and camera controllability.<n>We innovatively associate the quality of a condition with its ability to reduce uncertainty and interpret noisy cross-frame features as a form of noisy condition.
  arXiv  Detail & Related papers  (2024-10-21T12:36:27Z)
• SRPose: Two-view Relative Pose Estimation with Sparse Keypoints [51.49105161103385]
  SRPose is a sparse keypoint-based framework for two-view relative pose estimation in camera-to-world and object-to-camera scenarios. It achieves competitive or superior performance compared to state-of-the-art methods in terms of accuracy and speed. It is robust to different image sizes and camera intrinsics, and can be deployed with low computing resources.
  arXiv  Detail & Related papers  (2024-07-11T05:46:35Z)
• CameraCtrl: Enabling Camera Control for Text-to-Video Generation [86.36135895375425]
  Controllability plays a crucial role in video generation, as it allows users to create and edit content more precisely.<n>Existing models, however, lack control of camera pose that serves as a cinematic language to express deeper narrative nuances.<n>We introduce CameraCtrl, enabling accurate camera pose control for video diffusion models.
  arXiv  Detail & Related papers  (2024-04-02T16:52:41Z)
• Learning Generalizable Manipulation Policies with Object-Centric 3D Representations [65.55352131167213]
  GROOT is an imitation learning method for learning robust policies with object-centric and 3D priors. It builds policies that generalize beyond their initial training conditions for vision-based manipulation. GROOT's performance excels in generalization over background changes, camera viewpoint shifts, and the presence of new object instances.
  arXiv  Detail & Related papers  (2023-10-22T18:51:45Z)
• Robust Visual Sim-to-Real Transfer for Robotic Manipulation [79.66851068682779]
  Learning visuomotor policies in simulation is much safer and cheaper than in the real world. However, due to discrepancies between the simulated and real data, simulator-trained policies often fail when transferred to real robots. One common approach to bridge the visual sim-to-real domain gap is domain randomization (DR)
  arXiv  Detail & Related papers  (2023-07-28T05:47:24Z)
• The Treachery of Images: Bayesian Scene Keypoints for Deep Policy Learning in Robotic Manipulation [28.30126109684119]
  We present BASK, a Bayesian approach to tracking scale-invariant keypoints over time. We employ our method to learn challenging multi-object robot manipulation tasks from wrist camera observations.
  arXiv  Detail & Related papers  (2023-05-08T14:05:38Z)
• Pose-Controllable Talking Face Generation by Implicitly Modularized Audio-Visual Representation [96.66010515343106]
  We propose a clean yet effective framework to generate pose-controllable talking faces. We operate on raw face images, using only a single photo as an identity reference. Our model has multiple advanced capabilities including extreme view robustness and talking face frontalization.
  arXiv  Detail & Related papers  (2021-04-22T15:10:26Z)
• Wide-Baseline Multi-Camera Calibration using Person Re-Identification [27.965850489928457]
  We address the problem of estimating the 3D pose of a network of cameras for large-environment wide-baseline scenarios. Treating people in the scene as "keypoints" and associating them across different camera views can be an alternative method for obtaining correspondences. Our method first employs a re-ID method to associate human bounding boxes across cameras, then converts bounding box correspondences to point correspondences.
  arXiv  Detail & Related papers  (2021-04-17T15:09:18Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.