Geometry-Guided Camera Motion Understanding in VideoLLMs

• URL: http://arxiv.org/abs/2603.13119v1
• Date: Fri, 13 Mar 2026 16:13:09 GMT
• Title: Geometry-Guided Camera Motion Understanding in VideoLLMs
• Abstract summary: Camera motion is a fundamental geometric signal that shapes visual perception and cinematic style.<n>Current video-capable vision-injection models (VideoLLMs) rarely represent it explicitly and often fail on fine-grained motion primitives.<n>We formulate camera motion as constraint-aware multi-label recognition.<n> Experiments demonstrate improved motion recognition and more camera-aware model responses.
• Score: 8.928448861727274
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Camera motion is a fundamental geometric signal that shapes visual perception and cinematic style, yet current video-capable vision-language models (VideoLLMs) rarely represent it explicitly and often fail on fine-grained motion primitives. We address this gap with a framework of $\textbf{benchmarking}$, $\textbf{diagnosis}$, and $\textbf{injection}$. We curate $\textbf{CameraMotionDataset}$, a large-scale synthetic dataset with explicit camera control, formulate camera motion as constraint-aware multi-label recognition, and construct a VQA benchmark--$\textbf{CameraMotionVQA}$. Across diverse off-the-shelf VideoLLMs, we observe substantial errors in recognizing camera motion primitives. Probing experiments on a Qwen2.5-VL vision encoder suggest that camera motion cues are weakly represented, especially in deeper ViT blocks, helping explain the observed failure modes. To bridge this gap without costly training or fine-tuning, we propose a lightweight, model-agnostic pipeline that extracts geometric camera cues from 3D foundation models (3DFMs), predicts constrained motion primitives with a temporal classifier, and injects them into downstream VideoLLM inference via structured prompting. Experiments demonstrate improved motion recognition and more camera-aware model responses, highlighting geometry-driven cue extraction and structured prompting as practical steps toward a camera-aware VideoLLM and VLA system. The dataset and benchmark is publicly available at https://hf.co/datasets/fengyee/camera-motion-dataset-and-benchmark.

Related papers

• UniMoCa: Unifying Motion and Camera Controls as Visual Proxies for Faithful Human Video Generation [50.391925716274926]
  textbfUniMoCa is a representation-driven framework that unifies motion and camera controls in visual space.<n>We show that UniMoCa achieves substantial gains in human motion control, camera control, temporal consistency, and camera-aware robustness with minimal additional complexity.
  arXiv  Detail & Related papers  (2026-08-03T09:18:13Z)
• MVTrack4Gen: Multi-View Point Tracking as Geometric Supervision for 4D Video Generation [50.65653874052261]
  We introduce MVTrack4Gen, a motion-aware training framework for camera-conditioning-only novel-view video diffusion models.<n>Our key finding is that specific attention layers encode strong correspondence cues, where query features attend to key features at geometrically corresponding locations.<n>By explicitly strengthening these motion-aware correspondences, MVTrack4Gen improves existing models to better follow the motion in the reference view and maintain cross-view geometric consistency.
  arXiv  Detail & Related papers  (2026-06-24T17:56:33Z)
• Probing into Camera Control of Video Models [42.06310116603546]
  We show that camera control need not be modeled as an implicit mapping problem, but can instead be treated as a form of geometric guidance.<n>We reformulate camera control into a set of displacement fields and apply them via differentiable resampling of latent features during denoising.<n>Our simple approach achieves effective camera control with minimal degradation across diverse quality metrics compared to fine-tuned baselines.
  arXiv  Detail & Related papers  (2026-05-14T13:27:33Z)
• CamPilot: Improving Camera Control in Video Diffusion Model with Efficient Camera Reward Feedback [43.174121093566264]
  We build upon Reward Feedback Learning and aim to further improve camera controllability.<n>Current reward models lack the capacity to assess video-camera alignment.<n>We introduce an efficient camera-aware 3D decoder that decodes video latent into 3D representations for reward quantization.
  arXiv  Detail & Related papers  (2026-01-22T18:59:56Z)
• Pixel-to-4D: Camera-Controlled Image-to-Video Generation with Dynamic 3D Gaussians [7.051077403685518]
  Humans excel at forecasting the future dynamics of a scene given just a single image.<n>Video generation models that can mimic this ability are an essential component for intelligent systems.<n>Recent approaches have improved temporal coherence and 3D consistency in single-image-conditioned video generation.
  arXiv  Detail & Related papers  (2026-01-02T13:04:47Z)
• CETCAM: Camera-Controllable Video Generation via Consistent and Extensible Tokenization [32.42754288735215]
  CETCAM is a camera-controllable video generation framework.<n>It eliminates the need for camera annotations through a consistent and tokenization scheme.<n>It learns robust camera controllability from diverse raw video data and refines fine-grained visual quality using high-fidelity datasets.
  arXiv  Detail & Related papers  (2025-12-22T04:21:39Z)
• Infinite-Homography as Robust Conditioning for Camera-Controlled Video Generation [49.12018869332346]
  InfCam is a camera-controlled video-to-video generation framework with high pose fidelity.<n>The framework integrates two key components: (1) infinite homography warping, which encodes 3D camera rotations directly within the 2D latent space of a video diffusion model.
  arXiv  Detail & Related papers  (2025-12-18T20:03:05Z)
• Estimating 2D Camera Motion with Hybrid Motion Basis [45.971928868591334]
  CamFlow is a novel framework that represents camera motion using hybrid motion bases.<n>Our approach includes a hybrid probabilistic loss function based on the Laplace distribution.<n>Experiments show CamFlow outperforms state-of-the-art methods across diverse scenarios.
  arXiv  Detail & Related papers  (2025-07-30T08:30:37Z)
• MegaSaM: Accurate, Fast, and Robust Structure and Motion from Casual Dynamic Videos [104.1338295060383]
  We present a system that allows for accurate, fast, and robust estimation of camera parameters and depth maps from casual monocular videos of dynamic scenes.<n>Our system is significantly more accurate and robust at camera pose and depth estimation when compared with prior and concurrent work.
  arXiv  Detail & Related papers  (2024-12-05T18:59:42Z)
• AC3D: Analyzing and Improving 3D Camera Control in Video Diffusion Transformers [66.29824750770389]
  We analyze camera motion from a first principles perspective, uncovering insights that enable precise 3D camera manipulation.<n>We compound these findings to design the Advanced 3D Camera Control (AC3D) architecture.
  arXiv  Detail & Related papers  (2024-11-27T18:49:13Z)
• CamI2V: Camera-Controlled Image-to-Video Diffusion Model [11.762824216082508]
  Integrated camera pose is a user-friendly and physics-informed condition in video diffusion models, enabling precise camera control.<n>We identify one of the key challenges as effectively modeling noisy cross-frame interactions to enhance geometry consistency and camera controllability.<n>We innovatively associate the quality of a condition with its ability to reduce uncertainty and interpret noisy cross-frame features as a form of noisy condition.
  arXiv  Detail & Related papers  (2024-10-21T12:36:27Z)
• InstMove: Instance Motion for Object-centric Video Segmentation [70.16915119724757]
  In this work, we study the instance-level motion and present InstMove, which stands for Instance Motion for Object-centric Video. In comparison to pixel-wise motion, InstMove mainly relies on instance-level motion information that is free from image feature embeddings. With only a few lines of code, InstMove can be integrated into current SOTA methods for three different video segmentation tasks.
  arXiv  Detail & Related papers  (2023-03-14T17:58:44Z)
• FLEX: Parameter-free Multi-view 3D Human Motion Reconstruction [70.09086274139504]
  Multi-view algorithms strongly depend on camera parameters, in particular, the relative positions among the cameras. We introduce FLEX, an end-to-end parameter-free multi-view model. We demonstrate results on the Human3.6M and KTH Multi-view Football II datasets.
  arXiv  Detail & Related papers  (2021-05-05T09:08:12Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.