OmniCam: Unified Multimodal Video Generation via Camera Control

• URL: http://arxiv.org/abs/2504.02312v1
• Date: Thu, 03 Apr 2025 06:38:30 GMT
• Title: OmniCam: Unified Multimodal Video Generation via Camera Control
• Abstract summary: Camera control which achieves diverse visual effects by changing camera position and pose has attracted widespread attention.<n>Existing methods face challenges such as complex interaction and limited control capabilities.<n>We present OmniCam, a unified camera framework that generates guidance-temporally consistent videos.
• Score: 42.94206239207397
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Camera control, which achieves diverse visual effects by changing camera position and pose, has attracted widespread attention. However, existing methods face challenges such as complex interaction and limited control capabilities. To address these issues, we present OmniCam, a unified multimodal camera control framework. Leveraging large language models and video diffusion models, OmniCam generates spatio-temporally consistent videos. It supports various combinations of input modalities: the user can provide text or video with expected trajectory as camera path guidance, and image or video as content reference, enabling precise control over camera motion. To facilitate the training of OmniCam, we introduce the OmniTr dataset, which contains a large collection of high-quality long-sequence trajectories, videos, and corresponding descriptions. Experimental results demonstrate that our model achieves state-of-the-art performance in high-quality camera-controlled video generation across various metrics.

Related papers

• TriMotion: Modality-Agnostic Camera Control for Video Generation [65.5913929253637]
  TriMotion is a framework for camera-controlled video generation that maps video, pose, and text inputs, describing the same camera trajectory into a shared motion embedding space.<n>We show that TriMotion generates high-quality videos that accurately follow the target camera trajectories across all three modalities.
  arXiv  Detail & Related papers  (2026-06-18T16:07:05Z)
• OmniDirector: General Multi-Shot Camera Cloning without Cross-Paired Data [47.59993927255196]
  We introduce a general camera motion representation that encodes cameras as grid motion videos.<n>This camera grid represents the camera parameters visually and supports the integration of diverse trajectories for multi-shot video generation.<n>We propose OmniDirector, a unified framework trained on a million-scale camera grid-video pairs.
  arXiv  Detail & Related papers  (2026-06-11T14:58:28Z)
• OmniCamera: A Unified Framework for Multi-task Video Generation with Arbitrary Camera Control [49.41924736941193]
  Video fundamentally intertwines two crucial axes: the dynamic content of a scene and the camera motion through which it is observed.<n>Existing generation models often entangle these factors, limiting independent control.<n>We introduce OmniCamera, a unified framework designed to explicitly disentangle and command these two dimensions.
  arXiv  Detail & Related papers  (2026-04-07T16:06:02Z)
• FaceCam: Portrait Video Camera Control via Scale-Aware Conditioning [45.013802909442184]
  We introduce FaceCam, a system that generates video under customizable camera trajectories for monocular human portrait video input.<n> Experiments on Ava-256 dataset and diverse in-the-wild videos demonstrate that FaceCam achieves superior performance in camera controllability, visual quality, identity and motion preservation.
  arXiv  Detail & Related papers  (2026-03-05T18:59:58Z)
• CETCAM: Camera-Controllable Video Generation via Consistent and Extensible Tokenization [32.42754288735215]
  CETCAM is a camera-controllable video generation framework.<n>It eliminates the need for camera annotations through a consistent and tokenization scheme.<n>It learns robust camera controllability from diverse raw video data and refines fine-grained visual quality using high-fidelity datasets.
  arXiv  Detail & Related papers  (2025-12-22T04:21:39Z)
• CamCloneMaster: Enabling Reference-based Camera Control for Video Generation [39.68297612349062]
  CamCloneMaster is a framework that enables users to replicate camera movements from reference videos without requiring camera parameters or test-time fine-tuning.<n>We present a large-scale synthetic dataset designed for camera clone learning, encompassing diverse scenes, subjects, and camera movements.
  arXiv  Detail & Related papers  (2025-06-03T17:59:02Z)
• Modular-Cam: Modular Dynamic Camera-view Video Generation with LLM [43.889033468684445]
  We propose a novel text-to-video generation method, i.e., Modular-Cam. To better understand a given complex prompt, we utilize a large language model to analyze user instructions. To generate a video containing dynamic scenes that match the given camera-views, we incorporate the widely-used temporal transformer.
  arXiv  Detail & Related papers  (2025-04-16T13:04:01Z)
• Free-Form Motion Control: A Synthetic Video Generation Dataset with Controllable Camera and Object Motions [78.65431951506152]
  We introduce a Synthetic dataset for Free-Form Motion Control (SynFMC)<n>The proposed SynFMC dataset includes diverse objects and environments and covers various motion patterns according to specific rules.<n>We further propose a method, Free-Form Motion Control (FMC), which enables independent or simultaneous control of object and camera movements.
  arXiv  Detail & Related papers  (2025-01-02T18:59:45Z)
• Cavia: Camera-controllable Multi-view Video Diffusion with View-Integrated Attention [62.2447324481159]
  Cavia is a novel framework for camera-controllable, multi-view video generation. Our framework extends the spatial and temporal attention modules, improving both viewpoint and temporal consistency. Cavia is the first of its kind that allows the user to specify distinct camera motion while obtaining object motion.
  arXiv  Detail & Related papers  (2024-10-14T17:46:32Z)
• VD3D: Taming Large Video Diffusion Transformers for 3D Camera Control [74.5434726968562]
  We show how to tame transformers video for 3D camera control using a ControlNet-like conditioning mechanism. Our work is the first to enable camera control for transformer-based video diffusion models.
  arXiv  Detail & Related papers  (2024-07-17T17:59:05Z)
• Training-free Camera Control for Video Generation [15.79168688275606]
  We propose a training-free and robust solution to offer camera movement control for off-the-shelf video diffusion models.<n>Our method does not require any supervised finetuning on camera-annotated datasets or self-supervised training via data augmentation.<n>It can be plug-and-play with most pretrained video diffusion models and generate camera-controllable videos with a single image or text prompt as input.
  arXiv  Detail & Related papers  (2024-06-14T15:33:00Z)
• Collaborative Video Diffusion: Consistent Multi-video Generation with Camera Control [70.17137528953953]
  Collaborative video diffusion (CVD) is trained on top of a state-of-the-art camera-control module for video generation. CVD generates multiple videos rendered from different camera trajectories with significantly better consistency than baselines.
  arXiv  Detail & Related papers  (2024-05-27T17:58:01Z)
• MotionMaster: Training-free Camera Motion Transfer For Video Generation [48.706578330771386]
  We propose a novel training-free video motion transfer model, which disentangles camera motions and object motions in source videos. Our model can effectively decouple camera-object motion and apply the decoupled camera motion to a wide range of controllable video generation tasks.
  arXiv  Detail & Related papers  (2024-04-24T10:28:54Z)
• CameraCtrl: Enabling Camera Control for Text-to-Video Generation [86.36135895375425]
  Controllability plays a crucial role in video generation, as it allows users to create and edit content more precisely.<n>Existing models, however, lack control of camera pose that serves as a cinematic language to express deeper narrative nuances.<n>We introduce CameraCtrl, enabling accurate camera pose control for video diffusion models.
  arXiv  Detail & Related papers  (2024-04-02T16:52:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.