DiffusionCinema: Text-to-Aerial Cinematography

• URL: http://arxiv.org/abs/2601.17412v1
• Date: Sat, 24 Jan 2026 11:12:30 GMT
• Title: DiffusionCinema: Text-to-Aerial Cinematography
• Abstract summary: Instead of manually piloting the drone, the user simply describes the desired shot.<n>Our system encodes along with an initial visual snapshot from the onboard camera.<n>The generated flight trajectory is then executed autonomously by the UAV.
• Score: 2.186077977059593
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We propose a novel Unmanned Aerial Vehicles (UAV) assisted creative capture system that leverages diffusion models to interpret high-level natural language prompts and automatically generate optimal flight trajectories for cinematic video recording. Instead of manually piloting the drone, the user simply describes the desired shot (e.g., "orbit around me slowly from the right and reveal the background waterfall"). Our system encodes the prompt along with an initial visual snapshot from the onboard camera, and a diffusion model samples plausible spatio-temporal motion plans that satisfy both the scene geometry and shot semantics. The generated flight trajectory is then executed autonomously by the UAV to record smooth, repeatable video clips that match the prompt. User evaluation using NASA-TLX showed a significantly lower overall workload with our interface (M = 21.6) compared to a traditional remote controller (M = 58.1), demonstrating a substantial reduction in perceived effort. Mental demand (M = 11.5 vs. 60.5) and frustration (M = 14.0 vs. 54.5) were also markedly lower for our system, confirming clear usability advantages in autonomous text-driven flight control. This project demonstrates a new interaction paradigm: text-to-cinema flight, where diffusion models act as the "creative operator" converting story intentions directly into aerial motion.

Related papers

• SimWAM: A Simple World Action Model for End-to-End Autonomous Driving [66.93548079427477]
  We present SimWAM, a simple yet effective WAM that leverages future-video prediction as a training-time supervision signal.<n>An isolated attention mask keeps action prediction independent of future frames, allowing trajectory prediction without explicit future-frame generation at inference.<n>Our SimWAM achieves 91.5 PDMS on NAVSIM, surpasses state-of-the-art WAM-based planners with substantially lower latency, and transfers zero-shot to nuScenes.
  arXiv  Detail & Related papers  (2026-08-07T17:59:09Z)
• AeroAct: Action-Centered World-Action Models for Language-Conditioned Quadrotor Flight [42.75555298269899]
  Language-conditioned quadrotor flight requires a policy to ground semantic goals.<n>We present AeroAct, an action-centered world-action model (WAM) for quadrotor navigation.
  arXiv  Detail & Related papers  (2026-07-16T13:46:00Z)
• Generative AI for Safe and Photorealistic Drone Light Shows [51.71264054256171]
  Drone light shows are redefining aerial entertainment, yet their widespread adoption is bottlenecked by labor-intensive, manual animation.<n>We introduce SWAN, an end-to-end pipeline that synthesizes photorealistic, large-scale, and collision-free drone choreographies directly from text prompts.<n>We demonstrate scalability by safely orchestrating simulated 2,000-drone formations and validate physical feasibility on a dense real-world swarm of 49 quadcopters.
  arXiv  Detail & Related papers  (2026-06-24T06:34:44Z)
• ImagineUAV: Aerial Vision-Language Navigation via World-Action Modeling and Kinodynamic Planning [8.298446181889675]
  Vision-language navigation (VLN) for UAVs demands grounding free-form instructions into 6-DoF flight under partial observability.<n>We propose ImagineUAV, an imagination-driven framework leveraging cascaded world-action modeling.<n>ImagineUAV outperforms prior VLN and VLA baselines on benchmarks and real-world flights.
  arXiv  Detail & Related papers  (2026-05-31T12:39:44Z)
• Aero-World: Action-Conditioned Aerial Video Generation from Inertial Controls [57.166020875486474]
  We propose textbfAero-World, a method for converting a pretrained image-to-video diffusion model into a controllable aerial video generator.<n>A frozen latent-space Physics Probe, trained independently on real video--IMU pairs, provides differentiable inertial-consistency supervision during LoRA finetuning.<n>On AeroBench, Aero-World improves mean AAS from 57.7 to 63.6 over action-only finetuning and gives a stronger quality-control trade-off than AirScape.
  arXiv  Detail & Related papers  (2026-05-19T12:02:17Z)
• MAD: Motion Appearance Decoupling for efficient Driving World Models [94.40548866741791]
  We propose an efficient adaptation framework that converts generalist video models into controllable driving world models.<n>Key idea is to decouple motion learning from appearance synthesis.<n>Scaling to LTX, our MAD-LTX model outperforms all open-source competitors.
  arXiv  Detail & Related papers  (2026-01-14T12:52:23Z)
• Real-Time Motion-Controllable Autoregressive Video Diffusion [79.32730467857535]
  We propose AR-Drag, the first RL-enhanced few-step AR video diffusion model for real-time image-to-video generation with diverse motion control.<n>We first fine-tune a base I2V model to support basic motion control, then further improve it via reinforcement with a trajectory-based reward model.<n>Our design preserves the Markov property through a Self-Rollout learning mechanism and accelerates training by selectively denoising steps.
  arXiv  Detail & Related papers  (2025-10-09T12:17:11Z)
• Vision-Language Modeling with Regularized Spatial Transformer Networks for All Weather Crosswind Landing of Aircraft [0.3749861135832073]
  A vision-based system trained to clear weather-induced visual degradations requires a robust landing dataset under various climatic conditions. We propose to synthesize harsh weather landing images by training a prompt-based climatic diffusion network. Also, we optimize a weather distillation model using a novel diffusion-distillation loss to learn to clear these visual degradations.
  arXiv  Detail & Related papers  (2024-05-09T06:48:42Z)
• TAFormer: A Unified Target-Aware Transformer for Video and Motion Joint Prediction in Aerial Scenes [14.924741503611749]
  We introduce a novel task called Target-Aware Aerial Video Prediction, aiming to simultaneously predict future scenes and motion states of the target. We introduce Spatiotemporal Attention (STA), which decouples the learning of video dynamics into spatial static attention and temporal dynamic attention, effectively modeling the scene appearance and motion. To alleviate the difficulty of distinguishing targets in blurry predictions, we introduce Target-Sensitive Gaussian Loss (TSGL), enhancing the model's sensitivity to both target's position and content.
  arXiv  Detail & Related papers  (2024-03-27T04:03:55Z)
• ScatterNeRF: Seeing Through Fog with Physically-Based Inverse Neural Rendering [83.75284107397003]
  We introduce ScatterNeRF, a neural rendering method which renders scenes and decomposes the fog-free background. We propose a disentangled representation for the scattering volume and the scene objects, and learn the scene reconstruction with physics-inspired losses. We validate our method by capturing multi-view In-the-Wild data and controlled captures in a large-scale fog chamber.
  arXiv  Detail & Related papers  (2023-05-03T13:24:06Z)
• Trace and Pace: Controllable Pedestrian Animation via Guided Trajectory Diffusion [83.88829943619656]
  We introduce a method for generating realistic pedestrian trajectories and full-body animations that can be controlled to meet user-defined goals. Our guided diffusion model allows users to constrain trajectories through target waypoints, speed, and specified social groups. We propose utilizing the value function learned during RL training of the animation controller to guide diffusion to produce trajectories better suited for particular scenarios.
  arXiv  Detail & Related papers  (2023-04-04T15:46:42Z)
• AirPose: Multi-View Fusion Network for Aerial 3D Human Pose and Shape Estimation [51.17610485589701]
  We present a novel markerless 3D human motion capture (MoCap) system for unstructured, outdoor environments. AirPose estimates human pose and shape using images captured by multiple uncalibrated flying cameras. AirPose itself calibrates the cameras relative to the person instead of relying on any pre-calibration.
  arXiv  Detail & Related papers  (2022-01-20T09:46:20Z)
• CNN-based Visual Ego-Motion Estimation for Fast MAV Maneuvers [5.721124285238146]
  We study convolutional neural networks (CNNs) that predict the relative pose between subsequent images from a fast-moving monocular camera. The networks we study have similar small model sizes and high inference speeds. Images for training and testing have realistic motion blur.
  arXiv  Detail & Related papers  (2021-01-06T01:20:29Z)
• Batteries, camera, action! Learning a semantic control space for expressive robot cinematography [15.895161373307378]
  We develop a data-driven framework that enables editing of complex camera positioning parameters in a semantic space. First, we generate a database of video clips with a diverse range of shots in a photo-realistic simulator. We use hundreds of participants in a crowd-sourcing framework to obtain scores for a set of semantic descriptors for each clip.
  arXiv  Detail & Related papers  (2020-11-19T21:56:53Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.