AutoDirector: Online Auto-scheduling Agents for Multi-sensory Composition

• URL: http://arxiv.org/abs/2408.11564v1
• Date: Wed, 21 Aug 2024 12:18:22 GMT
• Title: AutoDirector: Online Auto-scheduling Agents for Multi-sensory Composition
• Authors: Minheng Ni, Chenfei Wu, Huaying Yuan, Zhengyuan Yang, Ming Gong, Lijuan Wang, Zicheng Liu, Wangmeng Zuo, Nan Duan,
• Abstract summary: AutoDirector is an interactive multi-sensory composition framework that supports long shots, special effects, music scoring, dubbing, and lip-syncing. It improves the efficiency of multi-sensory film production through automatic scheduling and supports the modification and improvement of interactive tasks to meet user needs.
• Score: 149.89952404881174
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: With the advancement of generative models, the synthesis of different sensory elements such as music, visuals, and speech has achieved significant realism. However, the approach to generate multi-sensory outputs has not been fully explored, limiting the application on high-value scenarios such as of directing a film. Developing a movie director agent faces two major challenges: (1) Lack of parallelism and online scheduling with production steps: In the production of multi-sensory films, there are complex dependencies between different sensory elements, and the production time for each element varies. (2) Diverse needs and clear communication demands with users: Users often cannot clearly express their needs until they see a draft, which requires human-computer interaction and iteration to continually adjust and optimize the film content based on user feedback. To address these issues, we introduce AutoDirector, an interactive multi-sensory composition framework that supports long shots, special effects, music scoring, dubbing, and lip-syncing. This framework improves the efficiency of multi-sensory film production through automatic scheduling and supports the modification and improvement of interactive tasks to meet user needs. AutoDirector not only expands the application scope of human-machine collaboration but also demonstrates the potential of AI in collaborating with humans in the role of a film director to complete multi-sensory films.

Related papers

• FilmAgent: A Multi-Agent Framework for End-to-End Film Automation in Virtual 3D Spaces [42.3549764892671]
  FilmAgent is a novel multi-agent collaborative framework for end-to-end film automation. FilmAgent simulates various crew roles, including directors, screenwriters, actors, and cinematographers. A team of agents collaborates through iterative feedback and revisions, thereby verifying intermediate scripts and reducing hallucinations.
  arXiv  Detail & Related papers  (2025-01-22T14:36:30Z)
• HumanVBench: Exploring Human-Centric Video Understanding Capabilities of MLLMs with Synthetic Benchmark Data [55.739633494946204]
  We present HumanVBench, an innovative benchmark meticulously crafted to bridge gaps in the evaluation of video MLLMs. HumanVBench comprises 17 carefully designed tasks that explore two primary dimensions: inner emotion and outer manifestations, spanning static and dynamic, basic and complex, as well as single-modal and cross-modal aspects.
  arXiv  Detail & Related papers  (2024-12-23T13:45:56Z)
• Two-in-One: Unified Multi-Person Interactive Motion Generation by Latent Diffusion Transformer [24.166147954731652]
  Multi-person interactive motion generation is a critical yet under-explored domain in computer character animation. Current research often employs separate module branches for individual motions, leading to a loss of interaction information. We propose a novel, unified approach that models multi-person motions and their interactions within a single latent space.
  arXiv  Detail & Related papers  (2024-12-21T15:35:50Z)
• It Takes Two: Real-time Co-Speech Two-person's Interaction Generation via Reactive Auto-regressive Diffusion Model [34.94330722832987]
  We introduce an audio-driven, auto-regressive system designed to synthesize dynamic movements for two characters during a conversation. To the best of our knowledge, this is the first system capable of generating interactive full-body motions for two characters from speech in an online manner.
  arXiv  Detail & Related papers  (2024-12-03T12:31:44Z)
• Vocal Sandbox: Continual Learning and Adaptation for Situated Human-Robot Collaboration [64.6107798750142]
  Vocal Sandbox is a framework for enabling seamless human-robot collaboration in situated environments. We design lightweight and interpretable learning algorithms that allow users to build an understanding and co-adapt to a robot's capabilities in real-time. We evaluate Vocal Sandbox in two settings: collaborative gift bag assembly and LEGO stop-motion animation.
  arXiv  Detail & Related papers  (2024-11-04T20:44:40Z)
• DeepInteraction++: Multi-Modality Interaction for Autonomous Driving [80.8837864849534]
  We introduce a novel modality interaction strategy that allows individual per-modality representations to be learned and maintained throughout. DeepInteraction++ is a multi-modal interaction framework characterized by a multi-modal representational interaction encoder and a multi-modal predictive interaction decoder. Experiments demonstrate the superior performance of the proposed framework on both 3D object detection and end-to-end autonomous driving tasks.
  arXiv  Detail & Related papers  (2024-08-09T14:04:21Z)
• Towards Embedding Dynamic Personas in Interactive Robots: Masquerading Animated Social Kinematics (MASK) [10.351714893090964]
  This paper presents the design and development of an innovative interactive robotic system to enhance audience engagement using character-like personas. Built upon the foundations of persona-driven dialog agents, this work extends the agent's application to the physical realm, employing robots to provide a more captivating and interactive experience.
  arXiv  Detail & Related papers  (2024-03-15T06:22:32Z)
• ReMoS: 3D Motion-Conditioned Reaction Synthesis for Two-Person Interactions [66.87211993793807]
  We present ReMoS, a denoising diffusion based model that synthesizes full body motion of a person in two person interaction scenario. We demonstrate ReMoS across challenging two person scenarios such as pair dancing, Ninjutsu, kickboxing, and acrobatics. We also contribute the ReMoCap dataset for two person interactions containing full body and finger motions.
  arXiv  Detail & Related papers  (2023-11-28T18:59:52Z)
• Smart Director: An Event-Driven Directing System for Live Broadcasting [110.30675947733167]
  Smart Director aims at mimicking the typical human-in-the-loop broadcasting process to automatically create near-professional broadcasting programs in real-time. Our system is the first end-to-end automated directing system for multi-camera sports broadcasting.
  arXiv  Detail & Related papers  (2022-01-11T16:14:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.