3D Human Reconstruction in the Wild with Collaborative Aerial Cameras

• URL: http://arxiv.org/abs/2108.03936v1
• Date: Mon, 9 Aug 2021 11:03:38 GMT
• Title: 3D Human Reconstruction in the Wild with Collaborative Aerial Cameras
• Authors: Cherie Ho, Andrew Jong, Harry Freeman, Rohan Rao, Rogerio Bonatti, Sebastian Scherer
• Abstract summary: We present a real-time aerial system for multi-camera control that can reconstruct human motions in natural environments without the use of special-purpose markers. We develop a multi-robot coordination scheme that maintains the optimal flight formation for target reconstruction quality amongst obstacles.
• Score: 3.3674370488883434
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Aerial vehicles are revolutionizing applications that require capturing the 3D structure of dynamic targets in the wild, such as sports, medicine, and entertainment. The core challenges in developing a motion-capture system that operates in outdoors environments are: (1) 3D inference requires multiple simultaneous viewpoints of the target, (2) occlusion caused by obstacles is frequent when tracking moving targets, and (3) the camera and vehicle state estimation is noisy. We present a real-time aerial system for multi-camera control that can reconstruct human motions in natural environments without the use of special-purpose markers. We develop a multi-robot coordination scheme that maintains the optimal flight formation for target reconstruction quality amongst obstacles. We provide studies evaluating system performance in simulation, and validate real-world performance using two drones while a target performs activities such as jogging and playing soccer. Supplementary video: https://youtu.be/jxt91vx0cns

Related papers

• Sitcom-Crafter: A Plot-Driven Human Motion Generation System in 3D Scenes [83.55301458112672]
  Sitcom-Crafter is a system for human motion generation in 3D space. Central to the function generation modules is our novel 3D scene-aware human-human interaction module. Augmentation modules encompass plot comprehension for command generation, motion synchronization for seamless integration of different motion types.
  arXiv  Detail & Related papers  (2024-10-14T17:56:19Z)
• Image Conductor: Precision Control for Interactive Video Synthesis [90.2353794019393]
  Filmmaking and animation production often require sophisticated techniques for coordinating camera transitions and object movements. Image Conductor is a method for precise control of camera transitions and object movements to generate video assets from a single image.
  arXiv  Detail & Related papers  (2024-06-21T17:55:05Z)
• Cinematic Behavior Transfer via NeRF-based Differentiable Filming [63.1622492808519]
  Existing SLAM methods face limitations in dynamic scenes and human pose estimation often focuses on 2D projections. We first introduce a reverse filming behavior estimation technique. We then introduce a cinematic transfer pipeline that is able to transfer various shot types to a new 2D video or a 3D virtual environment.
  arXiv  Detail & Related papers  (2023-11-29T15:56:58Z)
• Autonomous Marker-less Rapid Aerial Grasping [5.892028494793913]
  We propose a vision-based system for autonomous rapid aerial grasping. We generate a dense point cloud of the detected objects and perform geometry-based grasp planning. We show the first use of geometry-based grasping techniques with a flying platform.
  arXiv  Detail & Related papers  (2022-11-23T16:25:49Z)
• Towards Live 3D Reconstruction from Wearable Video: An Evaluation of V-SLAM, NeRF, and Videogrammetry Techniques [20.514826446476267]
  Mixed reality (MR) is a key technology which promises to change the future of warfare. To enable this technology, a large-scale 3D model of a physical environment must be maintained based on live sensor observations. We survey several 3D reconstruction algorithms for large-scale mapping for military applications given only live video.
  arXiv  Detail & Related papers  (2022-11-21T19:57:51Z)
• Learning Deep Sensorimotor Policies for Vision-based Autonomous Drone Racing [52.50284630866713]
  Existing systems often require hand-engineered components for state estimation, planning, and control. This paper tackles the vision-based autonomous-drone-racing problem by learning deep sensorimotor policies.
  arXiv  Detail & Related papers  (2022-10-26T19:03:17Z)
• MoCaNet: Motion Retargeting in-the-wild via Canonicalization Networks [77.56526918859345]
  We present a novel framework that brings the 3D motion task from controlled environments to in-the-wild scenarios. It is capable of body motion from a character in a 2D monocular video to a 3D character without using any motion capture system or 3D reconstruction procedure.
  arXiv  Detail & Related papers  (2021-12-19T07:52:05Z)
• GLAMR: Global Occlusion-Aware Human Mesh Recovery with Dynamic Cameras [99.07219478953982]
  We present an approach for 3D global human mesh recovery from monocular videos recorded with dynamic cameras. We first propose a deep generative motion infiller, which autoregressively infills the body motions of occluded humans based on visible motions. In contrast to prior work, our approach reconstructs human meshes in consistent global coordinates even with dynamic cameras.
  arXiv  Detail & Related papers  (2021-12-02T18:59:54Z)
• Do You See What I See? Coordinating Multiple Aerial Cameras for Robot Cinematography [9.870369982132678]
  We develop a real-time multi-UAV coordination system that is capable of recording dynamic targets while maximizing shot diversity and avoiding collisions. We show that our coordination scheme has low computational cost and takes only 1.17 ms on average to plan for a team of 3 UAVs over a 10 s time horizon.
  arXiv  Detail & Related papers  (2020-11-10T22:43:25Z)
• Reconstruction of 3D flight trajectories from ad-hoc camera networks [19.96488566402593]
  We present a method to reconstruct the 3D trajectory of an airborne robotic system only from videos recorded with cameras that are unsynchronized. Our approach enables robust and accurate outside-in tracking of dynamically flying targets, with cheap and easy-to-deploy equipment.
  arXiv  Detail & Related papers  (2020-03-10T14:57:32Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.