3D Human Reconstruction in the Wild with Collaborative Aerial Cameras

• URL: http://arxiv.org/abs/2108.03936v1
• Date: Mon, 9 Aug 2021 11:03:38 GMT
• Title: 3D Human Reconstruction in the Wild with Collaborative Aerial Cameras
• Authors: Cherie Ho, Andrew Jong, Harry Freeman, Rohan Rao, Rogerio Bonatti, Sebastian Scherer
• Abstract summary: We present a real-time aerial system for multi-camera control that can reconstruct human motions in natural environments without the use of special-purpose markers. We develop a multi-robot coordination scheme that maintains the optimal flight formation for target reconstruction quality amongst obstacles.
• Score: 3.3674370488883434
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Aerial vehicles are revolutionizing applications that require capturing the 3D structure of dynamic targets in the wild, such as sports, medicine, and entertainment. The core challenges in developing a motion-capture system that operates in outdoors environments are: (1) 3D inference requires multiple simultaneous viewpoints of the target, (2) occlusion caused by obstacles is frequent when tracking moving targets, and (3) the camera and vehicle state estimation is noisy. We present a real-time aerial system for multi-camera control that can reconstruct human motions in natural environments without the use of special-purpose markers. We develop a multi-robot coordination scheme that maintains the optimal flight formation for target reconstruction quality amongst obstacles. We provide studies evaluating system performance in simulation, and validate real-world performance using two drones while a target performs activities such as jogging and playing soccer. Supplementary video: https://youtu.be/jxt91vx0cns

Related papers

• Dyn-HaMR: Recovering 4D Interacting Hand Motion from a Dynamic Camera [49.82535393220003]
  Dyn-HaMR is the first approach to reconstruct 4D global hand motion from monocular videos recorded by dynamic cameras in the wild. We show that our approach significantly outperforms state-of-the-art methods in terms of 4D global mesh recovery. This establishes a new benchmark for hand motion reconstruction from monocular video with moving cameras.
  arXiv  Detail & Related papers  (2024-12-17T12:43:10Z)
• Learning Camera Movement Control from Real-World Drone Videos [25.10006841389459]
  Existing AI videography methods struggle with limited appearance diversity in simulation training. We propose a scalable method that involves collecting real-world training data. We show that our system effectively learns to perform challenging camera movements.
  arXiv  Detail & Related papers  (2024-12-12T18:59:54Z)
• 3DTrajMaster: Mastering 3D Trajectory for Multi-Entity Motion in Video Generation [83.98251722144195]
  Previous methods on controllable video generation primarily leverage 2D control signals to manipulate object motions. We introduce 3DTrajMaster, a robust controller that regulates multi-entity dynamics in 3D space. We show that 3DTrajMaster sets a new state-of-the-art in both accuracy and generalization for controlling multi-entity 3D motions.
  arXiv  Detail & Related papers  (2024-12-10T18:55:13Z)
• A Cross-Scene Benchmark for Open-World Drone Active Tracking [54.235808061746525]
  Drone Visual Active Tracking aims to autonomously follow a target object by controlling the motion system based on visual observations. We propose a unified cross-scene cross-domain benchmark for open-world drone active tracking called DAT. We also propose a reinforcement learning-based drone tracking method called R-VAT.
  arXiv  Detail & Related papers  (2024-12-01T09:37:46Z)
• Towards Live 3D Reconstruction from Wearable Video: An Evaluation of V-SLAM, NeRF, and Videogrammetry Techniques [20.514826446476267]
  Mixed reality (MR) is a key technology which promises to change the future of warfare. To enable this technology, a large-scale 3D model of a physical environment must be maintained based on live sensor observations. We survey several 3D reconstruction algorithms for large-scale mapping for military applications given only live video.
  arXiv  Detail & Related papers  (2022-11-21T19:57:51Z)
• Aerial Monocular 3D Object Detection [67.20369963664314]
  DVDET is proposed to achieve aerial monocular 3D object detection in both the 2D image space and the 3D physical space. To address the severe view deformation issue, we propose a novel trainable geo-deformable transformation module. To encourage more researchers to investigate this area, we will release the dataset and related code.
  arXiv  Detail & Related papers  (2022-08-08T08:32:56Z)
• MoCaNet: Motion Retargeting in-the-wild via Canonicalization Networks [77.56526918859345]
  We present a novel framework that brings the 3D motion task from controlled environments to in-the-wild scenarios. It is capable of body motion from a character in a 2D monocular video to a 3D character without using any motion capture system or 3D reconstruction procedure.
  arXiv  Detail & Related papers  (2021-12-19T07:52:05Z)
• GLAMR: Global Occlusion-Aware Human Mesh Recovery with Dynamic Cameras [99.07219478953982]
  We present an approach for 3D global human mesh recovery from monocular videos recorded with dynamic cameras. We first propose a deep generative motion infiller, which autoregressively infills the body motions of occluded humans based on visible motions. In contrast to prior work, our approach reconstructs human meshes in consistent global coordinates even with dynamic cameras.
  arXiv  Detail & Related papers  (2021-12-02T18:59:54Z)
• Do You See What I See? Coordinating Multiple Aerial Cameras for Robot Cinematography [9.870369982132678]
  We develop a real-time multi-UAV coordination system that is capable of recording dynamic targets while maximizing shot diversity and avoiding collisions. We show that our coordination scheme has low computational cost and takes only 1.17 ms on average to plan for a team of 3 UAVs over a 10 s time horizon.
  arXiv  Detail & Related papers  (2020-11-10T22:43:25Z)
• Reconstruction of 3D flight trajectories from ad-hoc camera networks [19.96488566402593]
  We present a method to reconstruct the 3D trajectory of an airborne robotic system only from videos recorded with cameras that are unsynchronized. Our approach enables robust and accurate outside-in tracking of dynamically flying targets, with cheap and easy-to-deploy equipment.
  arXiv  Detail & Related papers  (2020-03-10T14:57:32Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.