AutoNeRF: Training Implicit Scene Representations with Autonomous Agents
- URL: http://arxiv.org/abs/2304.11241v2
- Date: Fri, 22 Dec 2023 13:55:53 GMT
- Title: AutoNeRF: Training Implicit Scene Representations with Autonomous Agents
- Authors: Pierre Marza, Laetitia Matignon, Olivier Simonin, Dhruv Batra,
Christian Wolf, Devendra Singh Chaplot
- Abstract summary: Implicit representations such as Neural Radiance Fields (NeRF) have been shown to be very effective at novel view synthesis.
We present AutoNeRF, a method to collect data required to train NeRFs using autonomous embodied agents.
- Score: 42.90747351247687
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Implicit representations such as Neural Radiance Fields (NeRF) have been
shown to be very effective at novel view synthesis. However, these models
typically require manual and careful human data collection for training. In
this paper, we present AutoNeRF, a method to collect data required to train
NeRFs using autonomous embodied agents. Our method allows an agent to explore
an unseen environment efficiently and use the experience to build an implicit
map representation autonomously. We compare the impact of different exploration
strategies including handcrafted frontier-based exploration, end-to-end and
modular approaches composed of trained high-level planners and classical
low-level path followers. We train these models with different reward functions
tailored to this problem and evaluate the quality of the learned
representations on four different downstream tasks: classical viewpoint
rendering, map reconstruction, planning, and pose refinement. Empirical results
show that NeRFs can be trained on actively collected data using just a single
episode of experience in an unseen environment, and can be used for several
downstream robotic tasks, and that modular trained exploration models
outperform other classical and end-to-end baselines. Finally, we show that
AutoNeRF can reconstruct large-scale scenes, and is thus a useful tool to
perform scene-specific adaptation as the produced 3D environment models can be
loaded into a simulator to fine-tune a policy of interest.
Related papers
- Learning autonomous driving from aerial imagery [67.06858775696453]
Photogrammetric simulators allow the synthesis of novel views through the transformation of pre-generated assets into novel views.
We use a Neural Radiance Field (NeRF) as an intermediate representation to synthesize novel views from the point of view of a ground vehicle.
arXiv Detail & Related papers (2024-10-18T05:09:07Z) - Learning from the Giants: A Practical Approach to Underwater Depth and Surface Normals Estimation [3.0516727053033392]
This paper presents a novel deep learning model for Monocular Depth and Surface Normals Estimation (MDSNE)
It is specifically tailored for underwater environments, using a hybrid architecture that integrates CNNs with Transformers.
Our model reduces parameters by 90% and training costs by 80%, allowing real-time 3D perception on resource-constrained devices.
arXiv Detail & Related papers (2024-10-02T22:41:12Z) - Jointly Training and Pruning CNNs via Learnable Agent Guidance and Alignment [69.33930972652594]
We propose a novel structural pruning approach to jointly learn the weights and structurally prune architectures of CNN models.
The core element of our method is a Reinforcement Learning (RL) agent whose actions determine the pruning ratios of the CNN model's layers.
We conduct the joint training and pruning by iteratively training the model's weights and the agent's policy.
arXiv Detail & Related papers (2024-03-28T15:22:29Z) - Closing the Visual Sim-to-Real Gap with Object-Composable NeRFs [59.12526668734703]
We introduce Composable Object Volume NeRF (COV-NeRF), an object-composable NeRF model that is the centerpiece of a real-to-sim pipeline.
COV-NeRF extracts objects from real images and composes them into new scenes, generating photorealistic renderings and many types of 2D and 3D supervision.
arXiv Detail & Related papers (2024-03-07T00:00:02Z) - Divide and Conquer: Rethinking the Training Paradigm of Neural Radiance
Fields [24.99410489251996]
Neural radiance fields (NeRFs) have exhibited potential in high-fidelity views of 3D scenes.
Standard training paradigm of NeRF presupposes an equal importance for each image in the training set.
In this paper, we take a closer look at the implications of the current training paradigm and redesign this for more superior rendering quality.
arXiv Detail & Related papers (2024-01-29T13:23:34Z) - NSLF-OL: Online Learning of Neural Surface Light Fields alongside
Real-time Incremental 3D Reconstruction [0.76146285961466]
The paper proposes a novel Neural Surface Light Fields model that copes with the small range of view directions while producing a good result in unseen directions.
Our model learns online the Neural Surface Light Fields (NSLF) aside from real-time 3D reconstruction with a sequential data stream as the shared input.
In addition to online training, our model also provides real-time rendering after completing the data stream for visualization.
arXiv Detail & Related papers (2023-04-29T15:41:15Z) - Learning Multi-Object Dynamics with Compositional Neural Radiance Fields [63.424469458529906]
We present a method to learn compositional predictive models from image observations based on implicit object encoders, Neural Radiance Fields (NeRFs), and graph neural networks.
NeRFs have become a popular choice for representing scenes due to their strong 3D prior.
For planning, we utilize RRTs in the learned latent space, where we can exploit our model and the implicit object encoder to make sampling the latent space informative and more efficient.
arXiv Detail & Related papers (2022-02-24T01:31:29Z) - Generating Synthetic Training Data for Deep Learning-Based UAV
Trajectory Prediction [11.241614693184323]
We present an approach for generating synthetic trajectory data of unmanned-aerial-vehicles (UAVs) in image space.
We show that an RNN-based prediction model solely trained on the generated data can outperform classic reference models on a real-world UAV tracking dataset.
arXiv Detail & Related papers (2021-07-01T13:08:31Z) - Learning to Move with Affordance Maps [57.198806691838364]
The ability to autonomously explore and navigate a physical space is a fundamental requirement for virtually any mobile autonomous agent.
Traditional SLAM-based approaches for exploration and navigation largely focus on leveraging scene geometry.
We show that learned affordance maps can be used to augment traditional approaches for both exploration and navigation, providing significant improvements in performance.
arXiv Detail & Related papers (2020-01-08T04:05:11Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.