AutoNeRF: Training Implicit Scene Representations with Autonomous Agents
- URL: http://arxiv.org/abs/2304.11241v2
- Date: Fri, 22 Dec 2023 13:55:53 GMT
- Title: AutoNeRF: Training Implicit Scene Representations with Autonomous Agents
- Authors: Pierre Marza, Laetitia Matignon, Olivier Simonin, Dhruv Batra,
Christian Wolf, Devendra Singh Chaplot
- Abstract summary: Implicit representations such as Neural Radiance Fields (NeRF) have been shown to be very effective at novel view synthesis.
We present AutoNeRF, a method to collect data required to train NeRFs using autonomous embodied agents.
- Score: 42.90747351247687
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Implicit representations such as Neural Radiance Fields (NeRF) have been
shown to be very effective at novel view synthesis. However, these models
typically require manual and careful human data collection for training. In
this paper, we present AutoNeRF, a method to collect data required to train
NeRFs using autonomous embodied agents. Our method allows an agent to explore
an unseen environment efficiently and use the experience to build an implicit
map representation autonomously. We compare the impact of different exploration
strategies including handcrafted frontier-based exploration, end-to-end and
modular approaches composed of trained high-level planners and classical
low-level path followers. We train these models with different reward functions
tailored to this problem and evaluate the quality of the learned
representations on four different downstream tasks: classical viewpoint
rendering, map reconstruction, planning, and pose refinement. Empirical results
show that NeRFs can be trained on actively collected data using just a single
episode of experience in an unseen environment, and can be used for several
downstream robotic tasks, and that modular trained exploration models
outperform other classical and end-to-end baselines. Finally, we show that
AutoNeRF can reconstruct large-scale scenes, and is thus a useful tool to
perform scene-specific adaptation as the produced 3D environment models can be
loaded into a simulator to fine-tune a policy of interest.
Related papers
- Jointly Training and Pruning CNNs via Learnable Agent Guidance and Alignment [69.33930972652594]
We propose a novel structural pruning approach to jointly learn the weights and structurally prune architectures of CNN models.
The core element of our method is a Reinforcement Learning (RL) agent whose actions determine the pruning ratios of the CNN model's layers.
We conduct the joint training and pruning by iteratively training the model's weights and the agent's policy.
arXiv Detail & Related papers (2024-03-28T15:22:29Z) - Closing the Visual Sim-to-Real Gap with Object-Composable NeRFs [59.12526668734703]
We introduce Composable Object Volume NeRF (COV-NeRF), an object-composable NeRF model that is the centerpiece of a real-to-sim pipeline.
COV-NeRF extracts objects from real images and composes them into new scenes, generating photorealistic renderings and many types of 2D and 3D supervision.
arXiv Detail & Related papers (2024-03-07T00:00:02Z) - Divide and Conquer: Rethinking the Training Paradigm of Neural Radiance
Fields [24.99410489251996]
Neural radiance fields (NeRFs) have exhibited potential in high-fidelity views of 3D scenes.
Standard training paradigm of NeRF presupposes an equal importance for each image in the training set.
In this paper, we take a closer look at the implications of the current training paradigm and redesign this for more superior rendering quality.
arXiv Detail & Related papers (2024-01-29T13:23:34Z) - NSLF-OL: Online Learning of Neural Surface Light Fields alongside
Real-time Incremental 3D Reconstruction [0.76146285961466]
The paper proposes a novel Neural Surface Light Fields model that copes with the small range of view directions while producing a good result in unseen directions.
Our model learns online the Neural Surface Light Fields (NSLF) aside from real-time 3D reconstruction with a sequential data stream as the shared input.
In addition to online training, our model also provides real-time rendering after completing the data stream for visualization.
arXiv Detail & Related papers (2023-04-29T15:41:15Z) - Neural-Sim: Learning to Generate Training Data with NeRF [31.81496344354997]
We present the first fully differentiable synthetic data pipeline that uses Neural Radiance Fields (NeRFs) in a closed-loop with a target application's loss function.
Our approach generates data on-demand, with no human labor, to maximize accuracy for a target task.
arXiv Detail & Related papers (2022-07-22T22:48:33Z) - NeRF-Supervision: Learning Dense Object Descriptors from Neural Radiance
Fields [54.27264716713327]
We show that a Neural Radiance Fields (NeRF) representation of a scene can be used to train dense object descriptors.
We use an optimized NeRF to extract dense correspondences between multiple views of an object, and then use these correspondences as training data for learning a view-invariant representation of the object.
Dense correspondence models supervised with our method significantly outperform off-the-shelf learned descriptors by 106%.
arXiv Detail & Related papers (2022-03-03T18:49:57Z) - Learning Multi-Object Dynamics with Compositional Neural Radiance Fields [63.424469458529906]
We present a method to learn compositional predictive models from image observations based on implicit object encoders, Neural Radiance Fields (NeRFs), and graph neural networks.
NeRFs have become a popular choice for representing scenes due to their strong 3D prior.
For planning, we utilize RRTs in the learned latent space, where we can exploit our model and the implicit object encoder to make sampling the latent space informative and more efficient.
arXiv Detail & Related papers (2022-02-24T01:31:29Z) - Generating Synthetic Training Data for Deep Learning-Based UAV
Trajectory Prediction [11.241614693184323]
We present an approach for generating synthetic trajectory data of unmanned-aerial-vehicles (UAVs) in image space.
We show that an RNN-based prediction model solely trained on the generated data can outperform classic reference models on a real-world UAV tracking dataset.
arXiv Detail & Related papers (2021-07-01T13:08:31Z) - Learning to Move with Affordance Maps [57.198806691838364]
The ability to autonomously explore and navigate a physical space is a fundamental requirement for virtually any mobile autonomous agent.
Traditional SLAM-based approaches for exploration and navigation largely focus on leveraging scene geometry.
We show that learned affordance maps can be used to augment traditional approaches for both exploration and navigation, providing significant improvements in performance.
arXiv Detail & Related papers (2020-01-08T04:05:11Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.