NeRF-Aug: Data Augmentation for Robotics with Neural Radiance Fields

• URL: http://arxiv.org/abs/2411.02482v3
• Date: Sun, 14 Sep 2025 01:23:00 GMT
• Title: NeRF-Aug: Data Augmentation for Robotics with Neural Radiance Fields
• Authors: Eric Zhu, Mara Levy, Matthew Gwilliam, Abhinav Shrivastava,
• Abstract summary: NeRF-Aug is capable of teaching a policy to interact with objects that are not present in the dataset.<n>We demonstrate the effectiveness of our method on 5 tasks with 9 novel objects that are not present in the expert demonstrations.
• Score: 50.54135058422269
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Training a policy that can generalize to unknown objects is a long standing challenge within the field of robotics. The performance of a policy often drops significantly in situations where an object in the scene was not seen during training. To solve this problem, we present NeRF-Aug, a novel method that is capable of teaching a policy to interact with objects that are not present in the dataset. This approach differs from existing approaches by leveraging the speed, photorealism, and 3D consistency of a neural radiance field for augmentation. NeRF-Aug both creates more photorealistic data and runs 63% faster than existing methods. We demonstrate the effectiveness of our method on 5 tasks with 9 novel objects that are not present in the expert demonstrations. We achieve an average performance boost of 55.6% when comparing our method to the next best method. You can see video results at https://nerf-aug.github.io.

Related papers

• Unleashing the Power of Data Synthesis in Visual Localization [17.159091187694884]
  Methods that regress camera poses from query images have gained attention for fast inference. We aim to unleash the power of data synthesis to promote the generalizability of pose regression. We build a two-branch joint training pipeline, with an adversarial discriminator to bridge the syn-to-real gap.
  arXiv  Detail & Related papers  (2024-11-28T16:58:10Z)
• CTNeRF: Cross-Time Transformer for Dynamic Neural Radiance Field from Monocular Video [25.551944406980297]
  We propose a novel approach to generate high-quality novel views from monocular videos of complex and dynamic scenes. We introduce a module that operates in both the time and frequency domains to aggregate the features of object motion. Our experiments demonstrate significant improvements over state-of-the-art methods on dynamic scene datasets.
  arXiv  Detail & Related papers  (2024-01-10T00:40:05Z)
• Deformable Neural Radiance Fields using RGB and Event Cameras [65.40527279809474]
  We develop a novel method to model the deformable neural radiance fields using RGB and event cameras. The proposed method uses the asynchronous stream of events and sparse RGB frames. Experiments conducted on both realistically rendered graphics and real-world datasets demonstrate a significant benefit of the proposed method.
  arXiv  Detail & Related papers  (2023-09-15T14:19:36Z)
• BundleSDF: Neural 6-DoF Tracking and 3D Reconstruction of Unknown Objects [89.2314092102403]
  We present a near real-time method for 6-DoF tracking of an unknown object from a monocular RGBD video sequence. Our method works for arbitrary rigid objects, even when visual texture is largely absent.
  arXiv  Detail & Related papers  (2023-03-24T17:13:49Z)
• Learning Neural Volumetric Representations of Dynamic Humans in Minutes [49.10057060558854]
  We propose a novel method for learning neural volumetric videos of dynamic humans from sparse view videos in minutes with competitive visual quality. Specifically, we define a novel part-based voxelized human representation to better distribute the representational power of the network to different human parts. Experiments demonstrate that our model can be learned 100 times faster than prior per-scene optimization methods.
  arXiv  Detail & Related papers  (2023-02-23T18:57:01Z)
• Scaling Robot Learning with Semantically Imagined Experience [21.361979238427722]
  Recent advances in robot learning have shown promise in enabling robots to perform manipulation tasks. One of the key contributing factors to this progress is the scale of robot data used to train the models. We propose an alternative route and leverage text-to-image foundation models widely used in computer vision and natural language processing.
  arXiv  Detail & Related papers  (2023-02-22T18:47:51Z)
• NeRF in the Palm of Your Hand: Corrective Augmentation for Robotics via Novel-View Synthesis [50.93065653283523]
  SPARTN (Synthetic Perturbations for Augmenting Robot Trajectories via NeRF) is a fully-offline data augmentation scheme for improving robot policies. Our approach leverages neural radiance fields (NeRFs) to synthetically inject corrective noise into visual demonstrations. In a simulated 6-DoF visual grasping benchmark, SPARTN improves success rates by 2.8$times$ over imitation learning without the corrective augmentations.
  arXiv  Detail & Related papers  (2023-01-18T23:25:27Z)
• AligNeRF: High-Fidelity Neural Radiance Fields via Alignment-Aware Training [100.33713282611448]
  We conduct the first pilot study on training NeRF with high-resolution data. We propose the corresponding solutions, including marrying the multilayer perceptron with convolutional layers. Our approach is nearly free without introducing obvious training/testing costs.
  arXiv  Detail & Related papers  (2022-11-17T17:22:28Z)
• SelfNeRF: Fast Training NeRF for Human from Monocular Self-rotating Video [29.50059002228373]
  SelfNeRF is an efficient neural radiance field based novel view synthesis method for human performance. It can train from scratch and achieve high-fidelity results in about twenty minutes.
  arXiv  Detail & Related papers  (2022-10-04T14:54:40Z)
• EfficientNeRF: Efficient Neural Radiance Fields [63.76830521051605]
  We present EfficientNeRF as an efficient NeRF-based method to represent 3D scene and synthesize novel-view images. Our method can reduce over 88% of training time, reach rendering speed of over 200 FPS, while still achieving competitive accuracy.
  arXiv  Detail & Related papers  (2022-06-02T05:36:44Z)
• NeRF-Supervision: Learning Dense Object Descriptors from Neural Radiance Fields [54.27264716713327]
  We show that a Neural Radiance Fields (NeRF) representation of a scene can be used to train dense object descriptors. We use an optimized NeRF to extract dense correspondences between multiple views of an object, and then use these correspondences as training data for learning a view-invariant representation of the object. Dense correspondence models supervised with our method significantly outperform off-the-shelf learned descriptors by 106%.
  arXiv  Detail & Related papers  (2022-03-03T18:49:57Z)
• Object-Based Augmentation Improves Quality of Remote SensingSemantic Segmentation [0.0]
  This study focuses on the development and testing of object-based augmentation. We propose a novel pipeline for georeferenced image augmentation that enables a significant increase in the number of training samples. The presented pipeline is called object-based augmentation (OBA) and exploits objects' segmentation masks to produce new realistic training scenes.
  arXiv  Detail & Related papers  (2021-05-12T08:54:55Z)
• A Framework for Efficient Robotic Manipulation [79.10407063260473]
  We show that a single robotic arm can learn sparse-reward manipulation policies from pixels. We show that, given only 10 demonstrations, a single robotic arm can learn sparse-reward manipulation policies from pixels.
  arXiv  Detail & Related papers  (2020-12-14T22:18:39Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.