Evaluating Continual Learning Algorithms by Generating 3D Virtual Environments

• URL: http://arxiv.org/abs/2109.07855v1
• Date: Thu, 16 Sep 2021 10:37:21 GMT
• Title: Evaluating Continual Learning Algorithms by Generating 3D Virtual Environments
• Authors: Enrico Meloni, Alessandro Betti, Lapo Faggi, Simone Marullo, Matteo Tiezzi, Stefano Melacci
• Abstract summary: Continual learning refers to the ability of humans and animals to incrementally learn over time in a given environment. We propose to leverage recent advances in 3D virtual environments in order to approach the automatic generation of potentially life-long dynamic scenes with photo-realistic appearance. A novel element of this paper is that scenes are described in a parametric way, thus allowing the user to fully control the visual complexity of the input stream the agent perceives.
• Score: 66.83839051693695
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Continual learning refers to the ability of humans and animals to incrementally learn over time in a given environment. Trying to simulate this learning process in machines is a challenging task, also due to the inherent difficulty in creating conditions for designing continuously evolving dynamics that are typical of the real-world. Many existing research works usually involve training and testing of virtual agents on datasets of static images or short videos, considering sequences of distinct learning tasks. However, in order to devise continual learning algorithms that operate in more realistic conditions, it is fundamental to gain access to rich, fully customizable and controlled experimental playgrounds. Focussing on the specific case of vision, we thus propose to leverage recent advances in 3D virtual environments in order to approach the automatic generation of potentially life-long dynamic scenes with photo-realistic appearance. Scenes are composed of objects that move along variable routes with different and fully customizable timings, and randomness can also be included in their evolution. A novel element of this paper is that scenes are described in a parametric way, thus allowing the user to fully control the visual complexity of the input stream the agent perceives. These general principles are concretely implemented exploiting a recently published 3D virtual environment. The user can generate scenes without the need of having strong skills in computer graphics, since all the generation facilities are exposed through a simple high-level Python interface. We publicly share the proposed generator.

Related papers

• Scaling Instructable Agents Across Many Simulated Worlds [70.97268311053328]
  Our goal is to develop an agent that can accomplish anything a human can do in any simulated 3D environment. Our approach focuses on language-driven generality while imposing minimal assumptions. Our agents interact with environments in real-time using a generic, human-like interface.
  arXiv  Detail & Related papers  (2024-03-13T17:50:32Z)
• Self-supervised novel 2D view synthesis of large-scale scenes with efficient multi-scale voxel carving [77.07589573960436]
  We introduce an efficient multi-scale voxel carving method to generate novel views of real scenes. Our final high-resolution output is efficiently self-trained on data automatically generated by the voxel carving module. We demonstrate the effectiveness of our method on highly complex and large-scale scenes in real environments.
  arXiv  Detail & Related papers  (2023-06-26T13:57:05Z)
• 3D-IntPhys: Towards More Generalized 3D-grounded Visual Intuitive Physics under Challenging Scenes [68.66237114509264]
  We present a framework capable of learning 3D-grounded visual intuitive physics models from videos of complex scenes with fluids. We show our model can make long-horizon future predictions by learning from raw images and significantly outperforms models that do not employ an explicit 3D representation space.
  arXiv  Detail & Related papers  (2023-04-22T19:28:49Z)
• Stochastic Coherence Over Attention Trajectory For Continuous Learning In Video Streams [64.82800502603138]
  This paper proposes a novel neural-network-based approach to progressively and autonomously develop pixel-wise representations in a video stream. The proposed method is based on a human-like attention mechanism that allows the agent to learn by observing what is moving in the attended locations. Our experiments leverage 3D virtual environments and they show that the proposed agents can learn to distinguish objects just by observing the video stream.
  arXiv  Detail & Related papers  (2022-04-26T09:52:31Z)
• Messing Up 3D Virtual Environments: Transferable Adversarial 3D Objects [21.86544028303682]
  We study how to craft adversarial 3D objects by altering their textures, using a tool chain composed of easily accessible elements. We show that it is possible, and indeed simple, to create adversarial objects using off-the-shelf limited surrogates. We propose a saliency-based attack that intersects the two classes of adversarials in order to focus the alteration to those texture elements that are estimated to be effective in the target engine.
  arXiv  Detail & Related papers  (2021-09-17T11:06:23Z)
• iGibson, a Simulation Environment for Interactive Tasks in Large Realistic Scenes [54.04456391489063]
  iGibson is a novel simulation environment to develop robotic solutions for interactive tasks in large-scale realistic scenes. Our environment contains fifteen fully interactive home-sized scenes populated with rigid and articulated objects. iGibson features enable the generalization of navigation agents, and that the human-iGibson interface and integrated motion planners facilitate efficient imitation learning of simple human demonstrated behaviors.
  arXiv  Detail & Related papers  (2020-12-05T02:14:17Z)
• SceneGen: Generative Contextual Scene Augmentation using Scene Graph Priors [3.1969855247377827]
  We introduce SceneGen, a generative contextual augmentation framework that predicts virtual object positions and orientations within existing scenes. SceneGen takes a semantically segmented scene as input, and outputs positional and orientational probability maps for placing virtual content. We formulate a novel spatial Scene Graph representation, which encapsulates explicit topological properties between objects, object groups, and the room. To demonstrate our system in action, we develop an Augmented Reality application, in which objects can be contextually augmented in real-time.
  arXiv  Detail & Related papers  (2020-09-25T18:36:27Z)
• SAILenv: Learning in Virtual Visual Environments Made Simple [16.979621213790015]
  We present a novel platform that allows researchers to experiment visual recognition in virtual 3D scenes. A few lines of code are needed to interface every algorithm with the virtual world, and non-3D-graphics experts can easily customize the 3D environment itself. Our framework yields pixel-level semantic and instance labeling, depth, and, to the best of our knowledge, it is the only one that provides motion-related information directly inherited from the 3D engine.
  arXiv  Detail & Related papers  (2020-07-16T09:50:23Z)
• Stillleben: Realistic Scene Synthesis for Deep Learning in Robotics [33.30312206728974]
  We describe a synthesis pipeline capable of producing training data for cluttered scene perception tasks. Our approach arranges object meshes in physically realistic, dense scenes using physics simulation. Our pipeline can be run online during training of a deep neural network.
  arXiv  Detail & Related papers  (2020-05-12T10:11:00Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.