Learning Controllable 3D Level Generators

• URL: http://arxiv.org/abs/2206.13623v1
• Date: Mon, 27 Jun 2022 20:43:56 GMT
• Title: Learning Controllable 3D Level Generators
• Authors: Zehua Jiang, Sam Earle, Michael C. Green, Julian Togelius
• Abstract summary: We introduce several PCGRL tasks for the 3D domain, Minecraft (Mojang Studios, 2009) These tasks will challenge RL-based generators using affordances often found in 3D environments, such as jumping, multiple dimensional movement, and gravity. We train an agent to optimize each of these tasks to explore the capabilities of previous research in PCGRL.
• Score: 3.95471659767555
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Procedural Content Generation via Reinforcement Learning (PCGRL) foregoes the need for large human-authored data-sets and allows agents to train explicitly on functional constraints, using computable, user-defined measures of quality instead of target output. We explore the application of PCGRL to 3D domains, in which content-generation tasks naturally have greater complexity and potential pertinence to real-world applications. Here, we introduce several PCGRL tasks for the 3D domain, Minecraft (Mojang Studios, 2009). These tasks will challenge RL-based generators using affordances often found in 3D environments, such as jumping, multiple dimensional movement, and gravity. We train an agent to optimize each of these tasks to explore the capabilities of previous research in PCGRL. This agent is able to generate relatively complex and diverse levels, and generalize to random initial states and control targets. Controllability tests in the presented tasks demonstrate their utility to analyze success and failure for 3D generators.

Related papers

• Scalable Multi-Task Reinforcement Learning for Generalizable Spatial Intelligence in Visuomotor Agents [12.945269075811112]
  We show that RL-finetuned visuomotor agents in Minecraft can achieve zero-shot generalization to unseen worlds.<n>We propose automated task synthesis within the highly customizable Minecraft environment for large-scale multi-task RL training.
  arXiv  Detail & Related papers  (2025-07-31T16:20:02Z)
• Automating 3D Dataset Generation with Neural Radiance Fields [0.0]
  Training performant detection models require diverse, precisely annotated, and large scale datasets. We propose a pipeline for automatic generation of 3D datasets for arbitrary objects. Our pipeline is fast, easy to use and has a high degree of automation.
  arXiv  Detail & Related papers  (2025-03-20T10:01:32Z)
• Grammarization-Based Grasping with Deep Multi-Autoencoder Latent Space Exploration by Reinforcement Learning Agent [0.0]
  We propose a novel framework for robotic grasping based on the idea of compressing high-dimensional target and gripper features in a common latent space. Our approach simplifies grasping by using three autoencoders dedicated to the target, the gripper, and a third one that fuses their latent representations.
  arXiv  Detail & Related papers  (2024-11-13T12:26:08Z)
• PCGRL+: Scaling, Control and Generalization in Reinforcement Learning Level Generators [2.334978724544296]
  Procedural Content Generation via Reinforcement Learning (PCGRL) has been introduced as a means by which controllable designer agents can be trained. PCGRL offers a unique set of affordances for game designers, but it is constrained by the compute-intensive process of training RL agents. We implement several PCGRL environments in Jax so that all aspects of learning and simulation happen in parallel on the GPU.
  arXiv  Detail & Related papers  (2024-08-22T16:30:24Z)
• CityX: Controllable Procedural Content Generation for Unbounded 3D Cities [50.10101235281943]
  Current generative methods fall short in either diversity, controllability, or fidelity. In this work, we resort to the procedural content generation (PCG) technique for high-fidelity generation. We develop a multi-agent framework to transform multi-modal instructions, including OSM, semantic maps, and satellite images, into executable programs. Our method, named CityX, demonstrates its superiority in creating diverse, controllable, and realistic 3D urban scenes.
  arXiv  Detail & Related papers  (2024-07-24T18:05:13Z)
• PerLDiff: Controllable Street View Synthesis Using Perspective-Layout Diffusion Models [55.080748327139176]
  We introduce PerLDiff, a method for effective street view image generation that fully leverages perspective 3D geometric information. Our results justify that our PerLDiff markedly enhances the precision of generation on the NuScenes and KITTI datasets.
  arXiv  Detail & Related papers  (2024-07-08T16:46:47Z)
• SERL: A Software Suite for Sample-Efficient Robotic Reinforcement Learning [85.21378553454672]
  We develop a library containing a sample efficient off-policy deep RL method, together with methods for computing rewards and resetting the environment. We find that our implementation can achieve very efficient learning, acquiring policies for PCB board assembly, cable routing, and object relocation. These policies achieve perfect or near-perfect success rates, extreme robustness even under perturbations, and exhibit emergent robustness recovery and correction behaviors.
  arXiv  Detail & Related papers  (2024-01-29T10:01:10Z)
• Self-supervised Learning for Enhancing Geometrical Modeling in 3D-Aware Generative Adversarial Network [42.16520614686877]
  3D-GANs exhibit artifacts in their 3D geometrical modeling, such as mesh imperfections and holes. These shortcomings are primarily attributed to the limited availability of annotated 3D data. We present a Self-Supervised Learning technique tailored as an auxiliary loss for any 3D-GAN.
  arXiv  Detail & Related papers  (2023-12-19T04:55:33Z)
• Controlling the Latent Space of GANs through Reinforcement Learning: A Case Study on Task-based Image-to-Image Translation [5.881800919492065]
  Generative Adversarial Networks (GAN) have emerged as a formidable AI tool to generate realistic outputs based on training datasets. We propose a novel methodology to address this issue by integrating a reinforcement learning (RL) agent with a latent-space GAN (l-GAN) We have developed an actor-critic RL agent with a meticulously designed reward policy, enabling it to acquire proficiency in navigating the latent space of the l-GAN.
  arXiv  Detail & Related papers  (2023-07-26T06:34:24Z)
• Human-Timescale Adaptation in an Open-Ended Task Space [56.55530165036327]
  We show that training an RL agent at scale leads to a general in-context learning algorithm that can adapt to open-ended novel embodied 3D problems as quickly as humans. Our results lay the foundation for increasingly general and adaptive RL agents that perform well across ever-larger open-ended domains.
  arXiv  Detail & Related papers  (2023-01-18T15:39:21Z)
• On Transforming Reinforcement Learning by Transformer: The Development Trajectory [97.79247023389445]
  Transformer, originally devised for natural language processing, has also attested significant success in computer vision. We group existing developments in two categories: architecture enhancement and trajectory optimization. We examine the main applications of TRL in robotic manipulation, text-based games, navigation and autonomous driving.
  arXiv  Detail & Related papers  (2022-12-29T03:15:59Z)
• FLAG3D: A 3D Fitness Activity Dataset with Language Instruction [89.60371681477791]
  We present FLAG3D, a large-scale 3D fitness activity dataset with language instruction containing 180K sequences of 60 categories. We show that FLAG3D contributes great research value for various challenges, such as cross-domain human action recognition, dynamic human mesh recovery, and language-guided human action generation.
  arXiv  Detail & Related papers  (2022-12-09T02:33:33Z)
• Partial Observability during DRL for Robot Control [6.181642248900806]
  We investigate partial observability as a potential failure source of applying Deep Reinforcement Learning to robot control tasks. We compare the performance of three common DRL algorithms, TD3, SAC and PPO under various partial observability conditions. We find that TD3 and SAC become easily stuck in local optima and underperform PPO.
  arXiv  Detail & Related papers  (2022-09-12T03:12:04Z)
• Multitask Adaptation by Retrospective Exploration with Learned World Models [77.34726150561087]
  We propose a meta-learned addressing model called RAMa that provides training samples for the MBRL agent taken from task-agnostic storage. The model is trained to maximize the expected agent's performance by selecting promising trajectories solving prior tasks from the storage.
  arXiv  Detail & Related papers  (2021-10-25T20:02:57Z)
• ST3D: Self-training for Unsupervised Domain Adaptation on 3D ObjectDetection [78.71826145162092]
  We present a new domain adaptive self-training pipeline, named ST3D, for unsupervised domain adaptation on 3D object detection from point clouds. Our ST3D achieves state-of-the-art performance on all evaluated datasets and even surpasses fully supervised results on KITTI 3D object detection benchmark.
  arXiv  Detail & Related papers  (2021-03-09T10:51:24Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.