Isaac Gym: High Performance GPU-Based Physics Simulation For Robot Learning

• URL: http://arxiv.org/abs/2108.10470v2
• Date: Wed, 25 Aug 2021 23:42:59 GMT
• Title: Isaac Gym: High Performance GPU-Based Physics Simulation For Robot Learning
• Abstract summary: Isaac Gym offers a high performance learning platform to train policies for wide variety of robotics tasks on GPU. Both physics simulation and the neural network policy training training training communicate by directly passing data directly to PyTorch. This leads to blazing fast training times for complex robotics tasks on a single GPU with 2-3 orders of magnitude improvements.
• Score: 5.786157611032947
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Isaac Gym offers a high performance learning platform to train policies for wide variety of robotics tasks directly on GPU. Both physics simulation and the neural network policy training reside on GPU and communicate by directly passing data from physics buffers to PyTorch tensors without ever going through any CPU bottlenecks. This leads to blazing fast training times for complex robotics tasks on a single GPU with 2-3 orders of magnitude improvements compared to conventional RL training that uses a CPU based simulator and GPU for neural networks. We host the results and videos at \url{https://sites.google.com/view/isaacgym-nvidia} and isaac gym can be downloaded at \url{https://developer.nvidia.com/isaac-gym}.

Related papers

• UniLab: A Heterogeneous Architecture for Robot RL Beyond GPU-Dominant Paradigms [68.40808444348735]
  In simulation-dominated robot control, the essential question is not which processor runs physics, but whether simulation throughput, policy learning, and runtime synchronization form an efficient end-to-end loop.<n>We present UniLab, a heterogeneous CPU-resident / GPU-learning architecture that decouples CPU-parallel simulation from GPU policy updates through a unified runtime for data movement, buffering, and synchronization.
  arXiv  Detail & Related papers  (2026-05-28T17:53:50Z)
• GeoPT: Scaling Physics Simulation via Lifted Geometric Pre-Training [86.70824679370524]
  We present GeoPT, a unified pre-trained model for general physics simulation based on lifted geometric pre-training.<n>The core idea is to augment geometry with synthetic dynamics, enabling dynamics-aware self-supervision without physics labels.
  arXiv  Detail & Related papers  (2026-02-23T22:32:08Z)
• Isaac Lab: A GPU-Accelerated Simulation Framework for Multi-Modal Robot Learning [72.43357471969564]
  Isaac Lab combines high-fidelity GPU parallel physics, rendering, and a modular, composable architecture for designing environments and training robot policies.<n>We highlight its application to a diverse set of challenges, including whole-body control, cross-embodiment mobility, contact-rich and dexterous manipulation, and the integration of human demonstrations for skill acquisition.<n>We believe Isaac Lab's combination of advanced simulation capabilities, rich sensing, and data-center scale execution will help unlock the next generation of breakthroughs in robotics research.
  arXiv  Detail & Related papers  (2025-11-06T21:43:02Z)
• GaussGym: An open-source real-to-sim framework for learning locomotion from pixels [78.05453137978132]
  We present a novel approach for photorealistic robot simulation that integrates 3D Gaussian Splatting as a drop-in within vectorized physics simulators.<n>This enables unprecedented speed -- exceeding 100,000 steps per second on consumer GPU.<n>We additionally demonstrate its applicability in a sim-to-real robotics setting.
  arXiv  Detail & Related papers  (2025-10-17T06:34:52Z)
• ManiSkill3: GPU Parallelized Robotics Simulation and Rendering for Generalizable Embodied AI [27.00155119759743]
  ManiSkill3 is the fastest state-visual GPU parallelized robotics simulator with contact-rich physics targeting generalizable manipulation. ManiSkill3 supports GPU parallelization of many aspects including simulation+rendering, heterogeneous simulation, pointclouds/voxels visual input, and more.
  arXiv  Detail & Related papers  (2024-10-01T06:10:39Z)
• Flash3D: Feed-Forward Generalisable 3D Scene Reconstruction from a Single Image [80.48452783328995]
  Flash3D is a method for scene reconstruction and novel view synthesis from a single image. For generalisability, we start from a "foundation" model for monocular depth estimation. For efficiency, we base this extension on feed-forward Gaussian Splatting.
  arXiv  Detail & Related papers  (2024-06-06T17:59:56Z)
• Training a Large Video Model on a Single Machine in a Day [5.247398948623659]
  We show how to train a state-of-the-art video model on a single machine with eight consumer-grade GPUs in a day. We identify three bottlenecks, IO, CPU, and GPU, and optimize each. For comparable architectures, our pipeline achieves higher accuracies with $frac18$ of the computation compared to prior work.
  arXiv  Detail & Related papers  (2023-09-28T17:59:50Z)
• Comparative Analysis of CPU and GPU Profiling for Deep Learning Models [0.0]
  This paper presents the time and memory allocation of CPU and GPU while training deep neural networks using Pytorch. For a simpler network, there are not many significant improvements in GPU over the CPU.
  arXiv  Detail & Related papers  (2023-09-05T18:22:11Z)
• FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPU [89.2451963569343]
  FlexGen is a generation engine for running large language model (LLM) inference on a single commodity GPU. When running OPT-175B on a single 16GB GPU, FlexGen achieves significantly higher throughput compared to state-of-the-art offloading systems. On the HELM benchmark, FlexGen can benchmark a 30B model with a 16GB GPU on 7 representative sub-scenarios in 21 hours.
  arXiv  Detail & Related papers  (2023-03-13T05:19:28Z)
• Optimizing Data Collection in Deep Reinforcement Learning [4.9709347068704455]
  GPU vectorization can achieve up to $1024times$ speedup over commonly used CPU simulators. We show that simulator kernel fusion speedups with a simple simulator are $11.3times$ and increase by up to $1024times$ as simulator complexity increases in terms of memory bandwidth requirements.
  arXiv  Detail & Related papers  (2022-07-15T20:22:31Z)
• VRKitchen2.0-IndoorKit: A Tutorial for Augmented Indoor Scene Building in Omniverse [77.52012928882928]
  INDOORKIT is a built-in toolkit for NVIDIA OMNIVERSE. It provides flexible pipelines for indoor scene building, scene randomizing, and animation controls.
  arXiv  Detail & Related papers  (2022-06-23T17:53:33Z)
• ElegantRL-Podracer: Scalable and Elastic Library for Cloud-Native Deep Reinforcement Learning [141.58588761593955]
  We present a library ElegantRL-podracer for cloud-native deep reinforcement learning. It efficiently supports millions of cores to carry out massively parallel training at multiple levels. At a low-level, each pod simulates agent-environment interactions in parallel by fully utilizing nearly 7,000 GPU cores in a single GPU.
  arXiv  Detail & Related papers  (2021-12-11T06:31:21Z)
• Megaverse: Simulating Embodied Agents at One Million Experiences per Second [75.1191260838366]
  We present Megaverse, a new 3D simulation platform for reinforcement learning and embodied AI research. Megaverse is up to 70x faster than DeepMind Lab in fully-shaded 3D scenes with interactive objects. We use Megaverse to build a new benchmark that consists of several single-agent and multi-agent tasks.
  arXiv  Detail & Related papers  (2021-07-17T03:16:25Z)
• Large Batch Simulation for Deep Reinforcement Learning [101.01408262583378]
  We accelerate deep reinforcement learning-based training in visually complex 3D environments by two orders of magnitude over prior work. We realize end-to-end training speeds of over 19,000 frames of experience per second on a single and up to 72,000 frames per second on a single eight- GPU machine. By combining batch simulation and performance optimizations, we demonstrate that Point navigation agents can be trained in complex 3D environments on a single GPU in 1.5 days to 97% of the accuracy of agents trained on a prior state-of-the-art system.
  arXiv  Detail & Related papers  (2021-03-12T00:22:50Z)
• Computational Performance Predictions for Deep Neural Network Training: A Runtime-Based Approach [1.5857983167543392]
  We present a new practical technique to help users make informed and cost-efficient GPU selections. We make predictions by scaling the execution time of each operation in a training iteration from one GPU to another using either (i) wave scaling, a technique based on a GPU's execution model, or (ii) pre-trained multilayer perceptrons. We implement our technique into a Python library called Surfer and find that it makes accurate iteration execution time predictions on ResNet-50, Inception v3, the Transformer, GNMT, and DCGAN.
  arXiv  Detail & Related papers  (2021-01-31T20:17:46Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.