VizFlyt: Perception-centric Pedagogical Framework For Autonomous Aerial Robots

• URL: http://arxiv.org/abs/2503.22876v2
• Date: Tue, 01 Apr 2025 22:39:54 GMT
• Title: VizFlyt: Perception-centric Pedagogical Framework For Autonomous Aerial Robots
• Authors: Kushagra Srivastava, Rutwik Kulkarni, Manoj Velmurugan, Nitin J. Sanket,
• Abstract summary: We present VizFlyt, an open-source perception-centric Hardware-In-The-Loop (HITL) photorealistic testing framework for aerial robotics courses.<n>We utilize pose from an external localization system to hallucinate real-time and photorealistic visual sensors using 3D Gaussian Splatting.<n>This enables stress-free testing of autonomy algorithms on aerial robots without the risk of crashing into obstacles.
• Score: 5.669075778114126
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Autonomous aerial robots are becoming commonplace in our lives. Hands-on aerial robotics courses are pivotal in training the next-generation workforce to meet the growing market demands. Such an efficient and compelling course depends on a reliable testbed. In this paper, we present VizFlyt, an open-source perception-centric Hardware-In-The-Loop (HITL) photorealistic testing framework for aerial robotics courses. We utilize pose from an external localization system to hallucinate real-time and photorealistic visual sensors using 3D Gaussian Splatting. This enables stress-free testing of autonomy algorithms on aerial robots without the risk of crashing into obstacles. We achieve over 100Hz of system update rate. Lastly, we build upon our past experiences of offering hands-on aerial robotics courses and propose a new open-source and open-hardware curriculum based on VizFlyt for the future. We test our framework on various course projects in real-world HITL experiments and present the results showing the efficacy of such a system and its large potential use cases. Code, datasets, hardware guides and demo videos are available at https://pear.wpi.edu/research/vizflyt.html

Related papers

• Dexterity from Smart Lenses: Multi-Fingered Robot Manipulation with In-the-Wild Human Demonstrations [52.29884993824894]
  Learning multi-fingered robot policies from humans performing daily tasks in natural environments has long been a grand goal in the robotics community.<n>AINA enables learning multi-fingered policies from data collected by anyone, anywhere, and in any environment using Aria Gen 2 glasses.
  arXiv  Detail & Related papers  (2025-11-20T18:59:02Z)
• GaussGym: An open-source real-to-sim framework for learning locomotion from pixels [78.05453137978132]
  We present a novel approach for photorealistic robot simulation that integrates 3D Gaussian Splatting as a drop-in within vectorized physics simulators.<n>This enables unprecedented speed -- exceeding 100,000 steps per second on consumer GPU.<n>We additionally demonstrate its applicability in a sim-to-real robotics setting.
  arXiv  Detail & Related papers  (2025-10-17T06:34:52Z)
• Robot Control Stack: A Lean Ecosystem for Robot Learning at Scale [11.166320712764465]
  Vision-Language-Action models (VLAs) replace specialized architectures and task-tailored components of expert policies with large-scale data collection and setup-specific fine-tuning.<n>Traditional robotics software frameworks become a bottleneck, while robot simulations offer only limited support for transitioning from and to real-world experiments.<n>We introduce Robot Control Stack (RCS), a lean ecosystem designed from the ground up to support research in robot learning with large-scale generalist policies.
  arXiv  Detail & Related papers  (2025-09-18T13:12:16Z)
• From Tabula Rasa to Emergent Abilities: Discovering Robot Skills via Real-World Unsupervised Quality-Diversity [11.956963115619475]
  We propose Unsupervised Real-world Skill Acquisition (URSA), an extension of Quality-Diversity Actor-Critic (QDAC)<n>URSA enables robots to autonomously discover and master diverse, high-performing skills directly in the real world.<n>Our results establish a new framework for real-world robot learning, representing a significant step toward more autonomous and adaptable robotic systems.
  arXiv  Detail & Related papers  (2025-08-26T16:20:41Z)
• Indoor Localization for Autonomous Robot Navigation [0.0]
  This paper explores using indoor positioning systems (IPSs) for the indoor navigation of an autonomous robot. We developed an A* path-planning algorithm so that our robot could navigate itself using predicted directions. After testing different network structures, our robot was able to successfully navigate corners around 50 percent of the time.
  arXiv  Detail & Related papers  (2025-02-28T05:25:04Z)
• IRASim: Learning Interactive Real-Robot Action Simulators [24.591694756757278]
  We introduce a novel method, IRASim, to generate realistic videos of a robot arm that executes a given action trajectory. To validate the effectiveness of our method, we create a new benchmark, IRASim Benchmark, based on three real-robot datasets. Results show that IRASim outperforms all the baseline methods and is more preferable in human evaluations.
  arXiv  Detail & Related papers  (2024-06-20T17:50:16Z)
• HomeRobot: Open-Vocabulary Mobile Manipulation [107.05702777141178]
  Open-Vocabulary Mobile Manipulation (OVMM) is the problem of picking any object in any unseen environment, and placing it in a commanded location. HomeRobot has two components: a simulation component, which uses a large and diverse curated object set in new, high-quality multi-room home environments; and a real-world component, providing a software stack for the low-cost Hello Robot Stretch.
  arXiv  Detail & Related papers  (2023-06-20T14:30:32Z)
• Robot Learning with Sensorimotor Pre-training [98.7755895548928]
  We present a self-supervised sensorimotor pre-training approach for robotics. Our model, called RPT, is a Transformer that operates on sequences of sensorimotor tokens. We find that sensorimotor pre-training consistently outperforms training from scratch, has favorable scaling properties, and enables transfer across different tasks, environments, and robots.
  arXiv  Detail & Related papers  (2023-06-16T17:58:10Z)
• Knowledge-Driven Robot Program Synthesis from Human VR Demonstrations [16.321053835017942]
  We present a system for automatically generating executable robot control programs from human task demonstrations in virtual reality (VR) We leverage common-sense knowledge and game engine-based physics to semantically interpret human VR demonstrations. We demonstrate our approach in the context of force-sensitive fetch-and-place for a robotic shopping assistant.
  arXiv  Detail & Related papers  (2023-06-05T09:37:53Z)
• Quality-Diversity Optimisation on a Physical Robot Through Dynamics-Aware and Reset-Free Learning [4.260312058817663]
  We build upon the Reset-Free QD (RF-QD) algorithm to learn controllers directly on a physical robot. This method uses a dynamics model, learned from interactions between the robot and the environment, to predict the robot's behaviour. RF-QD also includes a recovery policy that returns the robot to a safe zone when it has walked outside of it, allowing continuous learning.
  arXiv  Detail & Related papers  (2023-04-24T13:24:00Z)
• Self-Improving Robots: End-to-End Autonomous Visuomotor Reinforcement Learning [54.636562516974884]
  In imitation and reinforcement learning, the cost of human supervision limits the amount of data that robots can be trained on. In this work, we propose MEDAL++, a novel design for self-improving robotic systems. The robot autonomously practices the task by learning to both do and undo the task, simultaneously inferring the reward function from the demonstrations.
  arXiv  Detail & Related papers  (2023-03-02T18:51:38Z)
• A Walk in the Park: Learning to Walk in 20 Minutes With Model-Free Reinforcement Learning [86.06110576808824]
  Deep reinforcement learning is a promising approach to learning policies in uncontrolled environments. Recent advancements in machine learning algorithms and libraries combined with a carefully tuned robot controller lead to learning quadruped in only 20 minutes in the real world.
  arXiv  Detail & Related papers  (2022-08-16T17:37:36Z)
• TACTO: A Fast, Flexible and Open-source Simulator for High-Resolution Vision-based Tactile Sensors [8.497185333795477]
  TACTO is a fast, flexible and open-source simulator for vision-based tactile sensors. It can render realistic high-resolution touch readings at hundreds of frames per second. We demonstrate TACTO on a perceptual task, by learning to predict grasp stability using touch from 1 million grasps.
  arXiv  Detail & Related papers  (2020-12-15T17:54:07Z)
• A Framework for Learning Predator-prey Agents from Simulation to Real World [0.0]
  We propose an evolutionary predatorprey robot system which can be generally implemented from simulation to the real world. Both the predators and prey are co-evolved by NeuroEvolution of Augmenting Topologies (NEAT) to learn the expected behaviours. For the convenience of users, the source code and videos of the simulated and real world are published on Github.
  arXiv  Detail & Related papers  (2020-10-29T17:33:38Z)
• LaND: Learning to Navigate from Disengagements [158.6392333480079]
  We present a reinforcement learning approach for learning to navigate from disengagements, or LaND. LaND learns a neural network model that predicts which actions lead to disengagements given the current sensory observation, and then at test time plans and executes actions that avoid disengagements. Our results demonstrate LaND can successfully learn to navigate in diverse, real world sidewalk environments, outperforming both imitation learning and reinforcement learning approaches.
  arXiv  Detail & Related papers  (2020-10-09T17:21:42Z)
• AirSim Drone Racing Lab [56.68291351736057]
  AirSim Drone Racing Lab is a simulation framework for enabling machine learning research in this domain. Our framework enables generation of racing tracks in multiple photo-realistic environments. We used our framework to host a simulation based drone racing competition at NeurIPS 2019.
  arXiv  Detail & Related papers  (2020-03-12T08:06:06Z)
• CRAVES: Controlling Robotic Arm with a Vision-based Economic System [96.56564257199474]
  Training a robotic arm to accomplish real-world tasks has been attracting increasing attention in both academia and industry.<n>This work discusses the role of computer vision algorithms in this field.<n>We present an alternative solution, which uses a 3D model to create a large number of synthetic data.
  arXiv  Detail & Related papers  (2018-12-03T13:28:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.