Exosense: A Vision-Centric Scene Understanding System For Safe Exoskeleton Navigation

• URL: http://arxiv.org/abs/2403.14320v1
• Date: Thu, 21 Mar 2024 11:41:39 GMT
• Title: Exosense: A Vision-Centric Scene Understanding System For Safe Exoskeleton Navigation
• Abstract summary: Exosense is a vision-centric scene understanding system. It generates rich, globally-consistent elevation maps, incorporating both semantic and terrain traversability information. We demonstrate the system's robustness to the challenges of typical periodic walking gaits.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Exoskeletons for daily use by those with mobility impairments are being developed. They will require accurate and robust scene understanding systems. Current research has used vision to identify immediate terrain and geometric obstacles, however these approaches are constrained to detections directly in front of the user and are limited to classifying a finite range of terrain types (e.g., stairs, ramps and level-ground). This paper presents Exosense, a vision-centric scene understanding system which is capable of generating rich, globally-consistent elevation maps, incorporating both semantic and terrain traversability information. It features an elastic Atlas mapping framework associated with a visual SLAM pose graph, embedded with open-vocabulary room labels from a Vision-Language Model (VLM). The device's design includes a wide field-of-view (FoV) fisheye multi-camera system to mitigate the challenges introduced by the exoskeleton walking pattern. We demonstrate the system's robustness to the challenges of typical periodic walking gaits, and its ability to construct accurate semantically-rich maps in indoor settings. Additionally, we showcase its potential for motion planning -- providing a step towards safe navigation for exoskeletons.

Related papers

• SPOT: Spatial Perception-Oriented Long-Horizon Humanoid Teleoperation [35.4182413141688]
  SPOT combines a robot-mounted fisheye camera, a wide-field stereoscopic display, viewpoint-decoupled free-looking, and visual stabilization.<n>We evaluate SPOT on perception-critical humanoid data-collection tasks spanning drop recovery, peripheral retrieval, large-workspace bimanual manipulation, fine alignment, and dynamic interaction.
  arXiv  Detail & Related papers  (2026-09-07T19:46:56Z)
• ACE-Data-0: Human-Centric Ambient Capture as Embodied Data Engine [83.02804728003862]
  Ambient Capture Engine (ACE) transforms real home environments into spatially, temporally synchronized recording studios.<n>ACE records egocentric and multi-view exocentric video, full-body and articulated hand motion, object geometry and 6-DoF trajectories, audio, and signals as a unified multisensory stream.<n>Dataset spans atomic manipulation, long-horizon chains of household activities, and human-scene interaction, while preserving natural behavioral variation through goal-level rather than step-by-step instructions.
  arXiv  Detail & Related papers  (2026-07-30T17:59:50Z)
• EgoHTR: Egocentric 4D Demonstrations of Human Terrain Traversal [63.906929035961134]
  We open-source a reconstruction pipeline capturing 55 scene-aligned 4D human motion sequences in diverse, complex environments.<n>The resulting dataset comprises over 150k frames, which we evaluate against motion-capture ground truth.<n>Our pipeline enables community-driven dataset extensions and factors the problem to help researchers build context-aware robots.
  arXiv  Detail & Related papers  (2026-07-15T06:02:41Z)
• GRAIL: Generating Humanoid Loco-Manipulation from 3D Assets and Video Priors [113.71148915419246]
  We present GRAIL, a digital generation pipeline that composes 3D assets, simulator-ready scenes, and priors from video foundation models (VFMs) to synthesize interactions without rebuilding physical environments or teleoperating the robot.<n>GRAIL starts from fully specified 3D configurations in which object geometry, camera parameters, metric scale, environment depth, and a robot-proportioned character are known before video generation and reused during reconstruction.<n>We retarget the recovered motions to a humanoid robot and train complementary task-general morphology trackers.<n>GRAIL produces over 20,000 sequences spanning pick-up, object manipulation, sitting
  arXiv  Detail & Related papers  (2026-06-03T17:57:45Z)
• EmbodMocap: In-the-Wild 4D Human-Scene Reconstruction for Embodied Agents [85.77432303199176]
  We propose EmbodMocap, a portable and affordable data collection pipeline using two moving iPhones.<n>Our key idea is to jointly calibrate dual RGB-D sequences to reconstruct both humans and scenes.<n>Based on the collected data, we empower three embodied AI tasks: monocular human-scene-reconstruction, where we fine-tune feedforward models that output metric-scale, world-space aligned humans and scenes; physics-based character animation, where we prove our data could be used to scale human-object interaction skills and scene-aware motion tracking; and robot motion control, where we train a humanoid robot via
  arXiv  Detail & Related papers  (2026-02-26T16:53:41Z)
• LT-Exosense: A Vision-centric Multi-session Mapping System for Lifelong Safe Navigation of Exoskeletons [2.3696699717752314]
  LT-Exosense is a vision-centric, multi-session mapping system designed to support long-term (semi)-autonomous navigation for exoskeleton users.<n>We demonstrate a scalable multi-session map that achieves an average point-to-point error below 5 cm when compared to ground-truth laser scans.
  arXiv  Detail & Related papers  (2025-10-25T05:23:50Z)
• HumanoidVerse: A Versatile Humanoid for Vision-Language Guided Multi-Object Rearrangement [51.16740261131198]
  We introduce HumanoidVerse, a novel framework for vision-language guided humanoid control.<n>HumanoidVerse supports consecutive manipulation of multiple objects, guided only by natural language instructions and egocentric camera RGB observations.<n>Our work represents a key step toward robust, general-purpose humanoid agents capable of executing complex, sequential tasks under real-world sensory constraints.
  arXiv  Detail & Related papers  (2025-08-23T08:23:14Z)
• Let Humanoids Hike! Integrative Skill Development on Complex Trails [39.30624277966043]
  We propose training humanoids to hike on complex trails, driving integrative skill development across visual perception, decision making, and motor execution.<n>We develop a learning framework, LEGO-H, that enables a vision-equipped humanoid robot to hike complex trails autonomously.
  arXiv  Detail & Related papers  (2025-05-09T17:53:02Z)
• Reasoning in visual navigation of end-to-end trained agents: a dynamical systems approach [23.52028824411467]
  We present a large-scale experimental study involving numepisodes navigation episodes in a real environment with a physical robot. We analyze the type of reasoning emerging from end-to-end training. We show in a post-hoc analysis that the value function learned by the agent relates to long-term planning.
  arXiv  Detail & Related papers  (2025-03-11T11:16:47Z)
• SPIN: Simultaneous Perception, Interaction and Navigation [33.408010508592824]
  We present a reactive mobile manipulation framework that uses an active visual system to consciously perceive and react to its environment. Similar to how humans leverage whole-body and hand-eye coordination, we develop a mobile manipulator that exploits its ability to move and see.
  arXiv  Detail & Related papers  (2024-05-13T17:59:36Z)
• EgoNav: Egocentric Scene-aware Human Trajectory Prediction [15.346096596482857]
  Wearable collaborative robots stand to assist human wearers who need fall prevention assistance or wear exoskeletons. Such a robot needs to be able to constantly adapt to the surrounding scene based on egocentric vision, and predict the ego motion of the wearer. In this work, we leveraged body-mounted cameras and sensors to anticipate the trajectory of human wearers through complex surroundings.
  arXiv  Detail & Related papers  (2024-03-27T21:43:12Z)
• Coupling Vision and Proprioception for Navigation of Legged Robots [65.59559699815512]
  We exploit the complementary strengths of vision and proprioception to achieve point goal navigation in a legged robot. We show superior performance compared to wheeled robot (LoCoBot) baselines. We also show the real-world deployment of our system on a quadruped robot with onboard sensors and compute.
  arXiv  Detail & Related papers  (2021-12-03T18:59:59Z)
• Augmented reality navigation system for visual prosthesis [67.09251544230744]
  We propose an augmented reality navigation system for visual prosthesis that incorporates a software of reactive navigation and path planning. It consists on four steps: locating the subject on a map, planning the subject trajectory, showing it to the subject and re-planning without obstacles. Results show how our augmented navigation system help navigation performance by reducing the time and distance to reach the goals, even significantly reducing the number of obstacles collisions.
  arXiv  Detail & Related papers  (2021-09-30T09:41:40Z)
• Learning Perceptual Locomotion on Uneven Terrains using Sparse Visual Observations [75.60524561611008]
  This work aims to exploit the use of sparse visual observations to achieve perceptual locomotion over a range of commonly seen bumps, ramps, and stairs in human-centred environments. We first formulate the selection of minimal visual input that can represent the uneven surfaces of interest, and propose a learning framework that integrates such exteroceptive and proprioceptive data. We validate the learned policy in tasks that require omnidirectional walking over flat ground and forward locomotion over terrains with obstacles, showing a high success rate.
  arXiv  Detail & Related papers  (2021-09-28T20:25:10Z)
• Rapid Exploration for Open-World Navigation with Latent Goal Models [78.45339342966196]
  We describe a robotic learning system for autonomous exploration and navigation in diverse, open-world environments. At the core of our method is a learned latent variable model of distances and actions, along with a non-parametric topological memory of images. We use an information bottleneck to regularize the learned policy, giving us (i) a compact visual representation of goals, (ii) improved generalization capabilities, and (iii) a mechanism for sampling feasible goals for exploration.
  arXiv  Detail & Related papers  (2021-04-12T23:14:41Z)
• Robot Perception enables Complex Navigation Behavior via Self-Supervised Learning [23.54696982881734]
  We propose an approach to unify successful robot perception systems for active target-driven navigation tasks via reinforcement learning (RL) Our method temporally incorporates compact motion and visual perception data, directly obtained using self-supervision from a single image sequence. We demonstrate our approach on two real-world driving dataset, KITTI and Oxford RobotCar, using the new interactive CityLearn framework.
  arXiv  Detail & Related papers  (2020-06-16T07:45:47Z)
• VisualEchoes: Spatial Image Representation Learning through Echolocation [97.23789910400387]
  Several animal species (e.g., bats, dolphins, and whales) and even visually impaired humans have the remarkable ability to perform echolocation. We propose a novel interaction-based representation learning framework that learns useful visual features via echolocation. Our work opens a new path for representation learning for embodied agents, where supervision comes from interacting with the physical world.
  arXiv  Detail & Related papers  (2020-05-04T16:16:58Z)
• An Exploration of Embodied Visual Exploration [97.21890864063872]
  Embodied computer vision considers perception for robots in novel, unstructured environments. We present a taxonomy for existing visual exploration algorithms and create a standard framework for benchmarking them. We then perform a thorough empirical study of the four state-of-the-art paradigms using the proposed framework.
  arXiv  Detail & Related papers  (2020-01-07T17:40:32Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.