Now You See That: Learning End-to-End Humanoid Locomotion from Raw Pixels

• URL: http://arxiv.org/abs/2602.06382v1
• Date: Fri, 06 Feb 2026 04:34:20 GMT
• Title: Now You See That: Learning End-to-End Humanoid Locomotion from Raw Pixels
• Abstract summary: We present an end-to-end framework for vision-driven humanoid locomotion.<n>For robust sim-to-real transfer, we develop a high-fidelity depth sensor simulation.<n>We propose a vision-aware behavior distillation approach that combines latent space alignment with noise-invariant auxiliary tasks.
• Score: 1.3015553594945661
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Achieving robust vision-based humanoid locomotion remains challenging due to two fundamental issues: the sim-to-real gap introduces significant perception noise that degrades performance on fine-grained tasks, and training a unified policy across diverse terrains is hindered by conflicting learning objectives. To address these challenges, we present an end-to-end framework for vision-driven humanoid locomotion. For robust sim-to-real transfer, we develop a high-fidelity depth sensor simulation that captures stereo matching artifacts and calibration uncertainties inherent in real-world sensing. We further propose a vision-aware behavior distillation approach that combines latent space alignment with noise-invariant auxiliary tasks, enabling effective knowledge transfer from privileged height maps to noisy depth observations. For versatile terrain adaptation, we introduce terrain-specific reward shaping integrated with multi-critic and multi-discriminator learning, where dedicated networks capture the distinct dynamics and motion priors of each terrain type. We validate our approach on two humanoid platforms equipped with different stereo depth cameras. The resulting policy demonstrates robust performance across diverse environments, seamlessly handling extreme challenges such as high platforms and wide gaps, as well as fine-grained tasks including bidirectional long-term staircase traversal.

Related papers

• DynaWM: Dynamics-Aware Distillation with World Model and Momentum Targets for Smooth Locomotion over Continuous Stairs [8.383969829401517]
  We propose DynaWM, a dynamics-aware representation learning framework.<n>We introduce a world model as a regularizer to enforce forward-dynamics awareness, preserving comprehensive terrain geometry.<n>We employ a momentum target encoder to provide consistent distillation targets, preventing dimensional collapse from non-stationary teacher updates.
  arXiv  Detail & Related papers  (2026-06-23T03:07:42Z)
• TAGA: Terrain-aware Active Gaze Learning for Generalizable Agile Humanoid Locomotion [13.927597120628064]
  We introduce TAGA, a Terrain-aware Active Gaze learning framework for Attention-based humanoid control.<n>Our framework guides the model to learn anticipatory cues and actively attend to specific areas of the height scan.<n>We find that such gaze behaviors can naturally emerge through reinforcement learning alone, without requiring additional supervision or explicit guidance.
  arXiv  Detail & Related papers  (2026-06-04T08:52:56Z)
• DreamVideo-Omni: Omni-Motion Controlled Multi-Subject Video Customization with Latent Identity Reinforcement Learning [82.28800081483224]
  We present DreamVideo- Omni, a unified framework enabling multisubject customization with omni-motion control.<n>To resolve multi-subject ambiguity, we introduce group and role embeddings to explicitly anchor motion signals to specific identities.<n>This provides motion-aware identity rewards in the latent space, prioritizing identity preservation aligned with human preferences.
  arXiv  Detail & Related papers  (2026-03-12T17:59:12Z)
• Revisiting Multi-Task Visual Representation Learning [52.93947931352643]
  We introduce MTV, a principled multi-task visual pretraining framework.<n>We leverage high-capacity "expert" models to synthesize dense, structured pseudo-labels at scale.<n>Our results demonstrate that MTV achieves "best-of-both-worlds" performance.
  arXiv  Detail & Related papers  (2026-01-20T11:59:19Z)
• Head Anchor Enhanced Detection and Association for Crowded Pedestrian Tracking [8.653608112604472]
  The proposed method incorporates detection features from both the regression and classification branches of an object detector.<n>In terms of motion modeling, we propose an iterative Kalman filtering approach designed to align with modern detector assumptions.
  arXiv  Detail & Related papers  (2025-08-07T15:47:34Z)
• DINO-CoDT: Multi-class Collaborative Detection and Tracking with Vision Foundation Models [11.34839442803445]
  We propose a multi-class collaborative detection and tracking framework tailored for diverse road users.<n>We first present a detector with a global spatial attention fusion (GSAF) module, enhancing multi-scale feature learning for objects of varying sizes.<n>Next, we introduce a tracklet RE-IDentification (REID) module that leverages visual semantics with a vision foundation model to effectively reduce ID SWitch (IDSW) errors.
  arXiv  Detail & Related papers  (2025-06-09T02:49:10Z)
• Diffeomorphic Obstacle Avoidance for Contractive Dynamical Systems via Implicit Representations [0.6144680854063939]
  This paper addresses the challenge of ensuring both safety and robustness in dynamic robot skills learned from demonstrations.<n>We build on neural contractive dynamical systems to provide robust extrapolation of the learned skills.<n>We design a full-body obstacle avoidance strategy that preserves contraction stability via diffeomorphic transforms.
  arXiv  Detail & Related papers  (2025-04-26T08:56:51Z)
• Dita: Scaling Diffusion Transformer for Generalist Vision-Language-Action Policy [73.75271615101754]
  We present Dita, a scalable framework that leverages Transformer architectures to directly denoise continuous action sequences.<n>Dita employs in-context conditioning -- enabling fine-grained alignment between denoised actions and raw visual tokens from historical observations.<n>Dita effectively integrates cross-embodiment datasets across diverse camera perspectives, observation scenes, tasks, and action spaces.
  arXiv  Detail & Related papers  (2025-03-25T15:19:56Z)
• Ground-level Viewpoint Vision-and-Language Navigation in Continuous Environments [10.953629652228024]
  Vision-and-Language Navigation (VLN) agents associate time-sequenced visual observations with corresponding instructions to make decisions.<n>In this paper, we address the mismatch between human-centric instructions and quadruped robots with a low-height field of view.<n>We propose a Ground-level Viewpoint Navigation (GVNav) approach to mitigate this issue.
  arXiv  Detail & Related papers  (2025-02-26T10:30:40Z)
• Open-World Drone Active Tracking with Goal-Centered Rewards [62.21394499788672]
  Drone Visual Active Tracking aims to autonomously follow a target object by controlling the motion system based on visual observations.<n>We propose DAT, the first open-world drone active air-to-ground tracking benchmark.<n>We also propose GC-VAT, which aims to improve the performance of drone tracking targets in complex scenarios.
  arXiv  Detail & Related papers  (2024-12-01T09:37:46Z)
• ZoomNeXt: A Unified Collaborative Pyramid Network for Camouflaged Object Detection [70.11264880907652]
  Recent object (COD) attempts to segment objects visually blended into their surroundings, which is extremely complex and difficult in real-world scenarios. We propose an effective unified collaborative pyramid network that mimics human behavior when observing vague images and camouflaged zooming in and out. Our framework consistently outperforms existing state-of-the-art methods in image and video COD benchmarks.
  arXiv  Detail & Related papers  (2023-10-31T06:11:23Z)
• Learning Perceptual Locomotion on Uneven Terrains using Sparse Visual Observations [75.60524561611008]
  This work aims to exploit the use of sparse visual observations to achieve perceptual locomotion over a range of commonly seen bumps, ramps, and stairs in human-centred environments. We first formulate the selection of minimal visual input that can represent the uneven surfaces of interest, and propose a learning framework that integrates such exteroceptive and proprioceptive data. We validate the learned policy in tasks that require omnidirectional walking over flat ground and forward locomotion over terrains with obstacles, showing a high success rate.
  arXiv  Detail & Related papers  (2021-09-28T20:25:10Z)
• Vision-Based Mobile Robotics Obstacle Avoidance With Deep Reinforcement Learning [49.04274612323564]
  Obstacle avoidance is a fundamental and challenging problem for autonomous navigation of mobile robots. In this paper, we consider the problem of obstacle avoidance in simple 3D environments where the robot has to solely rely on a single monocular camera. We tackle the obstacle avoidance problem as a data-driven end-to-end deep learning approach.
  arXiv  Detail & Related papers  (2021-03-08T13:05:46Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.