CReF: Cross-modal and Recurrent Fusion for Depth-conditioned Humanoid Locomotion

• URL: http://arxiv.org/abs/2603.29452v2
• Date: Wed, 01 Apr 2026 02:09:17 GMT
• Title: CReF: Cross-modal and Recurrent Fusion for Depth-conditioned Humanoid Locomotion
• Abstract summary: We propose CReF (Cross-modal and Recurrent Fusion), a single-stage depth-conditioned humanoid locomotion framework.<n>CReF learns locomotion-relevant features directly from raw forward-facing depth without explicit geometric intermediates.
• Score: 6.2747272941647045
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Stable traversal over geometrically complex terrain increasingly requires exteroceptive perception, yet prior perceptive humanoid locomotion methods often remain tied to explicit geometric abstractions, either by mediating control through robot-centric 2.5D terrain representations or by shaping depth learning with auxiliary geometry-related targets. Such designs inherit the representational bias of the intermediate or supervisory target and can be restrictive for vertical structures, perforated obstacles, and complex real-world clutter. We propose CReF (Cross-modal and Recurrent Fusion), a single-stage depth-conditioned humanoid locomotion framework that learns locomotion-relevant features directly from raw forward-facing depth without explicit geometric intermediates. CReF couples proprioception and depth tokens through proprioception-queried cross-modal attention, fuses the resulting representation with a gated residual fusion block, and performs temporal integration with a Gated Recurrent Unit (GRU) regulated by a highway-style output gate for state-dependent blending of recurrent and feedforward features. To further improve terrain interaction, we introduce a terrain-aware foothold placement reward that extracts supportable foothold candidates from foot-end point-cloud samples and rewards touchdown locations that lie close to the nearest supportable candidate. Experiments in simulation and on a physical humanoid demonstrate robust traversal over diverse terrains and effective zero-shot transfer to real-world scenes containing handrails, hollow pallet assemblies, severe reflective interference, and visually cluttered outdoor surroundings.

Related papers

• RASA: Disentangled Spatial-Motional Priors for Cross-Identity Character Animation [85.41886214825894]
  Cross-identity character animation aims to drive a target identity from a reference image to follow the motion of a source character from a driving video.<n>We introduce Reference-Aware Structural Alignment (RASA), a framework that disentangles spatial mapping from motion control.<n>Our work establishes a new paradigm showing disentangled spatial and motional priors are key to robust character animation.
  arXiv  Detail & Related papers  (2026-08-28T11:36:44Z)
• EgoHTR: Egocentric 4D Demonstrations of Human Terrain Traversal [63.906929035961134]
  We open-source a reconstruction pipeline capturing 55 scene-aligned 4D human motion sequences in diverse, complex environments.<n>The resulting dataset comprises over 150k frames, which we evaluate against motion-capture ground truth.<n>Our pipeline enables community-driven dataset extensions and factors the problem to help researchers build context-aware robots.
  arXiv  Detail & Related papers  (2026-07-15T06:02:41Z)
• AutoPath: Learning Transferable Goal-Conditioned Stochastic Path Prior for Safe Navigation Without Human Demonstrations [103.61864552260316]
  Real-time navigation in dynamic environments requires collision-free and dynamically feasible motion under limited perception.<n>We formulate navigation as learning a transferable goal-aligned path prior that models a reusable distribution over goal-aligned geometry-consistent local paths conditioned on local observations.<n>We develop a structured prior learning framework that parameterizes local paths using a geometry-aware polar action manifold and incorporates risk-sensitive utility shaping with multi-goal distributional rollouts for stable and safety-aware planning.
  arXiv  Detail & Related papers  (2026-07-13T16:00:41Z)
• Cross-Vehicle 3D Geometric Consistency for Self-Supervised Surround Depth Estimation on Articulated Vehicles [3.512352303010361]
  We propose a self-supervised framework for surround-view depth estimation on articulated vehicles.<n>We introduce multi-view spatial context enrichment strategy and a cross-view surface normal constraint to improve structural coherence.<n>To validate our proposed method, an articulated vehicle experiment platform was established with a dataset collected over it.
  arXiv  Detail & Related papers  (2026-04-03T02:10:11Z)
• PRIOR: Perceptive Learning for Humanoid Locomotion with Reference Gait Priors [12.48639232018629]
  PRIOR is an efficient and reproducible framework built on Isaac Lab.<n>We present a framework that achieves robust terrain traversal with human-like gaits.<n>We will open-source the complete PRIOR framework, including the training pipeline, parametric gait generator, and evaluation benchmarks.
  arXiv  Detail & Related papers  (2026-03-19T14:42:07Z)
• GeoLoco: Leveraging 3D Geometric Priors from Visual Foundation Model for Robust RGB-Only Humanoid Locomotion [17.719701533574412]
  GeoLoco conceptualizes monocular images as high-dimensional 3D latent representations by harnessing the powerful geometric priors of a frozen, scale-aware Visual Foundation Model (VFM)<n>GeoLoco achieves robust zero-shot transfer to the Unitree G1 humanoid and successfully negotiates challenging terrains.
  arXiv  Detail & Related papers  (2026-03-08T13:18:27Z)
• Vision-Guided MPPI for Agile Drone Racing: Navigating Arbitrary Gate Poses via Neural Signed Distance Fields [8.66703842496129]
  Drone racing requires tight coupling of perception, planning, and control under extreme agility.<n>Recent approaches rely on precomputed spatial reference trajectories or explicit 6-DoF gate pose estimation.<n>We propose a vision guided optimal control framework that enables reference-free agile flight through arbitrarily placed and oriented gates.
  arXiv  Detail & Related papers  (2026-03-07T13:18:07Z)
• MeshMimic: Geometry-Aware Humanoid Motion Learning through 3D Scene Reconstruction [54.36564144414704]
  MeshMimic is an innovative framework that bridges 3D scene reconstruction and embodied intelligence to enable humanoid robots to learn coupled "motion-terrain" interactions directly from video.<n>By leveraging state-of-the-art 3D vision models, our framework precisely segments and reconstructs both human trajectories and the underlying 3D geometry of terrains and objects.
  arXiv  Detail & Related papers  (2026-02-17T17:09:45Z)
• Now You See That: Learning End-to-End Humanoid Locomotion from Raw Pixels [1.3015553594945661]
  We present an end-to-end framework for vision-driven humanoid locomotion.<n>For robust sim-to-real transfer, we develop a high-fidelity depth sensor simulation.<n>We propose a vision-aware behavior distillation approach that combines latent space alignment with noise-invariant auxiliary tasks.
  arXiv  Detail & Related papers  (2026-02-06T04:34:20Z)
• START: Traversing Sparse Footholds with Terrain Reconstruction [11.632418196552324]
  We propose START, a single-stage learning framework that enables agile, stable locomotion on highly sparse footholds.<n> START leverages only low-cost onboard vision and proprioception to accurately reconstruct local terrain heightmap.<n> Experimental results demonstrate that START achieves zero-shot transfer across diverse real-world scenarios.
  arXiv  Detail & Related papers  (2025-12-15T10:02:41Z)
• Gait-Adaptive Perceptive Humanoid Locomotion with Real-Time Under-Base Terrain Reconstruction [86.53248703859718]
  We introduce a perceptive locomotion framework that merges terrain sensing, gait regulation, and whole-body control into a single reinforcement learning policy.<n>Experiments conducted on a 31-DoF, 1.65 m humanoid robot demonstrate robust locomotion in both simulation and real-world settings.
  arXiv  Detail & Related papers  (2025-12-08T11:42:41Z)
• HOSIG: Full-Body Human-Object-Scene Interaction Generation with Hierarchical Scene Perception [57.37135310143126]
  HO SIG is a novel framework for synthesizing full-body interactions through hierarchical scene perception.<n>Our framework supports unlimited motion length through autoregressive generation and requires minimal manual intervention.<n>This work bridges the critical gap between scene-aware navigation and dexterous object manipulation.
  arXiv  Detail & Related papers  (2025-06-02T12:08:08Z)
• ZoomNeXt: A Unified Collaborative Pyramid Network for Camouflaged Object Detection [70.11264880907652]
  Recent object (COD) attempts to segment objects visually blended into their surroundings, which is extremely complex and difficult in real-world scenarios. We propose an effective unified collaborative pyramid network that mimics human behavior when observing vague images and camouflaged zooming in and out. Our framework consistently outperforms existing state-of-the-art methods in image and video COD benchmarks.
  arXiv  Detail & Related papers  (2023-10-31T06:11:23Z)
• Aligning Silhouette Topology for Self-Adaptive 3D Human Pose Recovery [70.66865453410958]
  Articulation-centric 2D/3D pose supervision forms the core training objective in most existing 3D human pose estimation techniques. We propose a novel framework that relies only on silhouette supervision to adapt a source-trained model-based regressor. We develop a series of convolution-friendly spatial transformations in order to disentangle a topological-skeleton representation from the raw silhouette.
  arXiv  Detail & Related papers  (2022-04-04T06:58:15Z)
• Risk-Averse MPC via Visual-Inertial Input and Recurrent Networks for Online Collision Avoidance [95.86944752753564]
  We propose an online path planning architecture that extends the model predictive control (MPC) formulation to consider future location uncertainties. Our algorithm combines an object detection pipeline with a recurrent neural network (RNN) which infers the covariance of state estimates. The robustness of our methods is validated on complex quadruped robot dynamics and can be generally applied to most robotic platforms.
  arXiv  Detail & Related papers  (2020-07-28T07:34:30Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.