AME-2: Agile and Generalized Legged Locomotion via Attention-Based Neural Map Encoding

• URL: http://arxiv.org/abs/2601.08485v1
• Date: Tue, 13 Jan 2026 12:18:50 GMT
• Title: AME-2: Agile and Generalized Legged Locomotion via Attention-Based Neural Map Encoding
• Abstract summary: AME-2 is a unified reinforcement learning framework for agile and generalized locomotion.<n>It incorporates a novel attention-based map encoder in the control policy.<n>The pipeline also integrates with parallel simulation so that we can train controllers with online mapping.
• Score: 13.367858405827926
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Achieving agile and generalized legged locomotion across terrains requires tight integration of perception and control, especially under occlusions and sparse footholds. Existing methods have demonstrated agility on parkour courses but often rely on end-to-end sensorimotor models with limited generalization and interpretability. By contrast, methods targeting generalized locomotion typically exhibit limited agility and struggle with visual occlusions. We introduce AME-2, a unified reinforcement learning (RL) framework for agile and generalized locomotion that incorporates a novel attention-based map encoder in the control policy. This encoder extracts local and global mapping features and uses attention mechanisms to focus on salient regions, producing an interpretable and generalized embedding for RL-based control. We further propose a learning-based mapping pipeline that provides fast, uncertainty-aware terrain representations robust to noise and occlusions, serving as policy inputs. It uses neural networks to convert depth observations into local elevations with uncertainties, and fuses them with odometry. The pipeline also integrates with parallel simulation so that we can train controllers with online mapping, aiding sim-to-real transfer. We validate AME-2 with the proposed mapping pipeline on a quadruped and a biped robot, and the resulting controllers demonstrate strong agility and generalization to unseen terrains in simulation and in real-world experiments.

Related papers

• TAGA: Terrain-aware Active Gaze Learning for Generalizable Agile Humanoid Locomotion [13.927597120628064]
  We introduce TAGA, a Terrain-aware Active Gaze learning framework for Attention-based humanoid control.<n>Our framework guides the model to learn anticipatory cues and actively attend to specific areas of the height scan.<n>We find that such gaze behaviors can naturally emerge through reinforcement learning alone, without requiring additional supervision or explicit guidance.
  arXiv  Detail & Related papers  (2026-06-04T08:52:56Z)
• Global-Local Attention Decomposition for Terrain Encoding in Humanoid Perceptive Locomotion [7.888727249239356]
  We propose Global-Local Attention Decomposition (GLAD) for terrain encoding in humanoid locomotion.<n> Experiments demonstrate that GLAD enables reliable locomotion over challenging gaps, stepping stones, and stairs.<n>In real-world deployment on a Unitree G1 humanoid robot using onboard LiDAR, the proposed method achieves robust zero-shot sim-to-real transfer.
  arXiv  Detail & Related papers  (2026-05-30T09:23:07Z)
• Plan in Sandbox, Navigate in Open Worlds: Learning Physics-Grounded Abstracted Experience for Embodied Navigation [63.11032720821731]
  textitSAGE is a framework that enables agents to learn within a physics-grounded semantic abstraction rather than a photorealistic simulation.<n>We demonstrate that textitSAGE significantly improves planner-assisted embodied navigation, achieving a 53.21% LLM-Match Success Rate on A-EQA.
  arXiv  Detail & Related papers  (2026-05-11T07:34:30Z)
• Vision-Guided MPPI for Agile Drone Racing: Navigating Arbitrary Gate Poses via Neural Signed Distance Fields [8.66703842496129]
  Drone racing requires tight coupling of perception, planning, and control under extreme agility.<n>Recent approaches rely on precomputed spatial reference trajectories or explicit 6-DoF gate pose estimation.<n>We propose a vision guided optimal control framework that enables reference-free agile flight through arbitrarily placed and oriented gates.
  arXiv  Detail & Related papers  (2026-03-07T13:18:07Z)
• ULTRA: Unified Multimodal Control for Autonomous Humanoid Whole-Body Loco-Manipulation [55.467742403416175]
  We introduce a physics-driven neural algorithm that translates large-scale motion capture to humanoid embodiments.<n>We learn a unified multimodal controller that supports both dense references and sparse task specifications.<n>Results show that ULTRA generalizes to autonomous, goal-conditioned whole-body loco-manipulation from egocentric perception.
  arXiv  Detail & Related papers  (2026-03-03T18:59:29Z)
• AutoFly: Vision-Language-Action Model for UAV Autonomous Navigation in the Wild [62.47761809929869]
  Vision-language navigation (VLN) requires intelligent agents to navigate environments by interpreting linguistic instructions alongside visual observations.<n>Current VLN research for unmanned aerial vehicles (UAVs) relies on detailed, pre-specified instructions to guide the UAV along predetermined routes.<n>We propose AutoFly, an end-to-end Vision-Language-Action model for autonomous UAV navigation.
  arXiv  Detail & Related papers  (2026-02-10T11:08:07Z)
• ZTRS: Zero-Imitation End-to-end Autonomous Driving with Trajectory Scoring [52.195295396336526]
  ZTRS (Zero-Imitation End-to-End Autonomous Driving with Trajectory Scoring) is a framework that combines the strengths of both worlds: sensor inputs without losing information and RL training for robust planning.<n>ZTRS demonstrates strong performance across three benchmarks: Navtest, Navhard, and HUGSIM.
  arXiv  Detail & Related papers  (2025-10-28T06:26:36Z)
• No More Blind Spots: Learning Vision-Based Omnidirectional Bipedal Locomotion for Challenging Terrain [12.51464645002418]
  We present a learning framework for vision-based omnidirectional bipedal locomotion.<n>A key challenge is the high computational cost of rendering omnidirectional depth images in simulation.<n>Our method combines a robust blind controller with a teacher policy that supervises a vision-based student policy.
  arXiv  Detail & Related papers  (2025-08-16T06:20:46Z)
• Gaussian Splatting to Real World Flight Navigation Transfer with Liquid Networks [93.38375271826202]
  We present a method to improve generalization and robustness to distribution shifts in sim-to-real visual quadrotor navigation tasks. We first build a simulator by integrating Gaussian splatting with quadrotor flight dynamics, and then, train robust navigation policies using Liquid neural networks. In this way, we obtain a full-stack imitation learning protocol that combines advances in 3D Gaussian splatting radiance field rendering, programming of expert demonstration training data, and the task understanding capabilities of Liquid networks.
  arXiv  Detail & Related papers  (2024-06-21T13:48:37Z)
• SAFE-SIM: Safety-Critical Closed-Loop Traffic Simulation with Diffusion-Controllable Adversaries [94.84458417662407]
  We introduce SAFE-SIM, a controllable closed-loop safety-critical simulation framework. Our approach yields two distinct advantages: 1) generating realistic long-tail safety-critical scenarios that closely reflect real-world conditions, and 2) providing controllable adversarial behavior for more comprehensive and interactive evaluations. We validate our framework empirically using the nuScenes and nuPlan datasets across multiple planners, demonstrating improvements in both realism and controllability.
  arXiv  Detail & Related papers  (2023-12-31T04:14:43Z)
• Learning High-Speed Flight in the Wild [101.33104268902208]
  We propose an end-to-end approach that can autonomously fly quadrotors through complex natural and man-made environments at high speeds. The key principle is to directly map noisy sensory observations to collision-free trajectories in a receding-horizon fashion. By simulating realistic sensor noise, our approach achieves zero-shot transfer from simulation to challenging real-world environments.
  arXiv  Detail & Related papers  (2021-10-11T09:43:11Z)
• Learning Vision-Guided Quadrupedal Locomotion End-to-End with Cross-Modal Transformers [14.509254362627576]
  We propose to address quadrupedal locomotion tasks using Reinforcement Learning (RL) We introduce LocoTransformer, an end-to-end RL method for quadrupedal locomotion.
  arXiv  Detail & Related papers  (2021-07-08T17:41:55Z)
• RLOC: Terrain-Aware Legged Locomotion using Reinforcement Learning and Optimal Control [6.669503016190925]
  We present a unified model-based and data-driven approach for quadrupedal planning and control. We map sensory information and desired base velocity commands into footstep plans using a reinforcement learning policy. We train and evaluate our framework on a complex quadrupedal system, ANYmal B, and demonstrate transferability to a larger and heavier robot, ANYmal C, without requiring retraining.
  arXiv  Detail & Related papers  (2020-12-05T18:30:23Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.