Enhancing Vision-Based Policies with Omni-View and Cross-Modality Knowledge Distillation for Mobile Robots

• URL: http://arxiv.org/abs/2603.20679v1
• Date: Sat, 21 Mar 2026 06:38:30 GMT
• Title: Enhancing Vision-Based Policies with Omni-View and Cross-Modality Knowledge Distillation for Mobile Robots
• Abstract summary: Vision-based policies are widely applied in robotics for tasks such as manipulation and locomotion.<n>On lightweight mobile robots, however, they face a trilemma of limited scene transferability, restricted onboard computation resources, and sensor hardware cost.<n>We propose a knowledge distillation approach that transfers knowledge from an information-rich, appearance invariant omniview depth policy to a lightweight monocular policy.
• Score: 18.940326743352315
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Vision-based policies are widely applied in robotics for tasks such as manipulation and locomotion. On lightweight mobile robots, however, they face a trilemma of limited scene transferability, restricted onboard computation resources, and sensor hardware cost. To address these issues, we propose a knowledge distillation approach that transfers knowledge from an information-rich, appearance invariant omniview depth policy to a lightweight monocular policy. The key idea is to train the student not only to mimic the expert actions but also to align with the latent embeddings of the omni view depth teacher. Experiments demonstrate that omni-view and depth inputs improve the scene transfer and navigation performance, and that the proposed distillation method enhances the performance of a singleview monocular policy, compared with policies solely imitating actions. Real world experiments further validate the effectiveness and practicality of our approach. Code will be released publicly.

Related papers

• When would Vision-Proprioception Policies Fail in Robotic Manipulation? [24.653339846980376]
  Proprioceptive information is critical for precise servo control by providing real-time robotic states.<n>Recent studies have reported inconsistent observations on the generalization of vision-proprioception policies.<n>We propose the Gradient Adjustment with Phase-guidance (GAP) algorithm that adaptively modulates the optimization of proprioception.
  arXiv  Detail & Related papers  (2026-02-12T15:00:48Z)
• Dexplore: Scalable Neural Control for Dexterous Manipulation from Reference-Scoped Exploration [58.4036440289082]
  Hand-object motion-capture (MoCap) offer large-scale, contact-rich demonstrations and hold promise for dexterous robotic scopes.<n>We introduce Dexplore, a unified single-loop optimization that performs repositories and tracking to learn robot control policies directly from MoCap at scale.
  arXiv  Detail & Related papers  (2025-09-11T17:59:07Z)
• No More Blind Spots: Learning Vision-Based Omnidirectional Bipedal Locomotion for Challenging Terrain [12.51464645002418]
  We present a learning framework for vision-based omnidirectional bipedal locomotion.<n>A key challenge is the high computational cost of rendering omnidirectional depth images in simulation.<n>Our method combines a robust blind controller with a teacher policy that supervises a vision-based student policy.
  arXiv  Detail & Related papers  (2025-08-16T06:20:46Z)
• Sim-to-Real Reinforcement Learning for Vision-Based Dexterous Manipulation on Humanoids [56.892520712892804]
  We introduce a practical sim-to-real RL recipe that trains a humanoid robot to perform three dexterous manipulation tasks.<n>We demonstrate high success rates on unseen objects and robust, adaptive policy behaviors.
  arXiv  Detail & Related papers  (2025-02-27T18:59:52Z)
• ViViDex: Learning Vision-based Dexterous Manipulation from Human Videos [81.99559944822752]
  We propose ViViDex to improve vision-based policy learning from human videos.<n>It first uses reinforcement learning with trajectory guided rewards to train state-based policies for each video.<n>We then rollout successful episodes from state-based policies and train a unified visual policy without using any privileged information.
  arXiv  Detail & Related papers  (2024-04-24T07:58:28Z)
• MOKA: Open-World Robotic Manipulation through Mark-Based Visual Prompting [97.52388851329667]
  We introduce Marking Open-world Keypoint Affordances (MOKA) to solve robotic manipulation tasks specified by free-form language instructions. Central to our approach is a compact point-based representation of affordance, which bridges the VLM's predictions on observed images and the robot's actions in the physical world. We evaluate and analyze MOKA's performance on various table-top manipulation tasks including tool use, deformable body manipulation, and object rearrangement.
  arXiv  Detail & Related papers  (2024-03-05T18:08:45Z)
• The Power of the Senses: Generalizable Manipulation from Vision and Touch through Masked Multimodal Learning [60.91637862768949]
  We propose Masked Multimodal Learning (M3L) to fuse visual and tactile information in a reinforcement learning setting. M3L learns a policy and visual-tactile representations based on masked autoencoding. We evaluate M3L on three simulated environments with both visual and tactile observations.
  arXiv  Detail & Related papers  (2023-11-02T01:33:00Z)
• Contrastive Learning for Enhancing Robust Scene Transfer in Vision-based Agile Flight [21.728935597793473]
  This work proposes an adaptive multi-pair contrastive learning strategy for visual representation learning that enables zero-shot scene transfer and real-world deployment. We demonstrate the performance of our approach on the task of agile, vision-based quadrotor flight.
  arXiv  Detail & Related papers  (2023-09-18T15:25:59Z)
• Visual-Policy Learning through Multi-Camera View to Single-Camera View Knowledge Distillation for Robot Manipulation Tasks [4.820787231200527]
  We present a novel approach to enhance the generalization performance of vision-based Reinforcement Learning (RL) algorithms for robotic manipulation tasks. Our proposed method involves utilizing a technique known as knowledge distillation, in which a pre-trained teacher'' policy trained with multiple camera viewpoints guides a student'' policy in learning from a single camera viewpoint. The results demonstrate that the single-view visual student policy can successfully learn to grasp and lift a challenging object, which was not possible with a single-view policy alone.
  arXiv  Detail & Related papers  (2023-03-13T11:42:38Z)
• Distilling Motion Planner Augmented Policies into Visual Control Policies for Robot Manipulation [26.47544415550067]
  We propose to distill a state-based motion planner augmented policy to a visual control policy. We evaluate our method on three manipulation tasks in obstructed environments. Our framework is highly sample-efficient and outperforms the state-of-the-art algorithms.
  arXiv  Detail & Related papers  (2021-11-11T18:52:00Z)
• ViNG: Learning Open-World Navigation with Visual Goals [82.84193221280216]
  We propose a learning-based navigation system for reaching visually indicated goals. We show that our system, which we call ViNG, outperforms previously-proposed methods for goal-conditioned reinforcement learning. We demonstrate ViNG on a number of real-world applications, such as last-mile delivery and warehouse inspection.
  arXiv  Detail & Related papers  (2020-12-17T18:22:32Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.