Active Stereo-Camera Outperforms Multi-Sensor Setup in ACT Imitation Learning for Humanoid Manipulation

• URL: http://arxiv.org/abs/2603.28422v1
• Date: Mon, 30 Mar 2026 13:30:34 GMT
• Title: Active Stereo-Camera Outperforms Multi-Sensor Setup in ACT Imitation Learning for Humanoid Manipulation
• Abstract summary: This paper benchmarks 14 sensor combinations on the Unitree G1 humanoid robot equipped with three-finger hands for two manipulation tasks.<n>We evaluate the integration of tactile and proprioceptive modalities alongside active vision.
• Score: 4.864819846886142
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The complexity of teaching humanoid robots new tasks is one of the major reasons hindering their widespread adoption in the industry. While Imitation Learning (IL), particularly Action Chunking with Transformers (ACT), enables rapid task acquisition, there is no consensus yet on the optimal sensory hardware required for manipulation tasks. This paper benchmarks 14 sensor combinations on the Unitree G1 humanoid robot equipped with three-finger hands for two manipulation tasks. We explicitly evaluate the integration of tactile and proprioceptive modalities alongside active vision. Our analysis demonstrates that strategic sensor selection can outperform complex configurations in data-limited regimes while reducing computational overhead. We develop an open-source Unified Ablation Framework that utilizes sensor masking on a comprehensive master dataset. Results indicate that additional modalities often degrade performance for IL with limited data. A minimal active stereo-camera setup outperformed complex multi-sensor configurations, achieving 87.5% success in a spatial generalization task and 94.4% in a structured manipulation task. Conversely, adding pressure sensors to this setup reduced success to 67.3% in the latter task due to a low signal-to-noise ratio. We conclude that in data-limited regimes, active vision offers a superior trade-off between robustness and complexity. While tactile modalities may require larger datasets to be effective, our findings validate that strategic sensor selection is critical for designing an efficient learning process.

Related papers

• Robust Slip Detection and Material Classification via Spatiotemporal Transformers on a Uniformly-Illuminated Visuo-Tactile Sensor [52.33204893988409]
  We propose a visuo-tactile sensor featuring customized uniform illumination, alongside a unified perception framework.<n>The proposed hardware-software framework provides high-fidelity feedback and a powerful multi-modal perception baseline for complex robotic manipulation.
  arXiv  Detail & Related papers  (2026-08-25T07:25:27Z)
• An Intelligent-Cloud Edge Multimodal Interaction System for Robots [4.1212219465063304]
  This paper presents a cloud-edge multimodal interaction framework that integrates an enhanced YOLO-based gesture detector with coordinated large language model (LLM) and vision-language model (VLM) agents.<n>The cloud layer performs gesture detection, scene understanding, multimodal fusion, and action planning, whereas the TonyPi robot locally handles data acquisition, communication, action execution, and feedback.
  arXiv  Detail & Related papers  (2026-07-16T07:39:18Z)
• Multi-Resolution Tactile Imitation Learning for Contact-Rich Robotic Manipulation [22.015203192700586]
  We present Multi-Resolution Tactile Sensing (MiTaS), a representation framework that leverages multiple tactile sensors operating at different temporal resolutions.<n>MiTaS achieves an average success rate of 80 %, while vision-only (31 and visual-tactile (54 baselines cannot solve the task reliably)<n>A detailed sensor-reading and attention analysis reveals the importance of different sensors throughout task execution.
  arXiv  Detail & Related papers  (2026-06-04T15:23:03Z)
• VTouch++: A Multimodal Dataset with Vision-Based Tactile Enhancement for Bimanual Manipulation [10.08235353271524]
  Embodied intelligence has advanced rapidly in recent years; however, bimanual manipulation-especially in contact-rich tasks remains challenging.<n>This is largely due to the lack of datasets with rich physical interaction signals, systematic task organization, and sufficient scale.<n>We introduce the VTOUCH dataset to provide high-fidelity physical interaction signals, adopt a matrix-style task design, and employ automated data collection pipelines covering real-world, demand-driven scenarios.
  arXiv  Detail & Related papers  (2026-04-22T11:08:08Z)
• UniVTAC: A Unified Simulation Platform for Visuo-Tactile Manipulation Data Generation, Learning, and Benchmarking [47.900334665380115]
  We propose UniVTAC, a simulation-based visuo-tactile data platform.<n>We introduce the UniVTAC encoder, a visuo-tactile encoder trained on large-scale simulation-synthesized data.<n>We present the UniVTAC Benchmark, which consists of eight representative visuo-tactile manipulation tasks.
  arXiv  Detail & Related papers  (2026-02-10T18:57:00Z)
• UniForce: A Unified Latent Force Model for Robot Manipulation with Diverse Tactile Sensors [51.88112610411651]
  We propose a novel unified tactile representation learning framework that learns a shared latent force space across diverse tactile sensors.<n>UniForce reduces cross-sensor domain shift by jointly modeling inverse dynamics (image-to-force) and forward dynamics (force-to-image)<n>To avoid reliance on expensive external force/torque (F/T) sensors, we exploit static equilibrium and collect force-paired data via direct sensor--object--sensor interactions.
  arXiv  Detail & Related papers  (2026-02-01T11:03:01Z)
• Simultaneous Tactile-Visual Perception for Learning Multimodal Robot Manipulation [21.78866976181311]
  See-through-skin (STS) sensors combine tactile and visual perception.<n>Existing STS designs lack simultaneous multimodal perception and suffer from unreliable tactile tracking.<n>We introduce TacThru, an STS sensor enabling simultaneous visual perception and robust tactile signal extraction.
  arXiv  Detail & Related papers  (2025-12-10T17:35:13Z)
• Self-Supervised Multisensory Pretraining for Contact-Rich Robot Reinforcement Learning [10.782934021703783]
  MultiSensory Dynamic Pretraining (MSDP) is a framework for learning expressive multisensory representations tailored for task-oriented policy learning.<n>MSDP is based on masked autoencoding and trains a transformer-based encoder by reconstructing multisensory observations from only a subset of sensor embeddings.<n>For downstream policy learning, we introduce a novel asymmetric architecture, where a cross-attention mechanism allows the critic to extract dynamic, task-specific features from the frozen embeddings.
  arXiv  Detail & Related papers  (2025-11-18T12:32:23Z)
• StaMo: Unsupervised Learning of Generalizable Robot Motion from Compact State Representation [56.996371714721995]
  We propose an unsupervised approach that learns a highly compressed two-token state representation.<n>Our representation is efficient, interpretable, and integrates seamlessly into existing VLA-based models.<n>We name our method StaMo for its ability to learn generalizable robotic Motion from compact State representation.
  arXiv  Detail & Related papers  (2025-10-06T17:37:24Z)
• Rethinking Bimanual Robotic Manipulation: Learning with Decoupled Interaction Framework [51.39847596489193]
  Bimanual robotic manipulation is an emerging and critical topic in the robotics community.<n>We propose a novel decoupled interaction framework that considers the characteristics of different tasks in bimanual manipulation.<n>Our framework achieves outstanding performance, with a 23.5% boost over the SOTA method.
  arXiv  Detail & Related papers  (2025-03-12T09:28:41Z)
• AI-in-the-Loop Sensing and Communication Joint Design for Edge Intelligence [65.29835430845893]
  We propose a framework that enhances edge intelligence through AI-in-the-loop joint sensing and communication.<n>A key contribution of our work is establishing an explicit relationship between validation loss and the system's tunable parameters.<n>We show that our framework reduces communication energy consumption by up to 77 percent and sensing costs measured by the number of samples by up to 52 percent.
  arXiv  Detail & Related papers  (2025-02-14T14:56:58Z)
• SensorBench: Benchmarking LLMs in Coding-Based Sensor Processing [6.8009140511761546]
  Large Language Models (LLMs) have promising capabilities in processing sensory data, suggesting their potential as copilots for developing sensing systems.<n>We construct a comprehensive benchmark, SensorBench, to establish a quantifiable objective.<n>The results show that while LLMs exhibit considerable proficiency in simpler tasks, they face inherent challenges in processing compositional tasks.
  arXiv  Detail & Related papers  (2024-10-14T17:21:39Z)
• Dynamic Hand Gesture-Featured Human Motor Adaptation in Tool Delivery using Voice Recognition [5.13619372598999]
  This paper introduces an innovative human-robot collaborative framework. It seamlessly integrates hand gesture and dynamic movement recognition, voice recognition, and a switchable control adaptation strategy. Experiment results have demonstrated superior performance in hand gesture recognition.
  arXiv  Detail & Related papers  (2023-09-20T14:51:09Z)
• Bayesian Imitation Learning for End-to-End Mobile Manipulation [80.47771322489422]
  Augmenting policies with additional sensor inputs, such as RGB + depth cameras, is a straightforward approach to improving robot perception capabilities. We show that using the Variational Information Bottleneck to regularize convolutional neural networks improves generalization to held-out domains. We demonstrate that our method is able to help close the sim-to-real gap and successfully fuse RGB and depth modalities.
  arXiv  Detail & Related papers  (2022-02-15T17:38:30Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.