VibeAct: Vibration to Actions for Contact-Rich Reactive Robot Dexterity

• URL: http://arxiv.org/abs/2606.27344v1
• Date: Thu, 25 Jun 2026 17:50:07 GMT
• Title: VibeAct: Vibration to Actions for Contact-Rich Reactive Robot Dexterity
• Abstract summary: VibeAct is a framework that bridges real vibrotactile sensing and simulation-based reinforcement learning.<n>In the real world, we embed piezoelectric microphones into a dexterous robot hand and collect vibro-acoustic data through teleoperation.<n>A tactile estimator learns to predict contact and slip from real microphone waveforms, while manipulation policies are trained in simulation on the same representation computed directly from simulated contacts.
• Score: 19.20612096722837
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Dexterous manipulation depends on contact events that are fast, local, and often visually occluded. Piezoelectric microphones offer a compact and high-bandwidth way to sense these interactions, but the resulting vibro-acoustic signals are difficult to simulate faithfully enough for end-to-end sim-to-real policy learning on dexterous robot hands. We propose VibeAct, a framework that bridges real vibrotactile sensing and simulation-based reinforcement learning through a shared physical representation of contact and slip. In the real world, we embed piezoelectric microphones into a dexterous robot hand and collect vibro-acoustic data through teleoperation, then replay the recordings in a calibrated digital clone to automatically label per-finger contact and slip. A tactile estimator learns to predict contact and slip from real microphone waveforms, while manipulation policies are trained in simulation on the same representation computed directly from simulated contacts. This decoupling lets policies exploit rapid tactile feedback without simulating raw audio. Across five contact-rich tasks spanning regrasping, in-hand reorientation, and insertion, VibeAct consistently outperforms a proprioception-and-point-cloud baseline in simulation, with the largest gains on tasks requiring sustained reactive control, where the continuous slip-magnitude channel proves the most informative observation. The learned policies transfer to a physical dexterous hand-arm platform, improving success rates on deployed tasks. Project videos and additional details are at https://vibeact.github.io/.

Related papers

• Pre-training Visual Dexterity in Simulation [67.17477115017896]
  We introduce a pre-training framework for dexterous manipulation that uses data entirely collected in simulation.<n>With the help of five operators, we collect 75 hours of multi-task dexterous manipulation over one week, and use it to pre-train a causal transformer on a sequence modeling objective.<n>We find that our approach outperforms training behavior cloning policies from scratch, showing that simulation teleoperation is a viable pre-training source for real-world dexterous manipulation.
  arXiv  Detail & Related papers  (2026-08-16T20:21:33Z)
• ViTacWorld: Scaling Visuo-Tactile World Models for Contact-Rich Robot Manipulation [30.004466515831634]
  ViTacWorld is an action-conditioned visuo-tactile world model for scalable contact-rich robot manipulation.<n>It exploits the fact that tactile signals are directly grounded in physical contact and can exhibit a smaller simulation-to-real gap than purely visual observations.<n>It is the first framework that uses a world model for robot visuo-tactile-action trajectory generation and policy evaluation.
  arXiv  Detail & Related papers  (2026-07-24T17:59:33Z)
• Blind Dexterous Grasping via Real2Sim2Real Tactile Policy Learning [6.509572742369941]
  We propose a framework for tactile-only blind grasping that is deployable on a physical multi-fingered robotic hand.<n>We introduce a Real2Sim tactile calibration pipeline that constructs a contact-calibrated digital-twin simulator capable of reproducing real tactile signals.<n>Second, we improve the expressiveness of sparse tactile observations using a layout-aware tactile encoder, which incorporates sensor-geometry priors through self-supervised pretraining.<n>Third, to improve generalization to unseen objects, we train object-specific reinforcement-learning experts in the calibrated simulator and aggregate their successful grasp trajectories into a tactile-conditioned Diffusion Policy.
  arXiv  Detail & Related papers  (2026-06-10T07:46:38Z)
• TacCoRL: Integrating Tactile Feedback into VLA via Simulation [38.963250081631834]
  Vision-language-action (VLA) models provide strong visual, language, and action priors for robot manipulation.<n>We present TacCoRL, a scalable framework that injects Tactile feedback into VLA policies.<n>We use a real-aligned simulator as a closed-loop training environment for contact interaction.
  arXiv  Detail & Related papers  (2026-06-10T07:20:36Z)
• OmniVTA: Visuo-Tactile World Modeling for Contact-Rich Robotic Manipulation [57.133721026727706]
  We present textbfOmniViTac, a large-scale visuo-tactile-action dataset comprising $21,000+$ trajectories across $86$ tasks and $100+$ objects.<n>We propose textbf OmniVTA, a world-model-based visuo-tactile manipulation framework that integrates four tightly coupled modules.
  arXiv  Detail & Related papers  (2026-03-19T17:52:42Z)
• PTLD: Sim-to-real Privileged Tactile Latent Distillation for Dexterous Manipulation [43.81692375130905]
  We introduce PTLD: sim-to-real Privileged Tactile Latent Distillation, a novel approach to learning tactile manipulation skills without requiring tactile simulation.<n>We demonstrate that PTLD can be used to improve proprioceptive manipulation policies trained in simulation significantly by incorporating tactile sensing.
  arXiv  Detail & Related papers  (2026-03-04T19:17:42Z)
• Semantic-Contact Fields for Category-Level Generalizable Tactile Tool Manipulation [82.63833405368159]
  Generalizing tool manipulation requires both semantic planning and precise physical control.<n>We propose Semantic-Contact Fields (SCFields), a unified 3D representation fusing visual semantics with dense contact estimates.<n> Experiments on scraping, crayon drawing, and peeling demonstrate robust category-level generalization.
  arXiv  Detail & Related papers  (2026-02-14T16:05:08Z)
• TouchGuide: Inference-Time Steering of Visuomotor Policies via Touch Guidance [53.35296919674763]
  TouchGuide is a novel cross-policy visuo-tactile fusion paradigm that fuses modalities within a low-dimensional action space.<n>touchGuide operates in two stages to guide a pre-trained diffusion or flow-matching visuomotor policy.<n>To facilitate TouchGuide training with high-quality and cost-effective data, we introduce TacUMI, a data collection system.
  arXiv  Detail & Related papers  (2026-01-28T04:22:47Z)
• Tactile Memory with Soft Robot: Robust Object Insertion via Masked Encoding and Soft Wrist [10.982180941605256]
  We introduce Tactile Memory with Soft Robot (TaSo-bot), a system that integrates a soft wrist with retrieval-based control to enable safe and robust manipulation.<n>The core of this system is the Masked Tactile Trajectory Transformer (MATtext3$), which jointly models interactions between robot actions, tactile feedback, force-torque measurements, and proprioceptive signals.<n>MATtext3$ achieves higher success rates than the baselines over all conditions and shows remarkable capability to adapt to unseen pegs and conditions.
  arXiv  Detail & Related papers  (2026-01-27T07:04:01Z)
• Taccel: Scaling Up Vision-based Tactile Robotics via High-performance GPU Simulation [34.47272224723296]
  We present Taccel, a high-performance simulation platform that integrates IPC and ABD to model robots, tactile sensors, and objects with both accuracy and unprecedented speed.<n>Unlike previous simulators that operate at sub-real-time speeds with limited parallelization, Taccel provides precise physics simulation and realistic tactile signals.<n>These capabilities position Taccel as a powerful tool for scaling up tactile robotics research and development, potentially transforming how robots interact with and understand their physical environment.
  arXiv  Detail & Related papers  (2025-04-17T12:57:11Z)
• Sim2Real Transfer for Audio-Visual Navigation with Frequency-Adaptive Acoustic Field Prediction [51.71299452862839]
  We propose the first treatment of sim2real for audio-visual navigation by disentangling it into acoustic field prediction (AFP) and waypoint navigation. We then collect real-world data to measure the spectral difference between the simulation and the real world by training AFP models that only take a specific frequency subband as input. Lastly, we build a real robot platform and show that the transferred policy can successfully navigate to sounding objects.
  arXiv  Detail & Related papers  (2024-05-05T06:01:31Z)
• DeXtreme: Transfer of Agile In-hand Manipulation from Simulation to Reality [64.51295032956118]
  We train a policy that can perform robust dexterous manipulation on an anthropomorphic robot hand. Our work reaffirms the possibilities of sim-to-real transfer for dexterous manipulation in diverse kinds of hardware and simulator setups.
  arXiv  Detail & Related papers  (2022-10-25T01:51:36Z)
• Play it by Ear: Learning Skills amidst Occlusion through Audio-Visual Imitation Learning [62.83590925557013]
  We learn a set of challenging partially-observed manipulation tasks from visual and audio inputs. Our proposed system learns these tasks by combining offline imitation learning from tele-operated demonstrations and online finetuning. In a set of simulated tasks, we find that our system benefits from using audio, and that by using online interventions we are able to improve the success rate of offline imitation learning by 20%.
  arXiv  Detail & Related papers  (2022-05-30T04:52:58Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.