FingerEye: Continuous and Unified Vision-Tactile Sensing for Dexterous Manipulation

• URL: http://arxiv.org/abs/2604.20689v2
• Date: Thu, 23 Apr 2026 14:32:27 GMT
• Title: FingerEye: Continuous and Unified Vision-Tactile Sensing for Dexterous Manipulation
• Abstract summary: Dexterous robotic manipulation requires comprehensive perception across all phases of interaction.<n>We introduce FingerEye, a compact and cost-effective sensor that provides continuous vision-tactile feedback throughout the interaction process.<n>We develop a vision-tactile imitation learning policy that fuses signals from multiple FingerEye sensors to learn dexterous manipulation behaviors from limited real-world data.
• Score: 8.809778152652724
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Dexterous robotic manipulation requires comprehensive perception across all phases of interaction: pre-contact, contact initiation, and post-contact. Such continuous feedback allows a robot to adapt its actions throughout interaction. However, many existing tactile sensors, such as GelSight and its variants, only provide feedback after contact is established, limiting a robot's ability to precisely initiate contact. We introduce FingerEye, a compact and cost-effective sensor that provides continuous vision-tactile feedback throughout the interaction process. FingerEye integrates binocular RGB cameras to provide close-range visual perception with implicit stereo depth. Upon contact, external forces and torques deform a compliant ring structure; these deformations are captured via marker-based pose estimation and serve as a proxy for contact wrench sensing. This design enables a perception stream that smoothly transitions from pre-contact visual cues to post-contact tactile feedback. Building on this sensing capability, we develop a vision-tactile imitation learning policy that fuses signals from multiple FingerEye sensors to learn dexterous manipulation behaviors from limited real-world data. We further develop a digital twin of our sensor and robot platform to improve policy generalization. By combining real demonstrations with visually augmented simulated observations for representation learning, the learned policies become more robust to object appearance variations. Together, these design aspects enable dexterous manipulation across diverse object properties and interaction regimes, including coin standing, chip picking, letter retrieving, and syringe manipulation. The hardware design, code, appendix, and videos are available on our project website: https://nus-lins-lab.github.io/FingerEyeWeb/

Related papers

• NoContactNoWorries: Estimating Contact through Vision and Proprioception for In-Hand Dexterous Manipulation [0.7378853859331618]
  NoContactNoWorries is a framework that fuses RGB-D vision with the robot's proprioception to infer binary contact states.<n>We show that the inferred contact signal supports downstream reinforcement learning agents for in-hand object reorientation.
  arXiv  Detail & Related papers  (2026-06-23T11:36:46Z)
• UniForce: A Unified Latent Force Model for Robot Manipulation with Diverse Tactile Sensors [51.88112610411651]
  We propose a novel unified tactile representation learning framework that learns a shared latent force space across diverse tactile sensors.<n>UniForce reduces cross-sensor domain shift by jointly modeling inverse dynamics (image-to-force) and forward dynamics (force-to-image)<n>To avoid reliance on expensive external force/torque (F/T) sensors, we exploit static equilibrium and collect force-paired data via direct sensor--object--sensor interactions.
  arXiv  Detail & Related papers  (2026-02-01T11:03:01Z)
• OPENTOUCH: Bringing Full-Hand Touch to Real-World Interaction [93.88239833545623]
  We present OpenTouch, the first in-the-wild egocentric full-hand tactile dataset.<n>We show that tactile signals provide a compact yet powerful cue for grasp understanding.<n>We aim to advance multimodal egocentric perception, embodied learning, and contact-rich robotic manipulation.
  arXiv  Detail & Related papers  (2025-12-18T18:18:17Z)
• Touch in the Wild: Learning Fine-Grained Manipulation with a Portable Visuo-Tactile Gripper [7.618517580705364]
  We present a portable, lightweight gripper with integrated tactile sensors.<n>We propose a cross-modal representation learning framework that integrates visual and tactile signals.<n>We validate our approach on fine-grained tasks such as test tube insertion and pipette-based fluid transfer.
  arXiv  Detail & Related papers  (2025-07-20T17:53:59Z)
• Digitizing Touch with an Artificial Multimodal Fingertip [51.7029315337739]
  Humans and robots both benefit from using touch to perceive and interact with the surrounding environment. Here, we describe several conceptual and technological innovations to improve the digitization of touch. These advances are embodied in an artificial finger-shaped sensor with advanced sensing capabilities.
  arXiv  Detail & Related papers  (2024-11-04T18:38:50Z)
• Robot Synesthesia: In-Hand Manipulation with Visuotactile Sensing [15.970078821894758]
  We introduce a system that leverages visual and tactile sensory inputs to enable dexterous in-hand manipulation. Robot Synesthesia is a novel point cloud-based tactile representation inspired by human tactile-visual synesthesia.
  arXiv  Detail & Related papers  (2023-12-04T12:35:43Z)
• Multimodal and Force-Matched Imitation Learning with a See-Through Visuotactile Sensor [14.492202828369127]
  We leverage a multimodal visuotactile sensor within the framework of imitation learning (IL) to perform contact-rich tasks.<n>We introduce two algorithmic contributions, tactile force matching and learned mode switching, as complimentary methods for improving IL.<n>Our results show that the inclusion of force matching raises average policy success rates by 62.5%, visuotactile mode switching by 30.3%, and visuotactile data as a policy input by 42.5%.
  arXiv  Detail & Related papers  (2023-11-02T14:02:42Z)
• Tactile-Filter: Interactive Tactile Perception for Part Mating [54.46221808805662]
  Humans rely on touch and tactile sensing for a lot of dexterous manipulation tasks. vision-based tactile sensors are being widely used for various robotic perception and control tasks. We present a method for interactive perception using vision-based tactile sensors for a part mating task.
  arXiv  Detail & Related papers  (2023-03-10T16:27:37Z)
• Elastic Tactile Simulation Towards Tactile-Visual Perception [58.44106915440858]
  We propose Elastic Interaction of Particles (EIP) for tactile simulation. EIP models the tactile sensor as a group of coordinated particles, and the elastic property is applied to regulate the deformation of particles during contact. We further propose a tactile-visual perception network that enables information fusion between tactile data and visual images.
  arXiv  Detail & Related papers  (2021-08-11T03:49:59Z)
• OmniTact: A Multi-Directional High Resolution Touch Sensor [109.28703530853542]
  Existing tactile sensors are either flat, have small sensitive fields or only provide low-resolution signals. We introduce OmniTact, a multi-directional high-resolution tactile sensor. We evaluate the capabilities of OmniTact on a challenging robotic control task.
  arXiv  Detail & Related papers  (2020-03-16T01:31:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.