RGB-S: Image-Aligned Tactile Saliency for Robust Dexterous Manipulation

• URL: http://arxiv.org/abs/2606.08765v2
• Date: Thu, 11 Jun 2026 07:13:24 GMT
• Title: RGB-S: Image-Aligned Tactile Saliency for Robust Dexterous Manipulation
• Abstract summary: We propose a framework that explicitly grounds physical contacts in the image domain.<n>Our method injects physical contact priors into standard visual backbones while preserving pre-trained visual representations.<n>Real-world experiments show that explicit RGB-S grounding in the image domain improves real-world occluded manipulation success rates by $26.7$ percentage points.
• Score: 4.949265216654121
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Effective visuo-tactile integration is critical for robotic dexterous manipulation, especially when visual observations are unreliable or occluded. However, robustly aligning sparse, heterogeneous tactile measurements with dense visual representations remains a fundamental challenge. Most existing approaches require policies to learn cross-modal correspondences implicitly from limited demonstrations, without leveraging geometric priors. As a result, they are often data-inefficient and generalize poorly when visual observations are degraded. To address this limitation, we propose a framework that explicitly grounds physical contacts in the image domain. Using robot forward kinematics and camera calibration, we project tactile sensor locations directly onto the RGB image plane. We then render force-modulated Gaussian saliency maps to model spatial uncertainty arising from kinematic and calibration errors. By integrating these 2D spatial anchors through a zero-initialized conditioning architecture, our method injects physical contact priors into standard visual backbones while preserving pre-trained visual representations. We evaluate our method on six dexterous manipulation tasks in both simulation and the real world under severe visual occlusions. Real-world experiments show that explicit RGB-S grounding in the image domain improves real-world occluded manipulation success rates by $26.7$ percentage points over the strongest implicit visuo-tactile baseline, suggesting its improved spatial reasoning and robustness to occlusion. Project page: touch-as-saliency.github.io

Related papers

• Topology-Agnostic Mesh Reconstruction of Deformable Objects from Sparse Touch [0.0]
  We present a single topology-agnostic estimator that reconstructs the full mesh of a deformable object from only a few touches and no vision.<n>We show that the estimator's deep-ensemble uncertainty can be used to learn where to touch next.
  arXiv  Detail & Related papers  (2026-07-15T06:17:35Z)
• Image2Sim: Scaling Embodied Navigation via Generative Neural Simulator [70.34667754804168]
  Embodied navigation aims to build agents that interpret multimodal goals, reason in 3D space, and reach target destinations reliably in the real world.<n>We introduce Image2Sim, a real-time neural simulation framework that constructs high-quality interactive environments from posed RGB-D image sequences.
  arXiv  Detail & Related papers  (2026-07-07T02:42:41Z)
• GARDEN: Gravity-Aligned Reconstruction of Disentangled ENvironments from RGB images [61.2170105836525]
  We propose an RGB-only framework that reformulates reconstruction as physically-grounded scene factorization and outputs a structured hybrid scene representation.<n>The resulting representation combines explicit rigid bodies with a decoupled background, enabling direct physics simulation while preserving visual realism.
  arXiv  Detail & Related papers  (2026-06-02T17:13:01Z)
• Trans2Occ: Voxel Occupancy Estimation and Grasp for Transparent Objects from Simulation to Reality [125.76063573012853]
  We present a framework for transparent object perception and manipulation based on single-view RGB input.<n>Our approach predicts voxel-space occupancy directly from a single image, providing a geometry-aware representation that supports downstream robotic grasping.
  arXiv  Detail & Related papers  (2026-06-01T06:59:13Z)
• A Model-based Visual Contact Localization and Force Sensing System for Compliant Robotic Grippers [11.232660351404684]
  Integrating force sensing into deformable grippers negotiates trade-offs in cost, complexity, mechanical robustness, and performance.<n>Current approaches mostly utilize end-to-end deep learning, which can be brittle when generalizing to new scenarios.<n>We developed a model-based visual force sensing approach integrating an iterative contact localization with generalization to unseen objects.
  arXiv  Detail & Related papers  (2026-05-01T00:34:19Z)
• ViserDex: Visual Sim-to-Real for Robust Dexterous In-hand Reorientation [9.303398221598739]
  In-hand object reorientation requires precise estimation of the object pose to handle complex task dynamics.<n>While RGB sensing offers rich semantic cues for pose tracking, existing solutions rely on multi-camera setups or costly ray tracing.<n>We present a framework for monocular RGB in-hand reorientation that integrates 3D Gaussian Splatting (3DGS) to bridge the visual sim-to-real gap.
  arXiv  Detail & Related papers  (2026-04-13T07:50:41Z)
• TouchAnything: Diffusion-Guided 3D Reconstruction from Sparse Robot Touches [18.3533066960967]
  We present TouchAnything, a framework that leverages a pretrained vision diffusion model as a semantic and geometric prior for 3D reconstruction from sparse tactile measurements.<n>Our method reconstructs accurate geometries from only a few touches, outperforms existing baselines, and enables open-world 3D reconstruction of previously unseen object instances.
  arXiv  Detail & Related papers  (2026-04-10T04:26:36Z)
• HGGT: Robust and Flexible 3D Hand Mesh Reconstruction from Uncalibrated Images [81.42866295265443]
  High-fidelity 3D hand geometry is a critical task in computer vision.<n> scalable applications demand both accuracy and deployment flexibility.<n>We propose a feed-forward architecture that infers 3D hand meshes and camera poses from uncalibrated views.
  arXiv  Detail & Related papers  (2026-03-25T06:54:34Z)
• Semantic-Contact Fields for Category-Level Generalizable Tactile Tool Manipulation [82.63833405368159]
  Generalizing tool manipulation requires both semantic planning and precise physical control.<n>We propose Semantic-Contact Fields (SCFields), a unified 3D representation fusing visual semantics with dense contact estimates.<n> Experiments on scraping, crayon drawing, and peeling demonstrate robust category-level generalization.
  arXiv  Detail & Related papers  (2026-02-14T16:05:08Z)
• RFTrans: Leveraging Refractive Flow of Transparent Objects for Surface Normal Estimation and Manipulation [50.10282876199739]
  This paper introduces RFTrans, an RGB-D-based method for surface normal estimation and manipulation of transparent objects. It integrates the RFNet, which predicts refractive flow, object mask, and boundaries, followed by the F2Net, which estimates surface normal from the refractive flow. A real-world robot grasping task witnesses an 83% success rate, proving that refractive flow can help enable direct sim-to-real transfer.
  arXiv  Detail & Related papers  (2023-11-21T07:19:47Z)
• A Distance-Geometric Method for Recovering Robot Joint Angles From an RGB Image [7.971699294672282]
  We present a novel method for retrieving the joint angles of a robot manipulator using only a single RGB image of its current configuration. Our approach, based on a distance-geometric representation of the configuration space, exploits the knowledge of a robot's kinematic model.
  arXiv  Detail & Related papers  (2023-01-05T12:57:45Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.