Active Cross-Modal Visuo-Tactile Perception of Deformable Linear Objects

• URL: http://arxiv.org/abs/2601.13979v1
• Date: Tue, 20 Jan 2026 13:57:59 GMT
• Title: Active Cross-Modal Visuo-Tactile Perception of Deformable Linear Objects
• Abstract summary: This paper presents a novel cross-modal visuo-tactile perception framework for the 3D shape reconstruction of deformable linear objects (DLOs)<n>Unlike existing methods relying predominantly on vision, the proposed approach integrates foundation-model-based visual perception with adaptive exploration.<n>Results show that the framework accurately reconstructs both simple and highly curved single or multiple cable configurations, even when large portions are occluded.
• Score: 1.9735804580163874
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: This paper presents a novel cross-modal visuo-tactile perception framework for the 3D shape reconstruction of deformable linear objects (DLOs), with a specific focus on cables subject to severe visual occlusions. Unlike existing methods relying predominantly on vision, whose performance degrades under varying illumination, background clutter, or partial visibility, the proposed approach integrates foundation-model-based visual perception with adaptive tactile exploration. The visual pipeline exploits SAM for instance segmentation and Florence for semantic refinement, followed by skeletonization, endpoint detection, and point-cloud extraction. Occluded cable segments are autonomously identified and explored with a tactile sensor, which provides local point clouds that are merged with the visual data through Euclidean clustering and topology-preserving fusion. A B-spline interpolation driven by endpoint-guided point sorting yields a smooth and complete reconstruction of the cable shape. Experimental validation using a robotic manipulator equipped with an RGB-D camera and a tactile pad demonstrates that the proposed framework accurately reconstructs both simple and highly curved single or multiple cable configurations, even when large portions are occluded. These results highlight the potential of foundation-model-enhanced cross-modal perception for advancing robotic manipulation of deformable objects.

Related papers

• Monocular Vision Based Control Framework for Grasping [8.44254908378373]
  We present a vision-based grasping framework that targets both soft and rigid objects within a single control pipeline.<n>The proposed system combines open-vocabulary object detection, image segmentation, boundary-aware point assignment, real-time point tracking, and monocular depth estimation.<n>Results demonstrate that the framework achieves stable grasping across both soft and rigid objects using visual feedback alone.
  arXiv  Detail & Related papers  (2026-07-08T20:13:17Z)
• ExACT: Exemplar-Driven Calibrated Refinement for Training-Free Visual Grounding in Remote Sensing Images [51.53960096934913]
  Remote sensing visual grounding (RSVG) aims to locate specific objects in high-resolution RS imagery using free-form natural language descriptions.<n>Recent advances in multimodal large language models (MLLMs) show great potential for such open-vocabulary RSVG.<n>We propose Exemplar-driven Calibrated Refinement (ExACT), a training-free framework driven by a one-shot visual prompting mechanism.
  arXiv  Detail & Related papers  (2026-06-27T13:50:39Z)
• Physically Grounded 3D Generative Reconstruction under Hand Occlusion using Proprioception and Multi-Contact Touch [3.926587614210279]
  We propose a multimodal, physically grounded approach for metric-scale amodal object reconstruction and pose estimation under hand occlusion.<n>We leverage physical interaction signals: proprioception provides the posed hand geometry, and multi-contact touch constrains where the object surface must lie.
  arXiv  Detail & Related papers  (2026-04-10T08:32:51Z)
• Transparent Fragments Contour Estimation via Visual-Tactile Fusion for Autonomous Reassembly [69.97596838714595]
  We propose a general transparent fragments contour estimation framework based on visual-tactile fusion.<n>By fusing tactile information with visual cues, a visual-tactile fusion material is proposed.<n> experimental results demonstrate the validity of the proposed framework.
  arXiv  Detail & Related papers  (2026-03-18T14:58:17Z)
• LSP-ST: Ladder Shape-Biased Side-Tuning for Robust Infrared Small Target Detection [4.5138645285711165]
  We propose Ladder Shape-Biased Side-Tuning (LSP-ST), a novel approach that introduces a shape-aware inductive bias to facilitate effective adaptation beyond texture cues.<n>With only 4.72M learnable parameters, LSP-ST achieves state-of-the-art performance on multiple infrared small target detection benchmarks.
  arXiv  Detail & Related papers  (2025-04-20T04:12:38Z)
• SPAC-Net: Rethinking Point Cloud Completion with Structural Prior [17.334882020068697]
  Point cloud completion aims to infer a complete shape from its partial observation. Many approaches utilize a pure encoderdecoder paradigm in which complete shape can be directly predicted by shape priors learned from partial scans. We propose a novel framework,termed SPAC-Net, that aims to rethink the completion task under the guidance of a new structural prior.
  arXiv  Detail & Related papers  (2024-11-22T16:54:17Z)
• VortSDF: 3D Modeling with Centroidal Voronoi Tesselation on Signed Distance Field [5.573454319150408]
  We introduce a volumetric optimization framework that combines explicit SDF fields with a shallow color network, in order to estimate 3D shape properties over tetrahedral grids. Experimental results with Chamfer statistics validate this approach with unprecedented reconstruction quality on various scenarios such as objects, open scenes or human.
  arXiv  Detail & Related papers  (2024-07-29T09:46:39Z)
• KP-RED: Exploiting Semantic Keypoints for Joint 3D Shape Retrieval and Deformation [87.23575166061413]
  KP-RED is a unified KeyPoint-driven REtrieval and Deformation framework.<n>It takes object scans as input and jointly retrieves and deforms the most geometrically similar CAD models.
  arXiv  Detail & Related papers  (2024-03-15T08:44:56Z)
• PointOBB: Learning Oriented Object Detection via Single Point Supervision [55.88982271340328]
  This paper proposes PointOBB, the first single Point-based OBB generation method, for oriented object detection. PointOBB operates through the collaborative utilization of three distinctive views: an original view, a resized view, and a rotated/flipped (rot/flp) view. Experimental results on the DIOR-R and DOTA-v1.0 datasets demonstrate that PointOBB achieves promising performance.
  arXiv  Detail & Related papers  (2023-11-23T15:51:50Z)
• Geometric-aware Pretraining for Vision-centric 3D Object Detection [77.7979088689944]
  We propose a novel geometric-aware pretraining framework called GAPretrain. GAPretrain serves as a plug-and-play solution that can be flexibly applied to multiple state-of-the-art detectors. We achieve 46.2 mAP and 55.5 NDS on the nuScenes val set using the BEVFormer method, with a gain of 2.7 and 2.1 points, respectively.
  arXiv  Detail & Related papers  (2023-04-06T14:33:05Z)
• Active 3D Shape Reconstruction from Vision and Touch [66.08432412497443]
  Humans build 3D understandings of the world through active object exploration, using jointly their senses of vision and touch. In 3D shape reconstruction, most recent progress has relied on static datasets of limited sensory data such as RGB images, depth maps or haptic readings. We introduce a system composed of: 1) a haptic simulator leveraging high spatial resolution vision-based tactile sensors for active touching of 3D objects; 2) a mesh-based 3D shape reconstruction model that relies on tactile or visuotactile priors to guide the shape exploration; and 3) a set of data-driven solutions with either tactile or visuo
  arXiv  Detail & Related papers  (2021-07-20T15:56:52Z)
• From Points to Multi-Object 3D Reconstruction [71.17445805257196]
  We propose a method to detect and reconstruct multiple 3D objects from a single RGB image. A keypoint detector localizes objects as center points and directly predicts all object properties, including 9-DoF bounding boxes and 3D shapes. The presented approach performs lightweight reconstruction in a single-stage, it is real-time capable, fully differentiable and end-to-end trainable.
  arXiv  Detail & Related papers  (2020-12-21T18:52:21Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.