TACTIC: Tactile and Vision Conditioned Contact-Centric Control for Whole-Arm Manipulation

• URL: http://arxiv.org/abs/2607.09218v2
• Date: Mon, 13 Jul 2026 03:55:12 GMT
• Title: TACTIC: Tactile and Vision Conditioned Contact-Centric Control for Whole-Arm Manipulation
• Abstract summary: Whole-arm manipulation involves direct contact with the environment while the robot completes a task.<n>We propose TACTIC (Tactile and Vision Conditioned Contact-Centric Control), a receding-horizon controller for whole-arm manipulation.
• Score: 3.8628620899766553
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Whole-arm manipulation involves direct contact with the environment while the robot completes a task by distributing contact across multiple links as contacts form, slide, and break. This setting breaks common implicit assumptions in many learning-based manipulation pipelines: arm configuration tightly couples motion and contact forces, contact state is partially observed under occlusion, and purely learned rollouts can become physically inconsistent under distribution shift because many multi-link contact configurations are sparsely represented in the data. To address this, we propose TACTIC (Tactile and Vision Conditioned Contact-Centric Control), a receding-horizon controller for whole-arm manipulation. TACTIC uses a contact-centric hybrid predictive model that combines RGB-D, distributed tactile sensing, and a compact 2D proximity representation. The model couples a learned, action-conditioned latent dynamics model with analytical kinematics through contact Jacobians, enabling rollouts of future contact configurations and interaction forces. TACTIC integrates these rollouts into a sampling-based MPC planner with contact-aware action sampling: contact Jacobian-based projections steer sampled action sequences toward force-modulating directions, and objectives defined over predicted proximity and interaction forces trade task progress against whole-arm force regulation. We evaluate TACTIC in simulation against state-of-the-art model-based and model-free methods, and perform ablations that isolate the contribution of each design choice. TACTIC consistently outperforms other methods. We further demonstrate real-world performance on a robot with distributed tactile sensing across three whole-arm manipulation tasks that require multi-contact trajectories: turning over and repositioning a manikin, and goal-reaching in a 3D dynamic maze. Website: https://emprise.cs.cornell.edu/tactic

Related papers

• UniTacVLA: Unified Tactile Understanding and Prediction in Vision Language Action Models [32.87879913150572]
  We propose a unified tactile learning framework for contact-rich manipulation.<n>We show that our method improves success rate, manipulation accuracy, and contact robustness over existing methods.
  arXiv  Detail & Related papers  (2026-06-30T14:24:00Z)
• TacForeSight: Force-Guided Tactile World Model for Contact-Rich Manipulation [46.82050380051203]
  Contact-rich manipulation requires robots to continuously regulate and regulate physical dynamic contact transitions or complex surface geometries.<n>Recent imitation learning methods improve contact-aware control by incorporating tactile or force feedback, but they rarely model the asymmetric roles of global force and local tactile sensing.<n>We propose TacForeSight, a lightweight force foresight framework for real-time manipulation.
  arXiv  Detail & Related papers  (2026-06-09T17:59:03Z)
• OmniVTA: Visuo-Tactile World Modeling for Contact-Rich Robotic Manipulation [57.133721026727706]
  We present textbfOmniViTac, a large-scale visuo-tactile-action dataset comprising $21,000+$ trajectories across $86$ tasks and $100+$ objects.<n>We propose textbf OmniVTA, a world-model-based visuo-tactile manipulation framework that integrates four tightly coupled modules.
  arXiv  Detail & Related papers  (2026-03-19T17:52:42Z)
• CONTACT: CONtact-aware TACTile Learning for Robotic Disassembly [5.119250053135017]
  We investigate the role of tactile sensing in robotic disassembly through both simulation and real-world experiments.<n>We compare three sensing configurations: Vision Only, Vision + tactile RGB (TacRGB), and Vision + tactile force field (TacFF)<n>Our results show that tactile sensing plays a critical, task-dependent role in robotic disassembly, with structured force-field representations being particularly effective in contact-dominated scenarios.
  arXiv  Detail & Related papers  (2026-03-09T16:20:49Z)
• Contact-Grounded Policy: Dexterous Visuotactile Policy with Generative Contact Grounding [5.936373185672394]
  Contact-Grounded Policy (CGP) is a visuotactile policy that grounds multi-point contacts by predicting coupled trajectories of actual robot state and tactile feedback.<n>CGP consists of two components: (i) a conditional diffusion model that forecasts future robot state and tactile feedback in a compressed latent space, and (ii) a learned contact-consistency mapping.<n>We evaluate CGP using a physical four-finger Allegro V5 hand with Digit360 fingertip tactile sensors, and a simulated five-finger Tesollo DG-5F hand with dense whole-hand tactile arrays.
  arXiv  Detail & Related papers  (2026-03-05T21:22:49Z)
• Semantic-Contact Fields for Category-Level Generalizable Tactile Tool Manipulation [82.63833405368159]
  Generalizing tool manipulation requires both semantic planning and precise physical control.<n>We propose Semantic-Contact Fields (SCFields), a unified 3D representation fusing visual semantics with dense contact estimates.<n> Experiments on scraping, crayon drawing, and peeling demonstrate robust category-level generalization.
  arXiv  Detail & Related papers  (2026-02-14T16:05:08Z)
• UniForce: A Unified Latent Force Model for Robot Manipulation with Diverse Tactile Sensors [51.88112610411651]
  We propose a novel unified tactile representation learning framework that learns a shared latent force space across diverse tactile sensors.<n>UniForce reduces cross-sensor domain shift by jointly modeling inverse dynamics (image-to-force) and forward dynamics (force-to-image)<n>To avoid reliance on expensive external force/torque (F/T) sensors, we exploit static equilibrium and collect force-paired data via direct sensor--object--sensor interactions.
  arXiv  Detail & Related papers  (2026-02-01T11:03:01Z)
• Nonprehensile Riemannian Motion Predictive Control [57.295751294224765]
  We introduce a novel Real-to-Sim reward analysis technique to reliably imagine and predict the outcome of taking possible actions for a real robotic platform. We produce a closed-loop controller to reactively push objects in a continuous action space. We observe that RMPC is robust in cluttered as well as occluded environments and outperforms the baselines.
  arXiv  Detail & Related papers  (2021-11-15T18:50:04Z)
• Elastic Tactile Simulation Towards Tactile-Visual Perception [58.44106915440858]
  We propose Elastic Interaction of Particles (EIP) for tactile simulation. EIP models the tactile sensor as a group of coordinated particles, and the elastic property is applied to regulate the deformation of particles during contact. We further propose a tactile-visual perception network that enables information fusion between tactile data and visual images.
  arXiv  Detail & Related papers  (2021-08-11T03:49:59Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.