SoftVTBench: A Deformation-Aware Visuo-Tactile Dataset and Benchmark for Deformable-Object Manipulation

• URL: http://arxiv.org/abs/2608.18701v1
• Date: Wed, 19 Aug 2026 08:58:47 GMT
• Title: SoftVTBench: A Deformation-Aware Visuo-Tactile Dataset and Benchmark for Deformable-Object Manipulation
• Abstract summary: We introduce SoftVTBench, a visuo-tactile dataset for physical-interaction-aware deformable-object manipulation.<n>It contains 4,000 expert demonstrations and more than 50 assets, including volumetric deformable objects and visually matched rigid twins.<n>At 20 Hz, each episode synchronizes multi-view RGB, dual-finger tactile RGB and marker motion, proprioception, language, and binary and continuous gripper actions.
• Score: 56.16187327142104
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Physical interaction quality is central to deformable-object manipulation, yet most benchmarks evaluate task success alone. A policy may complete the task while allowing slip or causing excessive compression. A primary bottleneck is the absence of visuo-tactile datasets that pair policy-visible contact observations with independent physical ground truth over complete tasks. We introduce SoftVTBench, a visuo-tactile dataset for physical-interaction-aware deformable-object manipulation. It contains 4,000 expert demonstrations and more than 50 assets, including volumetric deformable objects and visually matched rigid twins. At 20 Hz, each episode synchronizes multi-view RGB, dual-finger tactile RGB and marker motion, proprioception, language, and binary and continuous gripper actions, alongside evaluator-only finite-element (FEM) states. Building upon this dataset, we establish a closed-loop benchmark that uses fixed object-specific calibration to define the Deformation-aware Success Rate (DSR), which counts a rollout as successful only when it completes the task and keeps peak normalized deformation within tolerance. Across Diffusion Policy, $π_{0.5}$, and FastWAM, all 12 in-distribution configurations contain successful rollouts that violate the deformation tolerance, accounting for 0.7--24% of each configuration's successes. Under distribution shift, visuo-tactile variants achieve higher task success in all six policy--suite comparisons and higher DSR in five, whereas their in-distribution benefits are mixed. These results show that making touch available does not by itself ensure effective multimodal fusion. SoftVTBench therefore provides a common visuo-tactile resource for studying not only whether a policy succeeds, but how it physically interacts with deformable objects and when touch improves that interaction.

Related papers

• Neptuna: A Comprehensive Machine Learning Framework for Benchmarking Complex Multiphase Flows [0.0]
  We introduce the first large-scale benchmark specifically designed for shock-driven compressible multiphase flows.<n>This benchmark comprises 2.4 TB of high-fidelity 2D and 3D datasets featuring shock-induced bubble collapse and droplet breakup.<n>We evaluate diverse surrogate model families on our benchmarking framework: Neptuna.
  arXiv  Detail & Related papers  (2026-07-24T13:20:01Z)
• SoftVTBench: A Safety-Aware Visuo-Tactile Benchmark for Physically Constrained Robotic Manipulation of Deformable Objects [56.16187327142104]
  We present SoftVTBench, a safety-aware visuo-tactile benchmark for deformable object manipulation.<n>Built in Isaac Sim with finite-element-simulated deformable objects, SoftVTBench provides multi-view RGB observations, tactile sensing with marker motion, proprioception, and language instructions.
  arXiv  Detail & Related papers  (2026-07-05T11:12:29Z)
• OmniVTA: Visuo-Tactile World Modeling for Contact-Rich Robotic Manipulation [57.133721026727706]
  We present textbfOmniViTac, a large-scale visuo-tactile-action dataset comprising $21,000+$ trajectories across $86$ tasks and $100+$ objects.<n>We propose textbf OmniVTA, a world-model-based visuo-tactile manipulation framework that integrates four tightly coupled modules.
  arXiv  Detail & Related papers  (2026-03-19T17:52:42Z)
• UniVTAC: A Unified Simulation Platform for Visuo-Tactile Manipulation Data Generation, Learning, and Benchmarking [47.900334665380115]
  We propose UniVTAC, a simulation-based visuo-tactile data platform.<n>We introduce the UniVTAC encoder, a visuo-tactile encoder trained on large-scale simulation-synthesized data.<n>We present the UniVTAC Benchmark, which consists of eight representative visuo-tactile manipulation tasks.
  arXiv  Detail & Related papers  (2026-02-10T18:57:00Z)
• TouchFormer: A Robust Transformer-based Framework for Multimodal Material Perception [8.939880394166348]
  We propose a robust multimodal fusion framework, TouchFormer.<n>We employ a Modality-Adaptive Gating mechanism and intra- and inter-modality attention mechanisms to adaptively integrate cross-modal features.<n>We show that TouchFormer achieves classification accuracy improvements of 2.48% and 6.83% on SSMC and subcategory tasks, respectively.
  arXiv  Detail & Related papers  (2025-11-24T00:43:59Z)
• UASTrack: A Unified Adaptive Selection Framework with Modality-Customization in Single Object Tracking [38.307090917666144]
  Multi-modal tracking is essential in single-object tracking (SOT)<n>Existing unified RGB-X trackers (X represents depth, event, or thermal modality) either rely on the task-specific training strategy for individual RGB-X image pairs or fail to address the critical importance of modality-adaptive perception in real-world applications.<n>We propose UASTrack, a unified adaptive selection framework that facilitates both model and parameter unification, as well as adaptive modality discrimination across various multi-modal tracking tasks.
  arXiv  Detail & Related papers  (2025-02-25T14:04:31Z)
• GRAPE: Generalizing Robot Policy via Preference Alignment [58.419992317452376]
  We present GRAPE: Generalizing Robot Policy via Preference Alignment.<n>We show GRAPE increases success rates on in-domain and unseen manipulation tasks by 51.79% and 58.20%, respectively.<n> GRAPE can be aligned with various objectives, such as safety and efficiency, reducing collision rates by 37.44% and rollout step-length by 11.15%, respectively.
  arXiv  Detail & Related papers  (2024-11-28T18:30:10Z)
• Single-Model and Any-Modality for Video Object Tracking [85.83753760853142]
  We introduce Un-Track, a Unified Tracker of a single set of parameters for any modality. To handle any modality, our method learns their common latent space through low-rank factorization and reconstruction techniques. Our Un-Track achieves +8.1 absolute F-score gain, on the DepthTrack dataset, by introducing only +2.14 (over 21.50) GFLOPs with +6.6M (over 93M) parameters.
  arXiv  Detail & Related papers  (2023-11-27T14:17:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.