Learning Gentle Grasping Using Vision, Sound, and Touch

• URL: http://arxiv.org/abs/2503.07926v2
• Date: Thu, 24 Jul 2025 16:08:12 GMT
• Title: Learning Gentle Grasping Using Vision, Sound, and Touch
• Abstract summary: This paper proposes using visual, tactile, and auditory signals to learn to grasp and regrasp objects stably and gently.<n>We use audio signals as an indicator of gentleness during the grasping, and then train an action-conditional model from raw visuo-tactile inputs.<n> Experimental results on a multi-fingered hand over 1,500 grasping trials demonstrated that our model is useful for gentle grasping.
• Score: 4.482495915731277
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: In our daily life, we often encounter objects that are fragile and can be damaged by excessive grasping force, such as fruits. For these objects, it is paramount to grasp gently -- not using the maximum amount of force possible, but rather the minimum amount of force necessary. This paper proposes using visual, tactile, and auditory signals to learn to grasp and regrasp objects stably and gently. Specifically, we use audio signals as an indicator of gentleness during the grasping, and then train an end-to-end action-conditional model from raw visuo-tactile inputs that predicts both the stability and the gentleness of future grasping candidates, thus allowing the selection and execution of the most promising action. Experimental results on a multi-fingered hand over 1,500 grasping trials demonstrated that our model is useful for gentle grasping by validating the predictive performance (3.27% higher accuracy than the vision-only variant) and providing interpretations of their behavior. Finally, real-world experiments confirmed that the grasping performance with the trained multi-modal model outperformed other baselines (17% higher rate for stable and gentle grasps than vision-only). Our approach requires neither tactile sensor calibration nor analytical force modeling, drastically reducing the engineering effort to grasp fragile objects. Dataset and videos are available at https://lasr.org/research/gentle-grasping.

Related papers

• Uncertainty-Aware Exploratory Direct Preference Optimization for Multimodal Large Language Models [53.15468578562038]
  We propose an Uncertainty-aware Exploratory Direct Preference Optimization (UE-DPO) method for Multimodal Large Language Models (MLLMs)<n>We first quantify the uncertainty from the model's failure to ground token predictions in the given image.<n>We then encourage more learning pressure on visually deficient tokens in preferred samples, and alleviate the over-penalization of beneficial knowledge in dispreferred samples.
  arXiv  Detail & Related papers  (2026-05-06T13:08:12Z)
• Vision-Language Models are Strong Noisy Label Detectors [76.07846780815794]
  This paper presents a Denoising Fine-Tuning framework, called DeFT, for adapting vision-language models. DeFT utilizes the robust alignment of textual and visual features pre-trained on millions of auxiliary image-text pairs to sieve out noisy labels. Experimental results on seven synthetic and real-world noisy datasets validate the effectiveness of DeFT in both noisy label detection and image classification.
  arXiv  Detail & Related papers  (2024-09-29T12:55:17Z)
• Learning Gentle Grasping from Human-Free Force Control Demonstration [4.08734863805696]
  We propose an approach for learning grasping from ideal force control demonstrations to achieve similar performance of human hands with limited data size.<n>Our approach utilizes objects with known contact characteristics to automatically generate reference force curves without human demonstrations.<n>The described method can be effectively applied in vision-based tactile sensors and enables gentle and stable grasping of objects from the ground.
  arXiv  Detail & Related papers  (2024-09-16T15:14:53Z)
• Adversarial Robustification via Text-to-Image Diffusion Models [56.37291240867549]
  Adrial robustness has been conventionally believed as a challenging property to encode for neural networks. We develop a scalable and model-agnostic solution to achieve adversarial robustness without using any data.
  arXiv  Detail & Related papers  (2024-07-26T10:49:14Z)
• Advancing the Robustness of Large Language Models through Self-Denoised Smoothing [50.54276872204319]
  Large language models (LLMs) have achieved significant success, but their vulnerability to adversarial perturbations has raised considerable concerns. We propose to leverage the multitasking nature of LLMs to first denoise the noisy inputs and then to make predictions based on these denoised versions. Unlike previous denoised smoothing techniques in computer vision, which require training a separate model to enhance the robustness of LLMs, our method offers significantly better efficiency and flexibility.
  arXiv  Detail & Related papers  (2024-04-18T15:47:00Z)
• Existence Is Chaos: Enhancing 3D Human Motion Prediction with Uncertainty Consideration [27.28184416632815]
  We argue that the recorded motion in training data could be an observation of possible future, rather than a predetermined result. A novel computationally efficient encoder-decoder model with uncertainty consideration is proposed.
  arXiv  Detail & Related papers  (2024-03-21T03:34:18Z)
• Neural feels with neural fields: Visuo-tactile perception for in-hand manipulation [57.60490773016364]
  We combine vision and touch sensing on a multi-fingered hand to estimate an object's pose and shape during in-hand manipulation. Our method, NeuralFeels, encodes object geometry by learning a neural field online and jointly tracks it by optimizing a pose graph problem. Our results demonstrate that touch, at the very least, refines and, at the very best, disambiguates visual estimates during in-hand manipulation.
  arXiv  Detail & Related papers  (2023-12-20T22:36:37Z)
• Exploring Model Transferability through the Lens of Potential Energy [78.60851825944212]
  Transfer learning has become crucial in computer vision tasks due to the vast availability of pre-trained deep learning models. Existing methods for measuring the transferability of pre-trained models rely on statistical correlations between encoded static features and task labels. We present an insightful physics-inspired approach named PED to address these challenges.
  arXiv  Detail & Related papers  (2023-08-29T07:15:57Z)
• Robustness of Demonstration-based Learning Under Limited Data Scenario [54.912936555876826]
  Demonstration-based learning has shown great potential in stimulating pretrained language models' ability under limited data scenario. Why such demonstrations are beneficial for the learning process remains unclear since there is no explicit alignment between the demonstrations and the predictions. In this paper, we design pathological demonstrations by gradually removing intuitively useful information from the standard ones to take a deep dive of the robustness of demonstration-based sequence labeling.
  arXiv  Detail & Related papers  (2022-10-19T16:15:04Z)
• PoseIt: A Visual-Tactile Dataset of Holding Poses for Grasp Stability Analysis [7.644487678804541]
  We present PoseIt, a novel dataset that contains visual and tactile data collected from a full cycle of grasping an object. Using data from PoseIt, we can formulate and tackle the task of predicting whether a grasped object is stable in a particular held pose. Our experimental results show that multi-modal models trained on PoseIt achieve higher accuracy than using solely vision or tactile data.
  arXiv  Detail & Related papers  (2022-09-12T04:49:41Z)
• Action Conditioned Tactile Prediction: case study on slip prediction [0.562479170374811]
  We present two novel data-driven action-conditioned models for predicting tactile signals during real-world physical robot interaction tasks. We use a magnetic-based tactile sensor that is challenging to analyse and test state-of-the-art predictive models and the only existing bespoke tactile prediction model. We perform the comparison study using our novel tactile-enabled dataset containing 51,000 tactile frames of a real-world robotic manipulation task with 11 flat-surfaced household objects.
  arXiv  Detail & Related papers  (2022-05-19T09:45:26Z)
• Visual-Tactile Multimodality for Following Deformable Linear Objects Using Reinforcement Learning [15.758583731036007]
  We study the problem of using vision and tactile inputs together to complete the task of following deformable linear objects. We create a Reinforcement Learning agent using different sensing modalities and investigate how its behaviour can be boosted. Our experiments show that the use of both vision and tactile inputs, together with proprioception, allows the agent to complete the task in up to 92% of cases.
  arXiv  Detail & Related papers  (2022-03-31T21:59:08Z)
• Leveraging distributed contact force measurements for slip detection: a physics-based approach enabled by a data-driven tactile sensor [5.027571997864706]
  This paper describes a novel model-based slip detection pipeline that can predict possibly failing grasps in real-time. A vision-based tactile sensor that accurately estimates distributed forces was integrated into a grasping setup composed of a six degrees-of-freedom cobot and a two-finger gripper. Results show that the system can reliably predict slip while manipulating objects of different shapes, materials, and weights.
  arXiv  Detail & Related papers  (2021-09-23T17:12:46Z)
• Visual Adversarial Imitation Learning using Variational Models [60.69745540036375]
  Reward function specification remains a major impediment for learning behaviors through deep reinforcement learning. Visual demonstrations of desired behaviors often presents an easier and more natural way to teach agents. We develop a variational model-based adversarial imitation learning algorithm.
  arXiv  Detail & Related papers  (2021-07-16T00:15:18Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.