Latent Object Characteristics Recognition with Visual to Haptic-Audio Cross-modal Transfer Learning

• URL: http://arxiv.org/abs/2403.10689v1
• Date: Fri, 15 Mar 2024 21:18:14 GMT
• Title: Latent Object Characteristics Recognition with Visual to Haptic-Audio Cross-modal Transfer Learning
• Authors: Namiko Saito, Joao Moura, Hiroki Uchida, Sethu Vijayakumar,
• Abstract summary: This work aims to recognise the latent unobservable object characteristics. vision is commonly used for object recognition by robots, but it is ineffective for detecting hidden objects. We propose a cross-modal transfer learning approach from vision to haptic-audio.
• Score: 9.178588671620963
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Recognising the characteristics of objects while a robot handles them is crucial for adjusting motions that ensure stable and efficient interactions with containers. Ahead of realising stable and efficient robot motions for handling/transferring the containers, this work aims to recognise the latent unobservable object characteristics. While vision is commonly used for object recognition by robots, it is ineffective for detecting hidden objects. However, recognising objects indirectly using other sensors is a challenging task. To address this challenge, we propose a cross-modal transfer learning approach from vision to haptic-audio. We initially train the model with vision, directly observing the target object. Subsequently, we transfer the latent space learned from vision to a second module, trained only with haptic-audio and motor data. This transfer learning framework facilitates the representation of object characteristics using indirect sensor data, thereby improving recognition accuracy. For evaluating the recognition accuracy of our proposed learning framework we selected shape, position, and orientation as the object characteristics. Finally, we demonstrate online recognition of both trained and untrained objects using the humanoid robot Nextage Open.

Related papers

• Localizing Active Objects from Egocentric Vision with Symbolic World Knowledge [62.981429762309226]
  The ability to actively ground task instructions from an egocentric view is crucial for AI agents to accomplish tasks or assist humans virtually. We propose to improve phrase grounding models' ability on localizing the active objects by: learning the role of objects undergoing change and extracting them accurately from the instructions. We evaluate our framework on Ego4D and Epic-Kitchens datasets.
  arXiv  Detail & Related papers  (2023-10-23T16:14:05Z)
• Human-oriented Representation Learning for Robotic Manipulation [64.59499047836637]
  Humans inherently possess generalizable visual representations that empower them to efficiently explore and interact with the environments in manipulation tasks. We formalize this idea through the lens of human-oriented multi-task fine-tuning on top of pre-trained visual encoders. Our Task Fusion Decoder consistently improves the representation of three state-of-the-art visual encoders for downstream manipulation policy-learning.
  arXiv  Detail & Related papers  (2023-10-04T17:59:38Z)
• Visual-Tactile Multimodality for Following Deformable Linear Objects Using Reinforcement Learning [15.758583731036007]
  We study the problem of using vision and tactile inputs together to complete the task of following deformable linear objects. We create a Reinforcement Learning agent using different sensing modalities and investigate how its behaviour can be boosted. Our experiments show that the use of both vision and tactile inputs, together with proprioception, allows the agent to complete the task in up to 92% of cases.
  arXiv  Detail & Related papers  (2022-03-31T21:59:08Z)
• SEAL: Self-supervised Embodied Active Learning using Exploration and 3D Consistency [122.18108118190334]
  We present a framework called Self- Embodied Embodied Active Learning (SEAL) It utilizes perception models trained on internet images to learn an active exploration policy. We and build utilize 3D semantic maps to learn both action and perception in a completely self-supervised manner.
  arXiv  Detail & Related papers  (2021-12-02T06:26:38Z)
• Skeleton-Based Mutually Assisted Interacted Object Localization and Human Action Recognition [111.87412719773889]
  We propose a joint learning framework for "interacted object localization" and "human action recognition" based on skeleton data. Our method achieves the best or competitive performance with the state-of-the-art methods for human action recognition.
  arXiv  Detail & Related papers  (2021-10-28T10:09:34Z)
• Learning Relative Interactions through Imitation [0.0]
  We show that a simple network, with relatively little training data, is able to reach very good performance on the fixed-pose task. We also explore the effect of ambiguities in the sensor readings, in particular caused by symmetries in the target object, on the behaviour of the learned controller.
  arXiv  Detail & Related papers  (2021-09-24T15:18:34Z)
• Maintaining a Reliable World Model using Action-aware Perceptual Anchoring [4.971403153199917]
  There is a need for robots to maintain a model of its surroundings even when objects go out of view and are no longer visible. This requires anchoring perceptual information onto symbols that represent the objects in the environment. We present a model for action-aware perceptual anchoring that enables robots to track objects in a persistent manner.
  arXiv  Detail & Related papers  (2021-07-07T06:35:14Z)
• Simultaneous Multi-View Object Recognition and Grasping in Open-Ended Domains [0.0]
  We propose a deep learning architecture with augmented memory capacities to handle open-ended object recognition and grasping simultaneously. We demonstrate the ability of our approach to grasp never-seen-before objects and to rapidly learn new object categories using very few examples on-site in both simulation and real-world settings.
  arXiv  Detail & Related papers  (2021-06-03T14:12:11Z)
• Careful with That! Observation of Human Movements to Estimate Objects Properties [106.925705883949]
  We focus on the features of human motor actions that communicate insights on the weight of an object. Our final goal is to enable a robot to autonomously infer the degree of care required in object handling.
  arXiv  Detail & Related papers  (2021-03-02T08:14:56Z)
• Unadversarial Examples: Designing Objects for Robust Vision [100.4627585672469]
  We develop a framework that exploits the sensitivity of modern machine learning algorithms to input perturbations in order to design "robust objects" We demonstrate the efficacy of the framework on a wide variety of vision-based tasks ranging from standard benchmarks to (in-simulation) robotics.
  arXiv  Detail & Related papers  (2020-12-22T18:26:07Z)
• A Transfer Learning Approach to Cross-Modal Object Recognition: From Visual Observation to Robotic Haptic Exploration [13.482253411041292]
  We introduce the problem of cross-modal visuo-tactile object recognition with robotic active exploration. We propose an approach constituted by four steps: finding a visuo-tactile common representation, defining a suitable set of features, transferring the features across the domains, and classifying the objects. The proposed approach achieves an accuracy of 94.7%, which is comparable with the accuracy of the monomodal case.
  arXiv  Detail & Related papers  (2020-01-18T14:47:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.