Adaptive Visual Imitation Learning for Robotic Assisted Feeding Across Varied Bowl Configurations and Food Types

• URL: http://arxiv.org/abs/2403.12891v1
• Date: Tue, 19 Mar 2024 16:40:57 GMT
• Title: Adaptive Visual Imitation Learning for Robotic Assisted Feeding Across Varied Bowl Configurations and Food Types
• Authors: Rui Liu, Amisha Bhaskar, Pratap Tokekar,
• Abstract summary: We introduce a novel visual imitation network with a spatial attention module for robotic assisted feeding (RAF) We propose a framework that integrates visual perception with imitation learning to enable the robot to handle diverse scenarios during scooping. Our approach, named AVIL (adaptive visual imitation learning), exhibits adaptability and robustness across different bowl configurations.
• Score: 17.835835270751176
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: In this study, we introduce a novel visual imitation network with a spatial attention module for robotic assisted feeding (RAF). The goal is to acquire (i.e., scoop) food items from a bowl. However, achieving robust and adaptive food manipulation is particularly challenging. To deal with this, we propose a framework that integrates visual perception with imitation learning to enable the robot to handle diverse scenarios during scooping. Our approach, named AVIL (adaptive visual imitation learning), exhibits adaptability and robustness across different bowl configurations in terms of material, size, and position, as well as diverse food types including granular, semi-solid, and liquid, even in the presence of distractors. We validate the effectiveness of our approach by conducting experiments on a real robot. We also compare its performance with a baseline. The results demonstrate improvement over the baseline across all scenarios, with an enhancement of up to 2.5 times in terms of a success metric. Notably, our model, trained solely on data from a transparent glass bowl containing granular cereals, showcases generalization ability when tested zero-shot on other bowl configurations with different types of food.

Related papers

• ActionCOMET: A Zero-shot Approach to Learn Image-specific Commonsense Concepts about Actions [66.20773952864802]
  We develop a dataset consisting of 8.5k images and 59.3k inferences about actions grounded in those images. We propose ActionCOMET, a framework to discern knowledge present in language models specific to the provided visual input.
  arXiv  Detail & Related papers  (2024-10-17T15:22:57Z)
• IMRL: Integrating Visual, Physical, Temporal, and Geometric Representations for Enhanced Food Acquisition [16.32678094159896]
  We introduce IMRL (Integrated Multi-Dimensional Representation Learning), which integrates visual, physical, temporal, and geometric representations to enhance robustness and generalizability of IL for food acquisition. Our approach captures food types and physical properties, models temporal dynamics of acquisition actions, and introduces geometric information to determine optimal scooping points. IMRL enables IL to adaptively adjust scooping strategies based on context, improving the robot's capability to handle diverse food acquisition scenarios.
  arXiv  Detail & Related papers  (2024-09-18T16:09:06Z)
• FLAIR: Feeding via Long-horizon AcquIsition of Realistic dishes [23.72810526053693]
  FLAIR is a system for long-horizon feeding which leverages the commonsense and few-shot reasoning capabilities of foundation models. In real-world evaluations across 6 realistic plates, we find that FLAIR can effectively tap into a varied library of skills for efficient food pickup.
  arXiv  Detail & Related papers  (2024-07-10T11:38:57Z)
• NutritionVerse: Empirical Study of Various Dietary Intake Estimation Approaches [59.38343165508926]
  Accurate dietary intake estimation is critical for informing policies and programs to support healthy eating. Recent work has focused on using computer vision and machine learning to automatically estimate dietary intake from food images. We introduce NutritionVerse- Synth, the first large-scale dataset of 84,984 synthetic 2D food images with associated dietary information. We also collect a real image dataset, NutritionVerse-Real, containing 889 images of 251 dishes to evaluate realism.
  arXiv  Detail & Related papers  (2023-09-14T13:29:41Z)
• Learning Sequential Acquisition Policies for Robot-Assisted Feeding [37.371967116072966]
  We propose Visual Action Planning OveR Sequences (VAPORS) as a framework for long-horizon food acquisition. VAPORS learns a policy for high-level action selection by leveraging learned latent plate dynamics in simulation. We validate our approach on complex real-world acquisition trials involving noodle acquisition and bimanual scooping of jelly beans.
  arXiv  Detail & Related papers  (2023-09-11T02:20:28Z)
• Food Image Classification and Segmentation with Attention-based Multiple Instance Learning [51.279800092581844]
  The paper presents a weakly supervised methodology for training food image classification and semantic segmentation models. The proposed methodology is based on a multiple instance learning approach in combination with an attention-based mechanism. We conduct experiments on two meta-classes within the FoodSeg103 data set to verify the feasibility of the proposed approach.
  arXiv  Detail & Related papers  (2023-08-22T13:59:47Z)
• Transferring Knowledge for Food Image Segmentation using Transformers and Convolutions [65.50975507723827]
  Food image segmentation is an important task that has ubiquitous applications, such as estimating the nutritional value of a plate of food. One challenge is that food items can overlap and mix, making them difficult to distinguish. Two models are trained and compared, one based on convolutional neural networks and the other on Bidirectional representation for Image Transformers (BEiT) The BEiT model outperforms the previous state-of-the-art model by achieving a mean intersection over union of 49.4 on FoodSeg103.
  arXiv  Detail & Related papers  (2023-06-15T15:38:10Z)
• Rethinking Cooking State Recognition with Vision Transformers [0.0]
  Self-attention mechanism of Vision Transformer (ViT) architecture is proposed for the Cooking State Recognition task. The proposed approach encapsulates the globally salient features from images, while also exploiting the weights learned from a larger dataset. Our framework has an accuracy of 94.3%, which significantly outperforms the state-of-the-art.
  arXiv  Detail & Related papers  (2022-12-16T17:06:28Z)
• Learning Visuo-Haptic Skewering Strategies for Robot-Assisted Feeding [13.381485293778654]
  We leverage visual and haptic observations during interaction with an item to plan skewering motions. We learn a generalizable, multimodal representation for a food item from raw sensory inputs. We propose a zero-shot framework to sense visuo-haptic properties of a previously unseen item and reactively skewer it.
  arXiv  Detail & Related papers  (2022-11-26T20:01:03Z)
• Visual Imitation Made Easy [102.36509665008732]
  We present an alternate interface for imitation that simplifies the data collection process while allowing for easy transfer to robots. We use commercially available reacher-grabber assistive tools both as a data collection device and as the robot's end-effector. We experimentally evaluate on two challenging tasks: non-prehensile pushing and prehensile stacking, with 1000 diverse demonstrations for each task.
  arXiv  Detail & Related papers  (2020-08-11T17:58:50Z)
• Cross-Modal Food Retrieval: Learning a Joint Embedding of Food Images and Recipes with Semantic Consistency and Attention Mechanism [70.85894675131624]
  We learn an embedding of images and recipes in a common feature space, such that the corresponding image-recipe embeddings lie close to one another. We propose Semantic-Consistent and Attention-based Networks (SCAN), which regularize the embeddings of the two modalities through aligning output semantic probabilities. We show that we can outperform several state-of-the-art cross-modal retrieval strategies for food images and cooking recipes by a significant margin.
  arXiv  Detail & Related papers  (2020-03-09T07:41:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.