Demonstrating Multi-Suction Item Picking at Scale via Multi-Modal Learning of Pick Success

• URL: http://arxiv.org/abs/2506.10359v1
• Date: Thu, 12 Jun 2025 05:35:51 GMT
• Title: Demonstrating Multi-Suction Item Picking at Scale via Multi-Modal Learning of Pick Success
• Authors: Che Wang, Jeroen van Baar, Chaitanya Mitash, Shuai Li, Dylan Randle, Weiyao Wang, Sumedh Sontakke, Kostas E. Bekris, Kapil Katyal,
• Abstract summary: This work demonstrates how autonomously learning aspects of robotic operation can provide with solutions that achieve improved performance.<n>Specifically, it focuses on multi-suction robot picking and performs a comprehensive study on the application of multi-modal visual encoders for predicting the success of candidate robotic picks.
• Score: 24.24429138384878
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: This work demonstrates how autonomously learning aspects of robotic operation from sparsely-labeled, real-world data of deployed, engineered solutions at industrial scale can provide with solutions that achieve improved performance. Specifically, it focuses on multi-suction robot picking and performs a comprehensive study on the application of multi-modal visual encoders for predicting the success of candidate robotic picks. Picking diverse items from unstructured piles is an important and challenging task for robot manipulation in real-world settings, such as warehouses. Methods for picking from clutter must work for an open set of items while simultaneously meeting latency constraints to achieve high throughput. The demonstrated approach utilizes multiple input modalities, such as RGB, depth and semantic segmentation, to estimate the quality of candidate multi-suction picks. The strategy is trained from real-world item picking data, with a combination of multimodal pretrain and finetune. The manuscript provides comprehensive experimental evaluation performed over a large item-picking dataset, an item-picking dataset targeted to include partial occlusions, and a package-picking dataset, which focuses on containers, such as boxes and envelopes, instead of unpackaged items. The evaluation measures performance for different item configurations, pick scenes, and object types. Ablations help to understand the effects of in-domain pretraining, the impact of different modalities and the importance of finetuning. These ablations reveal both the importance of training over multiple modalities but also the ability of models to learn during pretraining the relationship between modalities so that during finetuning and inference, only a subset of them can be used as input.

Related papers

• COLLAGE: Adaptive Fusion-based Retrieval for Augmented Policy Learning [19.173177969412656]
  We present COLLAGE, a method for COLLective data AGgrEgation in few-shot imitation learning.<n>Collage uses an adaptive late fusion mechanism to guide the selection of relevant demonstrations based on a task-specific combination of multiple cues.<n>Collage outperforms state-of-the-art retrieval and multi-task learning approaches by 5.1% in simulation across 10 tasks, and by 16.6% in the real world across 6 tasks.
  arXiv  Detail & Related papers  (2025-08-02T01:23:09Z)
• Exploiting Modality-Specific Features For Multi-Modal Manipulation Detection And Grounding [54.49214267905562]
  We construct a transformer-based framework for multi-modal manipulation detection and grounding tasks. Our framework simultaneously explores modality-specific features while preserving the capability for multi-modal alignment. We propose an implicit manipulation query (IMQ) that adaptively aggregates global contextual cues within each modality.
  arXiv  Detail & Related papers  (2023-09-22T06:55:41Z)
• Learning Unseen Modality Interaction [54.23533023883659]
  Multimodal learning assumes all modality combinations of interest are available during training to learn cross-modal correspondences. We pose the problem of unseen modality interaction and introduce a first solution. It exploits a module that projects the multidimensional features of different modalities into a common space with rich information preserved.
  arXiv  Detail & Related papers  (2023-06-22T10:53:10Z)
• SeMAIL: Eliminating Distractors in Visual Imitation via Separated Models [22.472167814814448]
  We propose a new model-based imitation learning algorithm named Separated Model-based Adversarial Imitation Learning (SeMAIL) Our method achieves near-expert performance on various visual control tasks with complex observations and the more challenging tasks with different backgrounds from expert observations.
  arXiv  Detail & Related papers  (2023-06-19T04:33:44Z)
• MetaGraspNet: A Large-Scale Benchmark Dataset for Scene-Aware Ambidextrous Bin Picking via Physics-based Metaverse Synthesis [72.85526892440251]
  We introduce MetaGraspNet, a large-scale photo-realistic bin picking dataset constructed via physics-based metaverse synthesis. The proposed dataset contains 217k RGBD images across 82 different article types, with full annotations for object detection, amodal perception, keypoint detection, manipulation order and ambidextrous grasp labels for a parallel-jaw and vacuum gripper. We also provide a real dataset consisting of over 2.3k fully annotated high-quality RGBD images, divided into 5 levels of difficulties and an unseen object set to evaluate different object and layout properties.
  arXiv  Detail & Related papers  (2022-08-08T08:15:34Z)
• Efficient and Robust Training of Dense Object Nets for Multi-Object Robot Manipulation [8.321536457963655]
  We propose a framework for robust and efficient training of Dense Object Nets (DON) We focus on training with multi-object data instead of singulated objects, combined with a well-chosen augmentation scheme. We demonstrate the robustness and accuracy of our proposed framework on a real-world robotic grasping task.
  arXiv  Detail & Related papers  (2022-06-24T08:24:42Z)
• Towards General and Efficient Active Learning [20.888364610175987]
  Active learning aims to select the most informative samples to exploit limited annotation budgets. We propose a novel general and efficient active learning (GEAL) method in this paper. Our method can conduct data selection processes on different datasets with a single-pass inference of the same model.
  arXiv  Detail & Related papers  (2021-12-15T08:35:28Z)
• Uni-Perceiver: Pre-training Unified Architecture for Generic Perception for Zero-shot and Few-shot Tasks [73.63892022944198]
  We present a generic perception architecture named Uni-Perceiver. It processes a variety of modalities and tasks with unified modeling and shared parameters. Results show that our pre-trained model without any tuning can achieve reasonable performance even on novel tasks.
  arXiv  Detail & Related papers  (2021-12-02T18:59:50Z)
• Mixing Deep Learning and Multiple Criteria Optimization: An Application to Distributed Learning with Multiple Datasets [0.0]
  Training phase is the most important stage during the machine learning process. We develop a multiple criteria optimization model in which each criterion measures the distance between the output associated with a specific input and its label. We propose a scalarization approach to implement this model and numerical experiments in digit classification using MNIST data.
  arXiv  Detail & Related papers  (2021-12-02T16:00:44Z)
• Pretext Tasks selection for multitask self-supervised speech representation learning [23.39079406674442]
  This paper introduces a method to select a group of pretext tasks among a set of candidates. Experiments conducted on speaker recognition and automatic speech recognition validate our approach.
  arXiv  Detail & Related papers  (2021-07-01T16:36:29Z)
• Diverse Complexity Measures for Dataset Curation in Self-driving [80.55417232642124]
  We propose a new data selection method that exploits a diverse set of criteria that quantize interestingness of traffic scenes. Our experiments show that the proposed curation pipeline is able to select datasets that lead to better generalization and higher performance.
  arXiv  Detail & Related papers  (2021-01-16T23:45:02Z)
• Improving Multi-Turn Response Selection Models with Complementary Last-Utterance Selection by Instance Weighting [84.9716460244444]
  We consider utilizing the underlying correlation in the data resource itself to derive different kinds of supervision signals. We conduct extensive experiments in two public datasets and obtain significant improvement in both datasets.
  arXiv  Detail & Related papers  (2020-02-18T06:29:01Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.