Panoptic Mapping with Fruit Completion and Pose Estimation for Horticultural Robots

• URL: http://arxiv.org/abs/2303.08923v2
• Date: Tue, 22 Aug 2023 08:26:40 GMT
• Title: Panoptic Mapping with Fruit Completion and Pose Estimation for Horticultural Robots
• Authors: Yue Pan, Federico Magistri, Thomas L\"abe, Elias Marks, Claus Smitt, Chris McCool, Jens Behley and Cyrill Stachniss
• Abstract summary: Monitoring plants and fruits at high resolution play a key role in the future of agriculture. Accurate 3D information can pave the way to a diverse number of robotic applications in agriculture ranging from autonomous harvesting to precise yield estimation. We address the problem of jointly estimating complete 3D shapes of fruit and their pose in a 3D multi-resolution map built by a mobile robot.
• Score: 33.21287030243106
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Monitoring plants and fruits at high resolution play a key role in the future of agriculture. Accurate 3D information can pave the way to a diverse number of robotic applications in agriculture ranging from autonomous harvesting to precise yield estimation. Obtaining such 3D information is non-trivial as agricultural environments are often repetitive and cluttered, and one has to account for the partial observability of fruit and plants. In this paper, we address the problem of jointly estimating complete 3D shapes of fruit and their pose in a 3D multi-resolution map built by a mobile robot. To this end, we propose an online multi-resolution panoptic mapping system where regions of interest are represented with a higher resolution. We exploit data to learn a general fruit shape representation that we use at inference time together with an occlusion-aware differentiable rendering pipeline to complete partial fruit observations and estimate the 7 DoF pose of each fruit in the map. The experiments presented in this paper evaluated both in the controlled environment and in a commercial greenhouse, show that our novel algorithm yields higher completion and pose estimation accuracy than existing methods, with an improvement of 41% in completion accuracy and 52% in pose estimation accuracy while keeping a low inference time of 0.6s in average. Codes are available at: https://github.com/PRBonn/HortiMapping.

Related papers

• A Dataset and Benchmark for Shape Completion of Fruits for Agricultural Robotics [30.46518628656399]
  We propose the first publicly available 3D shape completion dataset for agricultural vision systems. We provide an RGB-D dataset for estimating the 3D shape of fruits.
  arXiv  Detail & Related papers  (2024-07-18T09:07:23Z)
• A pipeline for multiple orange detection and tracking with 3-D fruit relocalization and neural-net based yield regression in commercial citrus orchards [0.0]
  We propose a non-invasive alternative that utilizes fruit counting from videos, implemented as a pipeline. To handle occluded and re-appeared fruit, we introduce a relocalization component that employs 3-D estimation of fruit locations. By ensuring that at least 30% of the fruit is accurately detected, tracked, and counted, our yield regressor achieves an impressive coefficient of determination of 0.85.
  arXiv  Detail & Related papers  (2023-12-27T21:22:43Z)
• Key Point-based Orientation Estimation of Strawberries for Robotic Fruit Picking [8.657107511095242]
  We introduce a novel key-point-based fruit orientation estimation method allowing for the prediction of 3D orientation from 2D images directly. Our proposed method achieves state-of-the-art performance with an average error as low as $8circ$, improving predictions by $sim30%$ compared to previous work presented incitewagnerefficient.
  arXiv  Detail & Related papers  (2023-10-17T15:12:11Z)
• HarvestNet: A Dataset for Detecting Smallholder Farming Activity Using Harvest Piles and Remote Sensing [50.4506590177605]
  HarvestNet is a dataset for mapping the presence of farms in the Ethiopian regions of Tigray and Amhara during 2020-2023. We introduce a new approach based on the detection of harvest piles characteristic of many smallholder systems. We conclude that remote sensing of harvest piles can contribute to more timely and accurate cropland assessments in food insecure regions.
  arXiv  Detail & Related papers  (2023-08-23T11:03:28Z)
• Development and evaluation of automated localisation and reconstruction of all fruits on tomato plants in a greenhouse based on multi-view perception and 3D multi-object tracking [0.0]
  This paper presents a novel approach for building generic representations in occluded agro-food environments. It is based on a detection algorithm that generates partial point clouds for each detected object, followed by a 3D multi-object tracking algorithm. The accuracy of the representation was evaluated in a real-world environment, where successful representation and localisation of tomatoes in tomato plants were achieved.
  arXiv  Detail & Related papers  (2022-11-04T21:51:53Z)
• End-to-end deep learning for directly estimating grape yield from ground-based imagery [53.086864957064876]
  This study demonstrates the application of proximal imaging combined with deep learning for yield estimation in vineyards. Three model architectures were tested: object detection, CNN regression, and transformer models. The study showed the applicability of proximal imaging and deep learning for prediction of grapevine yield on a large scale.
  arXiv  Detail & Related papers  (2022-08-04T01:34:46Z)
• Geometry-Aware Fruit Grasping Estimation for Robotic Harvesting in Orchards [6.963582954232132]
  geometry-aware network, A3N, is proposed to perform end-to-end instance segmentation and grasping estimation. We implement a global-to-local scanning strategy, which enables robots to accurately recognise and retrieve fruits in field environments. Overall, the robotic system achieves success rate of harvesting ranging from 70% - 85% in field harvesting experiments.
  arXiv  Detail & Related papers  (2021-12-08T16:17:26Z)
• Multi-View Multi-Person 3D Pose Estimation with Plane Sweep Stereo [71.59494156155309]
  Existing approaches for multi-view 3D pose estimation explicitly establish cross-view correspondences to group 2D pose detections from multiple camera views. We present our multi-view 3D pose estimation approach based on plane sweep stereo to jointly address the cross-view fusion and 3D pose reconstruction in a single shot.
  arXiv  Detail & Related papers  (2021-04-06T03:49:35Z)
• Deep Multi-Task Learning for Joint Localization, Perception, and Prediction [68.50217234419922]
  This paper investigates the issues that arise in state-of-the-art autonomy stacks under localization error. We design a system that jointly performs perception, prediction, and localization. Our architecture is able to reuse computation between both tasks, and is thus able to correct localization errors efficiently.
  arXiv  Detail & Related papers  (2021-01-17T17:20:31Z)
• OmniSLAM: Omnidirectional Localization and Dense Mapping for Wide-baseline Multi-camera Systems [88.41004332322788]
  We present an omnidirectional localization and dense mapping system for a wide-baseline multiview stereo setup with ultra-wide field-of-view (FOV) fisheye cameras. For more practical and accurate reconstruction, we first introduce improved and light-weighted deep neural networks for the omnidirectional depth estimation. We integrate our omnidirectional depth estimates into the visual odometry (VO) and add a loop closing module for global consistency.
  arXiv  Detail & Related papers  (2020-03-18T05:52:10Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.