Multi-Sensor Next-Best-View Planning as Matroid-Constrained Submodular Maximization

• URL: http://arxiv.org/abs/2007.02084v1
• Date: Sat, 4 Jul 2020 12:28:18 GMT
• Title: Multi-Sensor Next-Best-View Planning as Matroid-Constrained Submodular Maximization
• Abstract summary: We consider the problem of creating a 3D model using depth images captured by a team of multiple robots. Next-best-view planning uses the current scene model to select the next viewpoints. We show that multi-sensor next-best-view planning with this utility function is an instance of submodular under a matroid constraint.
• Score: 42.2136160962607
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: 3D scene models are useful in robotics for tasks such as path planning, object manipulation, and structural inspection. We consider the problem of creating a 3D model using depth images captured by a team of multiple robots. Each robot selects a viewpoint and captures a depth image from it, and the images are fused to update the scene model. The process is repeated until a scene model of desired quality is obtained. Next-best-view planning uses the current scene model to select the next viewpoints. The objective is to select viewpoints so that the images captured using them improve the quality of the scene model the most. In this paper, we address next-best-view planning for multiple depth cameras. We propose a utility function that scores sets of viewpoints and avoids overlap between multiple sensors. We show that multi-sensor next-best-view planning with this utility function is an instance of submodular maximization under a matroid constraint. This allows the planning problem to be solved by a polynomial-time greedy algorithm that yields a solution within a constant factor from the optimal. We evaluate the performance of our planning algorithm in simulated experiments with up to 8 sensors, and in real-world experiments using two robot arms equipped with depth cameras.

Related papers

• Viewpoint Matters: Dynamically Optimizing Viewpoints with Masked Autoencoder for Visual Manipulation [9.420906356149874]
  We propose MAE-Select, a novel framework for active viewpoint selection in single-camera robotic systems.<n>MaE-Select fully leverages pre-trained multi-view masked autoencoder representations and dynamically selects the next most informative viewpoint at each time chunk.<n>Experiments demonstrate that MAE-Select improves the capabilities of single-camera systems and, in some cases, even surpasses multi-camera setups.
  arXiv  Detail & Related papers  (2026-02-04T06:05:39Z)
• Manipulation as in Simulation: Enabling Accurate Geometry Perception in Robots [55.43376513158555]
  Camera Depth Models (CDMs) are a simple plugin on daily-use depth cameras.<n>We develop a neural data engine that generates high-quality paired data from simulation by modeling a depth camera's noise pattern.<n>For the first time, our experiments demonstrate, for the first time, that a policy trained on raw simulated depth, without the need for adding noise or real-world fine-tuning, generalizes seamlessly to real-world robots.
  arXiv  Detail & Related papers  (2025-09-02T17:29:38Z)
• Omni-Scan: Creating Visually-Accurate Digital Twin Object Models Using a Bimanual Robot with Handover and Gaussian Splat Merging [17.607640140471936]
  "Digital twins" are useful for simulations, virtual reality, marketing, robot policy fine-tuning, and part inspection.<n>We propose Omni-Scan, a pipeline for producing high-quality 3D Gaussian Splat models using a bi-manual robot that grasps an object with one gripper and rotates the object with respect to a stationary camera.<n>We present the Omni-Scan robot pipeline using DepthAny-thing, Segment Anything, as well as RAFT optical flow models to identify and isolate objects held by a robot gripper while removing the gripper and the background.
  arXiv  Detail & Related papers  (2025-08-01T06:36:19Z)
• Track Initialization and Re-Identification for~3D Multi-View Multi-Object Tracking [12.389483990547223]
  We propose a 3D multi-object tracking (MOT) solution using only 2D detections from monocular cameras. We exploit the 2D detections and extracted features from multiple cameras to provide a better approximation of the multi-object filtering density.
  arXiv  Detail & Related papers  (2024-05-28T21:36:16Z)
• Multi-Model 3D Registration: Finding Multiple Moving Objects in Cluttered Point Clouds [23.923838486208524]
  We investigate a variation of the 3D registration problem, named multi-model 3D registration. In the multi-model registration problem, we are given two point clouds picturing a set of objects at different poses. We want to simultaneously reconstruct how all objects moved between the two point clouds.
  arXiv  Detail & Related papers  (2024-02-16T18:01:43Z)
• ImageManip: Image-based Robotic Manipulation with Affordance-guided Next View Selection [10.162882793554191]
  3D articulated object manipulation is essential for enabling robots to interact with their environment. Many existing studies make use of 3D point clouds as the primary input for manipulation policies. RGB images offer high-resolution observations using cost effective devices but lack spatial 3D geometric information. This framework is designed to capture multiple perspectives of the target object and infer depth information to complement its geometry.
  arXiv  Detail & Related papers  (2023-10-13T12:42:54Z)
• Monocular 3D Object Detection with Depth from Motion [74.29588921594853]
  We take advantage of camera ego-motion for accurate object depth estimation and detection. Our framework, named Depth from Motion (DfM), then uses the established geometry to lift 2D image features to the 3D space and detects 3D objects thereon. Our framework outperforms state-of-the-art methods by a large margin on the KITTI benchmark.
  arXiv  Detail & Related papers  (2022-07-26T15:48:46Z)
• SurroundDepth: Entangling Surrounding Views for Self-Supervised Multi-Camera Depth Estimation [101.55622133406446]
  We propose a SurroundDepth method to incorporate the information from multiple surrounding views to predict depth maps across cameras. Specifically, we employ a joint network to process all the surrounding views and propose a cross-view transformer to effectively fuse the information from multiple views. In experiments, our method achieves the state-of-the-art performance on the challenging multi-camera depth estimation datasets.
  arXiv  Detail & Related papers  (2022-04-07T17:58:47Z)
• 3D modelling of survey scene from images enhanced with a multi-exposure fusion [1.7816843507516948]
  We propose a method that can be used to improve the visibility of the images, and eventually reduce the errors of the 3D scene model. The idea is inspired by image dehazing. Each original image is first transformed into multiple exposure images. The enhanced images can reconstruct 3D scene models with sub-millimeter mean errors.
  arXiv  Detail & Related papers  (2021-11-10T06:02:44Z)
• Soft Expectation and Deep Maximization for Image Feature Detection [68.8204255655161]
  We propose SEDM, an iterative semi-supervised learning process that flips the question and first looks for repeatable 3D points, then trains a detector to localize them in image space. Our results show that this new model trained using SEDM is able to better localize the underlying 3D points in a scene.
  arXiv  Detail & Related papers  (2021-04-21T00:35:32Z)
• CAMPARI: Camera-Aware Decomposed Generative Neural Radiance Fields [67.76151996543588]
  We learn a 3D- and camera-aware generative model which faithfully recovers not only the image but also the camera data distribution. At test time, our model generates images with explicit control over the camera as well as the shape and appearance of the scene.
  arXiv  Detail & Related papers  (2021-03-31T17:59:24Z)
• Shape and Viewpoint without Keypoints [63.26977130704171]
  We present a learning framework that learns to recover the 3D shape, pose and texture from a single image. We trained on an image collection without any ground truth 3D shape, multi-view, camera viewpoints or keypoint supervision. We obtain state-of-the-art camera prediction results and show that we can learn to predict diverse shapes and textures across objects.
  arXiv  Detail & Related papers  (2020-07-21T17:58:28Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.