Realistic Surgical Image Dataset Generation Based On 3D Gaussian Splatting

• URL: http://arxiv.org/abs/2407.14846v1
• Date: Sat, 20 Jul 2024 11:20:07 GMT
• Title: Realistic Surgical Image Dataset Generation Based On 3D Gaussian Splatting
• Authors: Tianle Zeng, Gerardo Loza Galindo, Junlei Hu, Pietro Valdastri, Dominic Jones,
• Abstract summary: This research introduces a novel method that employs 3D Gaussian Splatting to generate synthetic surgical datasets. We developed a data recording system capable of acquiring images alongside tool and camera poses in a surgical scene. Using this pose data, we synthetically replicate the scene, thereby enabling direct comparisons of the synthetic image quality.
• Score: 3.5351922399745166
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Computer vision technologies markedly enhance the automation capabilities of robotic-assisted minimally invasive surgery (RAMIS) through advanced tool tracking, detection, and localization. However, the limited availability of comprehensive surgical datasets for training represents a significant challenge in this field. This research introduces a novel method that employs 3D Gaussian Splatting to generate synthetic surgical datasets. We propose a method for extracting and combining 3D Gaussian representations of surgical instruments and background operating environments, transforming and combining them to generate high-fidelity synthetic surgical scenarios. We developed a data recording system capable of acquiring images alongside tool and camera poses in a surgical scene. Using this pose data, we synthetically replicate the scene, thereby enabling direct comparisons of the synthetic image quality (29.592 PSNR). As a further validation, we compared two YOLOv5 models trained on the synthetic and real data, respectively, and assessed their performance in an unseen real-world test dataset. Comparing the performances, we observe an improvement in neural network performance, with the synthetic-trained model outperforming the real-world trained model by 12%, testing both on real-world data.

Related papers

• Evaluating Utility of Memory Efficient Medical Image Generation: A Study on Lung Nodule Segmentation [0.0]
  This work proposes a memory-efficient patch-wise denoising diffusion probabilistic model (DDPM) for generating synthetic medical images. Our approach generates high-utility synthetic images with nodule segmentation while efficiently managing memory constraints. We evaluate the method in two scenarios: training a segmentation model exclusively on synthetic data, and augmenting real-world training data with synthetic images.
  arXiv  Detail & Related papers  (2024-10-16T13:20:57Z)
• Deep Domain Adaptation: A Sim2Real Neural Approach for Improving Eye-Tracking Systems [80.62854148838359]
  Eye image segmentation is a critical step in eye tracking that has great influence over the final gaze estimate. We use dimensionality-reduction techniques to measure the overlap between the target eye images and synthetic training data. Our methods result in robust, improved performance when tackling the discrepancy between simulation and real-world data samples.
  arXiv  Detail & Related papers  (2024-03-23T22:32:06Z)
• Zero123-6D: Zero-shot Novel View Synthesis for RGB Category-level 6D Pose Estimation [66.3814684757376]
  This work presents Zero123-6D, the first work to demonstrate the utility of Diffusion Model-based novel-view-synthesizers in enhancing RGB 6D pose estimation at category-level. The outlined method shows reduction in data requirements, removal of the necessity of depth information in zero-shot category-level 6D pose estimation task, and increased performance, quantitatively demonstrated through experiments on the CO3D dataset.
  arXiv  Detail & Related papers  (2024-03-21T10:38:18Z)
• Domain adaptation strategies for 3D reconstruction of the lumbar spine using real fluoroscopy data [9.21828361691977]
  This study tackles key obstacles in adopting surgical navigation in orthopedic surgeries. It shows an approach for generating 3D anatomical models of the spine from only a few fluoroscopic images. It achieved an 84% F1 score, matching the accuracy of our previous synthetic data-based research.
  arXiv  Detail & Related papers  (2024-01-29T10:22:45Z)
• UAV-Sim: NeRF-based Synthetic Data Generation for UAV-based Perception [62.71374902455154]
  We leverage recent advancements in neural rendering to improve static and dynamic novelview UAV-based image rendering. We demonstrate a considerable performance boost when a state-of-the-art detection model is optimized primarily on hybrid sets of real and synthetic data.
  arXiv  Detail & Related papers  (2023-10-25T00:20:37Z)
• Synthetic Image Data for Deep Learning [0.294944680995069]
  Realistic synthetic image data rendered from 3D models can be used to augment image sets and train image classification semantic segmentation models. We show how high quality physically-based rendering and domain randomization can efficiently create a large synthetic dataset based on production 3D CAD models of a real vehicle.
  arXiv  Detail & Related papers  (2022-12-12T20:28:13Z)
• Is synthetic data from generative models ready for image recognition? [69.42645602062024]
  We study whether and how synthetic images generated from state-of-the-art text-to-image generation models can be used for image recognition tasks. We showcase the powerfulness and shortcomings of synthetic data from existing generative models, and propose strategies for better applying synthetic data for recognition tasks.
  arXiv  Detail & Related papers  (2022-10-14T06:54:24Z)
• SyntheX: Scaling Up Learning-based X-ray Image Analysis Through In Silico Experiments [12.019996672009375]
  We show that creating realistic simulated images from human models is a viable alternative to large-scale in situ data collection. Because synthetic generation of training data from human-based models scales easily, we find that our model transfer paradigm for X-ray image analysis, which we refer to as SyntheX, can even outperform real data-trained models.
  arXiv  Detail & Related papers  (2022-06-13T13:08:41Z)
• Simple and Effective Synthesis of Indoor 3D Scenes [78.95697556834536]
  We study the problem of immersive 3D indoor scenes from one or more images. Our aim is to generate high-resolution images and videos from novel viewpoints. We propose an image-to-image GAN that maps directly from reprojections of incomplete point clouds to full high-resolution RGB-D images.
  arXiv  Detail & Related papers  (2022-04-06T17:54:46Z)
• Synthetic Data and Hierarchical Object Detection in Overhead Imagery [0.0]
  We develop novel synthetic data generation and augmentation techniques for enhancing low/zero-sample learning in satellite imagery. To test the effectiveness of synthetic imagery, we employ it in the training of detection models and our two stage model, and evaluate the resulting models on real satellite images.
  arXiv  Detail & Related papers  (2021-01-29T22:52:47Z)
• Intrinsic Autoencoders for Joint Neural Rendering and Intrinsic Image Decomposition [67.9464567157846]
  We propose an autoencoder for joint generation of realistic images from synthetic 3D models while simultaneously decomposing real images into their intrinsic shape and appearance properties. Our experiments confirm that a joint treatment of rendering and decomposition is indeed beneficial and that our approach outperforms state-of-the-art image-to-image translation baselines both qualitatively and quantitatively.
  arXiv  Detail & Related papers  (2020-06-29T12:53:58Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.