CapsDT: Diffusion-Transformer for Capsule Robot Manipulation

• URL: http://arxiv.org/abs/2506.16263v1
• Date: Thu, 19 Jun 2025 12:25:48 GMT
• Title: CapsDT: Diffusion-Transformer for Capsule Robot Manipulation
• Abstract summary: Vision-Language-Action (VLA) models have emerged as a prominent research area, showcasing significant potential across a variety of applications.<n>In this work, we design CapsDT, a Diffusion Transformer model for capsule robot manipulation in the stomach.<n>We develop a capsule endoscopy robot system, a capsule robot controlled by a robotic arm-held magnet, addressing different levels of four endoscopy tasks.
• Score: 6.540622306548993
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Vision-Language-Action (VLA) models have emerged as a prominent research area, showcasing significant potential across a variety of applications. However, their performance in endoscopy robotics, particularly endoscopy capsule robots that perform actions within the digestive system, remains unexplored. The integration of VLA models into endoscopy robots allows more intuitive and efficient interactions between human operators and medical devices, improving both diagnostic accuracy and treatment outcomes. In this work, we design CapsDT, a Diffusion Transformer model for capsule robot manipulation in the stomach. By processing interleaved visual inputs, and textual instructions, CapsDT can infer corresponding robotic control signals to facilitate endoscopy tasks. In addition, we developed a capsule endoscopy robot system, a capsule robot controlled by a robotic arm-held magnet, addressing different levels of four endoscopy tasks and creating corresponding capsule robot datasets within the stomach simulator. Comprehensive evaluations on various robotic tasks indicate that CapsDT can serve as a robust vision-language generalist, achieving state-of-the-art performance in various levels of endoscopy tasks while achieving a 26.25% success rate in real-world simulation manipulation.

Related papers

• Scene Graph-Driven Haptic Feedback for Safety Enhancement in Robotic Ophthalmic Surgery via Physically Simulated iOCT [2.7913039067663195]
  This paper presents a novel haptic feedback system for subretinal injection tasks leveraging Scene Graphs (SG)<n>The SG serves as a semantic abstraction layer for the surgical scene, which is then utilized by a deterministic, rule-based engine to generate state-dependent haptic feedback on a robotic input device.<n>Results demonstrate that the SG-driven haptic feedback improved surgical precision, reducing needle alignment error by 14% and improving System Usability Scale (SUS) scores by 8%.
  arXiv  Detail & Related papers  (2026-09-07T18:16:37Z)
• Point Cloud Segmentation for Autonomous Clip Positioning in Laparoscopic Cholecystectomy on a Phantom [18.991664638461227]
  We present the first robotic system to demonstrate autonomous clip positioning on a physical phantom in laparoscopic surgery.<n>We overcome this with a combination of pre-training on 128,000 synthetic point clouds and two novel data augmentation techniques.<n>In real robot experiments, our system localizes targets with the required precision of 0.75mm at a 95% success rate and executes autonomous clip positioning with a 100% success rate.
  arXiv  Detail & Related papers  (2026-06-10T13:12:45Z)
• Open-H-Embodiment: A Large-Scale Dataset for Enabling Foundation Models in Medical Robotics [131.25998619318702]
  We introduce Open-H-Embodiment, the largest open dataset of medical robotic video with synchronized kinematics to date.<n> GR00T-H is the first open foundation vision-language-action model for medical robotics.<n>We train Cosmos-H-Surgical-Simulator, the first action-conditioned world model to enable multi-embodiment surgical simulation.
  arXiv  Detail & Related papers  (2026-04-22T19:05:17Z)
• Learning Surgical Robotic Manipulation with 3D Spatial Priors [73.00031539525202]
  We introduce the Spatial Surgical Transformer (SST), an end-to-end visuomotor policy that empowers surgical robots with 3D spatial awareness.<n>Based on Surgical3D, we finetune a powerful geometric transformer to extract robust 3D latent representations from stereo endoscopes images.<n>SST achieves state-of-the-art performance and strong spatial generalization on complex surgical tasks.
  arXiv  Detail & Related papers  (2026-03-04T07:19:56Z)
• Robotic Grasping and Placement Controlled by EEG-Based Hybrid Visual and Motor Imagery [64.82869118243723]
  We present a framework that integrates EEG-based visual and motor imagery (VI/MI) with robotic control to enable real-time, intention-driven grasping and placement.<n>Motivated by the promise of BCI-driven robotics, this system bridges neural signals with physical control by deploying offline-pretrained decoders in a zero-shot manner.
  arXiv  Detail & Related papers  (2026-03-03T17:41:42Z)
• MiVLA: Towards Generalizable Vision-Language-Action Model with Human-Robot Mutual Imitation Pre-training [102.850162490626]
  We propose MiVLA, a vision-language-action model empowered by human-robot mutual imitation pre-training.<n>We show that MiVLA achieves strong improved generalization capability, outperforming state-of-the-art VLAs.
  arXiv  Detail & Related papers  (2025-12-17T12:59:41Z)
• RobotSeg: A Model and Dataset for Segmenting Robots in Image and Video [56.9581053843815]
  We introduce RobotSeg, a foundation model for robot segmentation in image and video.<n>It addresses the lack of adaptation to articulated robots, reliance on manual prompts, and the need for per-frame training mask annotations.<n>It achieves state-of-the-art performance on both images and videos.
  arXiv  Detail & Related papers  (2025-11-28T07:51:02Z)
• FAST: Efficient Action Tokenization for Vision-Language-Action Models [98.15494168962563]
  We propose a new compression-based tokenization scheme for robot actions, based on the discrete cosine transform.<n>Based on FAST, we release FAST+, a universal robot action tokenizer, trained on 1M real robot action trajectories.
  arXiv  Detail & Related papers  (2025-01-16T18:57:04Z)
• LLaRA: Supercharging Robot Learning Data for Vision-Language Policy [56.505551117094534]
  We introduce LLaRA: Large Language and Robotics Assistant, a framework that formulates robot action policy as visuo-textual conversations.<n>First, we present an automated pipeline to generate conversation-style instruction tuning data for robots from existing behavior cloning datasets.<n>We show that a VLM finetuned with a limited amount of such datasets can produce meaningful action decisions for robotic control.
  arXiv  Detail & Related papers  (2024-06-28T17:59:12Z)
• Creating a Digital Twin of Spinal Surgery: A Proof of Concept [68.37190859183663]
  Surgery digitalization is the process of creating a virtual replica of real-world surgery. We present a proof of concept (PoC) for surgery digitalization that is applied to an ex-vivo spinal surgery. We employ five RGB-D cameras for dynamic 3D reconstruction of the surgeon, a high-end camera for 3D reconstruction of the anatomy, an infrared stereo camera for surgical instrument tracking, and a laser scanner for 3D reconstruction of the operating room and data fusion.
  arXiv  Detail & Related papers  (2024-03-25T13:09:40Z)
• RoboScript: Code Generation for Free-Form Manipulation Tasks across Real and Simulation [77.41969287400977]
  This paper presents textbfRobotScript, a platform for a deployable robot manipulation pipeline powered by code generation. We also present a benchmark for a code generation benchmark for robot manipulation tasks in free-form natural language. We demonstrate the adaptability of our code generation framework across multiple robot embodiments, including the Franka and UR5 robot arms.
  arXiv  Detail & Related papers  (2024-02-22T15:12:00Z)
• General-purpose foundation models for increased autonomy in robot-assisted surgery [4.155479231940454]
  This perspective article aims to provide a path toward increasing robot autonomy in robot-assisted surgery. We argue that surgical robots are uniquely positioned to benefit from general-purpose models and provide three guiding actions toward increased autonomy in robot-assisted surgery.
  arXiv  Detail & Related papers  (2024-01-01T06:15:16Z)
• RoboCat: A Self-Improving Generalist Agent for Robotic Manipulation [33.10577695383743]
  We propose a multi-embodiment, multi-task generalist agent for robotic manipulation called RoboCat. This data spans a large repertoire of motor control skills from simulated and real robotic arms with varying sets of observations and actions. With RoboCat, we demonstrate the ability to generalise to new tasks and robots, both zero-shot as well as through adaptation using only 100-1000 examples.
  arXiv  Detail & Related papers  (2023-06-20T17:35:20Z)
• Robot Learning with Sensorimotor Pre-training [98.7755895548928]
  We present a self-supervised sensorimotor pre-training approach for robotics. Our model, called RPT, is a Transformer that operates on sequences of sensorimotor tokens. We find that sensorimotor pre-training consistently outperforms training from scratch, has favorable scaling properties, and enables transfer across different tasks, environments, and robots.
  arXiv  Detail & Related papers  (2023-06-16T17:58:10Z)
• Collaborative Robotic Ultrasound Tissue Scanning for Surgical Resection Guidance in Neurosurgery [1.372026330898297]
  The aim of this paper is to introduce a robotic platform for autonomous iUS tissue scanning. A key application of the proposed platform is the scanning of brain tissue to guide tumour resection.
  arXiv  Detail & Related papers  (2023-01-19T17:05:07Z)
• Robotic Navigation Autonomy for Subretinal Injection via Intelligent Real-Time Virtual iOCT Volume Slicing [88.99939660183881]
  We propose a framework for autonomous robotic navigation for subretinal injection. Our method consists of an instrument pose estimation method, an online registration between the robotic and the i OCT system, and trajectory planning tailored for navigation to an injection target. Our experiments on ex-vivo porcine eyes demonstrate the precision and repeatability of the method.
  arXiv  Detail & Related papers  (2023-01-17T21:41:21Z)
• ColibriDoc: An Eye-in-Hand Autonomous Trocar Docking System [46.91300647669861]
  We present a platform for autonomous trocar docking that combines computer vision and a robotic setup. Inspired by the Cuban Colibri (hummingbird) aligning its beak to a flower using only vision, we mount a camera onto the endeffector of a robotic system.
  arXiv  Detail & Related papers  (2021-11-30T13:21:37Z)
• Using Conditional Generative Adversarial Networks to Reduce the Effects of Latency in Robotic Telesurgery [0.0]
  In surgery, any micro-delay can injure a patient severely and in some cases, result in fatality. Current surgical robots use calibrated sensors to measure the position of the arms and tools. In this work we present a purely optical approach that provides a measurement of the tool position in relation to the patient's tissues.
  arXiv  Detail & Related papers  (2020-10-07T13:40:44Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.