Refining 3D Medical Segmentation with Verbal Instruction

• URL: http://arxiv.org/abs/2603.14496v1
• Date: Sun, 15 Mar 2026 17:25:34 GMT
• Title: Refining 3D Medical Segmentation with Verbal Instruction
• Abstract summary: We propose an iterative refinement model that represents 3D shapes as vector sets and interacts with textual instructions to progressively update the target shape.<n> Experimental results demonstrate that our method achieves significant improvements over corrupted inputs and competitive baselines.
• Score: 17.789951952699685
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Accurate 3D anatomical segmentation is essential for clinical diagnosis and surgical planning. However, automated models frequently generate suboptimal shape predictions due to factors such as limited and imbalanced training data, inadequate labeling quality, and distribution shifts between training and deployment settings. A natural solution is to iteratively refine the predicted shape based on the radiologists' verbal instructions. However, this is hindered by the scarcity of paired data that explicitly links erroneous shapes to corresponding corrective instructions. As an initial step toward addressing this limitation, we introduce CoWTalk, a benchmark comprising 3D arterial anatomies with controllable synthesized anatomical errors and their corresponding repairing instructions. Building on this benchmark, we further propose an iterative refinement model that represents 3D shapes as vector sets and interacts with textual instructions to progressively update the target shape. Experimental results demonstrate that our method achieves significant improvements over corrupted inputs and competitive baselines, highlighting the feasibility of language-driven clinician-in-the-loop refinement for 3D medical shapes modeling.

Related papers

• Flow Matching Meets 3D Curvilinear Structure Segmentation in Medical Imaging [0.0]
  We present 3D-CurvSegFlow, a flow matching-based model for 3D curvilinear structure segmentation.<n>The model learns a continuous transformation from a simple source distribution to the target vascular representation, enabling progressive refinement of complex curvilinear geometries.
  arXiv  Detail & Related papers  (2026-08-20T12:35:27Z)
• Anatomy-Grounded Synthetic Coronary Angiography for Geometry-Informed Multi-View Matching [6.620961894485603]
  We introduce a physically-grounded data generation framework that synthesizes high-fidelity Digital Reconstructed Radiographs (DRRs) from 3D Coronary CT Angiography (CCTA) volumes.<n>Our framework generates dense, highly accurate 3D-to-2D projection labels by simulating realistic C-arm acquisition geometry on patient anatomy at zero human cost.<n>Unlike real angiography where assessment relies on subjective human annotation, our dataset provides 2D correspondence labels with paired images, allowing human-free evaluation.
  arXiv  Detail & Related papers  (2026-06-26T14:58:23Z)
• Analyzing and Improving Fine-grained Preference Optimization in Medical LVLMs [53.697403481143404]
  Large Vision-Language Models (LVLMs) have achieved strong performance across medical imaging tasks.<n>But they remain prone to factual inconsistencies, poor visual grounding, and misalignment with clinically meaningful feedback.
  arXiv  Detail & Related papers  (2026-06-10T18:35:36Z)
• Weakly-Supervised Lung Nodule Segmentation via Training-Free Guidance of 3D Rectified Flow [13.15220794725922]
  The proposed method produces improved-quality segmentations for two separate predictors, consistently detecting lung nodules of varying size and shapes.<n>Experiments on LUNA16 demonstrate improvements over baseline methods, highlighting the potential of generative foundation models as tools for weakly supervised 3D medical image segmentation.
  arXiv  Detail & Related papers  (2026-04-09T14:46:14Z)
• MedPruner: Training-Free Hierarchical Token Pruning for Efficient 3D Medical Image Understanding in Vision-Language Models [59.180043227905294]
  We propose MedPruner, a training-free and model-agnostic hierarchical token pruning framework for efficient 3D medical image understanding.<n>We show that MedPruner enables models such as MedGemma to maintain or even exceed their original performance while retaining fewer than 5% of visual tokens.
  arXiv  Detail & Related papers  (2026-03-12T07:37:00Z)
• ClipGS: Clippable Gaussian Splatting for Interactive Cinematic Visualization of Volumetric Medical Data [51.095474325541794]
  We introduce ClipGS, an innovative Gaussian splatting framework with the clipping plane supported, for interactive cinematic visualization of medical data.<n>We validate our method on five volumetric medical data, and reach an average 36.635 PSNR rendering quality with 156 FPS and 16.1 MB model size.
  arXiv  Detail & Related papers  (2025-07-09T08:24:28Z)
• ReshapeIT: Reliable Shape Interaction with Implicit Template for Anatomical Structure Reconstruction [59.971808117043366]
  ReShapeIT represents an anatomical structure with an implicit template field shared within the same category. It ensures the implicit template field generates valid templates by strengthening the constraint of the correspondence between the instance shape and the template shape. A template Interaction Module is introduced to reconstruct unseen shapes by interacting the valid template shapes with the instance-wise latent codes.
  arXiv  Detail & Related papers  (2023-12-11T07:09:32Z)
• From Sparse to Precise: A Practical Editing Approach for Intracardiac Echocardiography Segmentation [2.6910401398827117]
  We propose an interactive editing framework that allows users to edit segmentation output by drawing scribbles on a 2D frame. Our framework accommodates multiple edits to the segmentation output in a sequential manner without compromising previous edits.
  arXiv  Detail & Related papers  (2023-03-20T11:47:02Z)
• 3D Vessel Segmentation with Limited Guidance of 2D Structure-agnostic Vessel Annotations [3.6314292723682784]
  Supervised deep learning has demonstrated its superior capacity in automatic 3D vessel segmentation. The reliance on expensive 3D manual annotations and limited capacity for annotation reuse hinder the clinical applications of supervised models. This paper proposes a novel 3D shape-guided local discrimination model for 3D vascular segmentation under limited guidance from public 2D vessel annotations.
  arXiv  Detail & Related papers  (2023-02-07T07:26:00Z)
• A Self-Supervised Deep Framework for Reference Bony Shape Estimation in Orthognathic Surgical Planning [55.30223654196882]
  A virtual orthognathic surgical planning involves simulating surgical corrections of jaw deformities on 3D facial bony shape models. A reference facial bony shape model representing normal anatomies can provide an objective guidance to improve planning accuracy. We propose a self-supervised deep framework to automatically estimate reference facial bony shape models.
  arXiv  Detail & Related papers  (2021-09-11T05:24:40Z)
• Self Context and Shape Prior for Sensorless Freehand 3D Ultrasound Reconstruction [61.62191904755521]
  3D freehand US reconstruction is promising in addressing the problem by providing broad range and freeform scan. Existing deep learning based methods only focus on the basic cases of skill sequences. We propose a novel approach to sensorless freehand 3D US reconstruction considering the complex skill sequences.
  arXiv  Detail & Related papers  (2021-07-31T16:06:50Z)
• Interactive Radiotherapy Target Delineation with 3D-Fused Context Propagation [28.97228589610255]
  Convolutional neural networks (CNNs) have been predominated on automatic 3D medical segmentation tasks. We propose 3D-fused context propagation, which propagates any edited slice to the whole 3D volume.
  arXiv  Detail & Related papers  (2020-12-12T17:46:20Z)
• Weakly-supervised Learning For Catheter Segmentation in 3D Frustum Ultrasound [74.22397862400177]
  We propose a novel Frustum ultrasound based catheter segmentation method. The proposed method achieved the state-of-the-art performance with an efficiency of 0.25 second per volume.
  arXiv  Detail & Related papers  (2020-10-19T13:56:22Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.