Medical Referring Image Segmentation via Next-Token Mask Prediction

• URL: http://arxiv.org/abs/2511.05044v1
• Date: Fri, 07 Nov 2025 07:29:19 GMT
• Title: Medical Referring Image Segmentation via Next-Token Mask Prediction
• Authors: Xinyu Chen, Yiran Wang, Gaoyang Pang, Jiafu Hao, Chentao Yue, Luping Zhou, Yonghui Li,
• Abstract summary: Medical Referring Image (MRIS) involves segmenting target regions in medical images based on natural language descriptions.<n>We propose NTP-MRISeg, a novel framework that reformulates MRIS as an autoregressive next-token prediction task over a unified multimodal sequence of tokenized image, text, and mask representations.
• Score: 40.827152909794336
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Medical Referring Image Segmentation (MRIS) involves segmenting target regions in medical images based on natural language descriptions. While achieving promising results, recent approaches usually involve complex design of multimodal fusion or multi-stage decoders. In this work, we propose NTP-MRISeg, a novel framework that reformulates MRIS as an autoregressive next-token prediction task over a unified multimodal sequence of tokenized image, text, and mask representations. This formulation streamlines model design by eliminating the need for modality-specific fusion and external segmentation models, supports a unified architecture for end-to-end training. It also enables the use of pretrained tokenizers from emerging large-scale multimodal models, enhancing generalization and adaptability. More importantly, to address challenges under this formulation-such as exposure bias, long-tail token distributions, and fine-grained lesion edges-we propose three novel strategies: (1) a Next-k Token Prediction (NkTP) scheme to reduce cumulative prediction errors, (2) Token-level Contrastive Learning (TCL) to enhance boundary sensitivity and mitigate long-tail distribution effects, and (3) a memory-based Hard Error Token (HET) optimization strategy that emphasizes difficult tokens during training. Extensive experiments on the QaTa-COV19 and MosMedData+ datasets demonstrate that NTP-MRISeg achieves new state-of-the-art performance, offering a streamlined and effective alternative to traditional MRIS pipelines.

Related papers

• MultiModal Fine-tuning with Synthetic Captions [9.572235167281686]
  We propose a novel approach that transforms unimodal datasets into multimodal ones using Multimodal Large Language Models (MLLMs)<n>Our method employs carefully designed prompts incorporating class labels and domain context to produce high-quality captions for classification tasks.<n>Our work establishes a new paradigm for dataset enhancement that effectively bridges the gap between multimodal pre-training and fine-tuning.
  arXiv  Detail & Related papers  (2026-01-29T09:03:45Z)
• IBISAgent: Reinforcing Pixel-Level Visual Reasoning in MLLMs for Universal Biomedical Object Referring and Segmentation [44.89730606641666]
  IBISAgent reformulates segmentation as a vision-centric, multi-step decision-making process.<n>IBISAgent consistently outperforms both closed-source and open-source SOTA methods.<n>All datasets, code, and trained models will be released publicly.
  arXiv  Detail & Related papers  (2026-01-06T14:37:50Z)
• MedSeqFT: Sequential Fine-tuning Foundation Models for 3D Medical Image Segmentation [55.37355146924576]
  MedSeqFT is a sequential fine-tuning framework for medical image analysis.<n>It adapts pre-trained models to new tasks while refining their representational capacity.<n>It consistently outperforms state-of-the-art fine-tuning strategies.
  arXiv  Detail & Related papers  (2025-09-07T15:22:53Z)
• Autoregressive Medical Image Segmentation via Next-Scale Mask Prediction [16.026171689438637]
  We propose the AutoRegressive framework via next-scale mask prediction, termed AR-Seg.<n>AR-Seg progressively predicts the next-scale mask by explicitly modeling dependencies across all previous scales.<n>We show that AR-Seg outperforms state-of-the-art methods while explicitly visualizing the intermediate coarse-to-fine segmentation process.
  arXiv  Detail & Related papers  (2025-02-28T07:05:58Z)
• Multi-task Visual Grounding with Coarse-to-Fine Consistency Constraints [15.541287957548771]
  We propose a Coarse-to-fine Consistency Constraints Visual Grounding architecture.<n>It integrates implicit and explicit modeling approaches within a two-stage framework.<n>It significantly outperforms state-of-the-art REC and RIS methods by a substantial margin.
  arXiv  Detail & Related papers  (2025-01-12T04:30:13Z)
• MFCLIP: Multi-modal Fine-grained CLIP for Generalizable Diffusion Face Forgery Detection [64.29452783056253]
  The rapid development of photo-realistic face generation methods has raised significant concerns in society and academia.<n>Although existing approaches mainly capture face forgery patterns using image modality, other modalities like fine-grained noises and texts are not fully explored.<n>We propose a novel multi-modal fine-grained CLIP (MFCLIP) model, which mines comprehensive and fine-grained forgery traces across image-noise modalities.
  arXiv  Detail & Related papers  (2024-09-15T13:08:59Z)
• TokenUnify: Scaling Up Autoregressive Pretraining for Neuron Segmentation [65.65530016765615]
  We propose a hierarchical predictive coding framework that captures multi-scale dependencies through three complementary learning objectives.<n> TokenUnify integrates random token prediction, next-token prediction, and next-all token prediction to create a comprehensive representational space.<n>We also introduce a large-scale EM dataset with 1.2 billion annotated voxels, offering ideal long-sequence visual data with spatial continuity.
  arXiv  Detail & Related papers  (2024-05-27T05:45:51Z)
• Unleashing Network Potentials for Semantic Scene Completion [50.95486458217653]
  This paper proposes a novel SSC framework - Adrial Modality Modulation Network (AMMNet) AMMNet introduces two core modules: a cross-modal modulation enabling the interdependence of gradient flows between modalities, and a customized adversarial training scheme leveraging dynamic gradient competition. Extensive experimental results demonstrate that AMMNet outperforms state-of-the-art SSC methods by a large margin.
  arXiv  Detail & Related papers  (2024-03-12T11:48:49Z)
• IMKGA-SM: Interpretable Multimodal Knowledge Graph Answer Prediction via Sequence Modeling [3.867363075280544]
  Multimodal knowledge graph link prediction aims to improve the accuracy and efficiency of link prediction tasks for multimodal data. New model is developed, namely Interpretable Multimodal Knowledge Graph Answer Prediction via Sequence Modeling (IMKGA-SM) Model achieves much better performance than SOTA baselines on multimodal link prediction datasets of different sizes.
  arXiv  Detail & Related papers  (2023-01-06T10:08:11Z)
• Enforcing Mutual Consistency of Hard Regions for Semi-supervised Medical Image Segmentation [68.9233942579956]
  We propose a novel mutual consistency network (MC-Net+) to exploit the unlabeled hard regions for semi-supervised medical image segmentation. The MC-Net+ model is motivated by the observation that deep models trained with limited annotations are prone to output highly uncertain and easily mis-classified predictions. We compare the segmentation results of the MC-Net+ with five state-of-the-art semi-supervised approaches on three public medical datasets.
  arXiv  Detail & Related papers  (2021-09-21T04:47:42Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.