MedUP: Awakening Unified Understanding and Perception in Medical Vision-Language Models

• URL: http://arxiv.org/abs/2608.10635v1
• Date: Tue, 11 Aug 2026 08:22:13 GMT
• Title: MedUP: Awakening Unified Understanding and Perception in Medical Vision-Language Models
• Abstract summary: We present MedUP, a Med-VLM that unifies perception and understanding within a shared token space.<n>At its core lies UniMedTok, a region tokenizer that encodes masks as discrete tokens in the LLM vocabulary.<n>We curate UniMed-Train, a 1.84M-instance corpus spanning text-guided segmentation, region-grounded understanding, medical VQA and CoT-based segmentation, and introduce UniMed-Bench for unified evaluation.
• Score: 27.291593841223943
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Medical Vision-Language Models (Med-VLMs) excel at verbalizing visual content, yet precise visual perception, segmentation, and grounding remain challenging. Existing approaches either verbalize regions as coordinate strings or rely on external modules that decouple perception from understanding, creating representation gaps for region-language alignment. We present MedUP, a Med-VLM that natively unifies perception and understanding within a shared token space. At its core lies UniMedTok, a region tokenizer that encodes masks as discrete tokens in the LLM vocabulary, enabling the model to seamlessly interleave mask tokens with text. We curate UniMed-Train, a 1.84M-instance corpus spanning text-guided segmentation, region-grounded understanding, medical VQA and CoT-based segmentation, and introduce UniMed-Bench for unified evaluation. Extensive experiments show that MedUP outperforms native, agentic, and dual-decoder Med-VLMs across all tasks while remaining competitive with specialist segmentors, demonstrating the strong potential of unified understanding and perception modeling.

Related papers

• MedPlex: Deep Vision-Language Co-Adaptation for Clinically Grounded Medical Segmentation [8.760474194841908]
  MedPlex is an end-to-end VLM framework that makes text guidance a continuous, clinically grounded component of segmentation learning.<n>MedPlex achieves state-of-the-art performance across CT and MR benchmarks for multi-organ, cardiac substructure, and tumor segmentation.
  arXiv  Detail & Related papers  (2026-08-13T18:37:13Z)
• MedPixel: A Unified Pixel-Language Model for Medical Reasoning and Segmentation [28.78924935242642]
  We present MedPixel, a unified medical pixel-language model built around a shared language--mask interface.<n>MedPixel supports a broad spectrum of tasks spanning explicit grounding, implicit reasoning, spatial interaction, grounded explanation, and medical VQA.
  arXiv  Detail & Related papers  (2026-08-10T16:37:24Z)
• MedSIGHT: Towards Grounded Visual Comprehension in Medical Large Vision-Language Models [42.44822236388223]
  We present MedSIGHT, a unified framework that equips Med-LVLMs with structured, pixel-level understanding for grounded visual comprehension.<n>MedSIGHT introduces a novel Region Perceiver module that produces region-centric tokens, encoding spatial information directly into representation space of the language model.<n>We further propose a medical region codebook into the LLM vocabulary, allowing the model to generate discrete region codes as symbolic representations of anatomical and pathological regions.
  arXiv  Detail & Related papers  (2026-06-04T22:54:59Z)
• MedP-CLIP: Medical CLIP with Region-Aware Prompt Integration [22.87996664536728]
  We propose MedP-CLIP, a region-aware medical vision-language model (VLM)<n>We pre-train the model on a meticulously constructed large-scale dataset.<n>Experiments demonstrate that MedP-CLIP significantly outperforms baseline methods in various medical tasks.
  arXiv  Detail & Related papers  (2026-04-13T08:53:36Z)
• MedGround: Bridging the Evidence Gap in Medical Vision-Language Models with Verified Grounding Data [32.65971100171597]
  We introduce MedGround, an automated pipeline that transforms segmentation resources into high-quality medical referring grounding data.<n>We also present MedGround-35K, a novel multimodal medical dataset.
  arXiv  Detail & Related papers  (2026-01-11T10:34:18Z)
• RAU: Reference-based Anatomical Understanding with Vision Language Models [26.06602931463068]
  We introduce RAU, a framework for reference-based anatomical understanding with vision-language models (VLMs)<n>We first show that a VLM learns to identify anatomical regions through relative spatial reasoning between reference and target images.<n>Next, we demonstrate that the VLM-derived spatial cues can be seamlessly integrated with the fine-grained segmentation capability of SAM2.
  arXiv  Detail & Related papers  (2025-09-26T14:32:03Z)
• MedSeg-R: Reasoning Segmentation in Medical Images with Multimodal Large Language Models [48.24824129683951]
  We introduce medical image reasoning segmentation, a novel task that aims to generate segmentation masks based on complex and implicit medical instructions.<n>To address this, we propose MedSeg-R, an end-to-end framework that leverages the reasoning abilities of MLLMs to interpret clinical questions.<n>It is built on two core components: 1) a global context understanding module that interprets images and comprehends complex medical instructions to generate multi-modal intermediate tokens, and 2) a pixel-level grounding module that decodes these tokens to produce precise segmentation masks.
  arXiv  Detail & Related papers  (2025-06-12T08:13:38Z)
• LIMIS: Towards Language-based Interactive Medical Image Segmentation [58.553786162527686]
  LIMIS is the first purely language-based interactive medical image segmentation model. We adapt Grounded SAM to the medical domain and design a language-based model interaction strategy. We evaluate LIMIS on three publicly available medical datasets in terms of performance and usability.
  arXiv  Detail & Related papers  (2024-10-22T12:13:47Z)
• Unlocking the Power of Spatial and Temporal Information in Medical Multimodal Pre-training [99.2891802841936]
  We introduce the Med-ST framework for fine-grained spatial and temporal modeling. For spatial modeling, Med-ST employs the Mixture of View Expert (MoVE) architecture to integrate different visual features from both frontal and lateral views. For temporal modeling, we propose a novel cross-modal bidirectional cycle consistency objective by forward mapping classification (FMC) and reverse mapping regression (RMR)
  arXiv  Detail & Related papers  (2024-05-30T03:15:09Z)
• Medical Vision-Language Pre-Training for Brain Abnormalities [96.1408455065347]
  We show how to automatically collect medical image-text aligned data for pretraining from public resources such as PubMed. In particular, we present a pipeline that streamlines the pre-training process by initially collecting a large brain image-text dataset. We also investigate the unique challenge of mapping subfigures to subcaptions in the medical domain.
  arXiv  Detail & Related papers  (2024-04-27T05:03:42Z)
• Multi-Modal Masked Autoencoders for Medical Vision-and-Language Pre-Training [62.215025958347105]
  We propose a self-supervised learning paradigm with multi-modal masked autoencoders. We learn cross-modal domain knowledge by reconstructing missing pixels and tokens from randomly masked images and texts.
  arXiv  Detail & Related papers  (2022-09-15T07:26:43Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.