MedSIGHT: Towards Grounded Visual Comprehension in Medical Large Vision-Language Models

• URL: http://arxiv.org/abs/2606.06760v1
• Date: Thu, 04 Jun 2026 22:54:59 GMT
• Title: MedSIGHT: Towards Grounded Visual Comprehension in Medical Large Vision-Language Models
• Abstract summary: We present MedSIGHT, a unified framework that equips Med-LVLMs with structured, pixel-level understanding for grounded visual comprehension.<n>MedSIGHT introduces a novel Region Perceiver module that produces region-centric tokens, encoding spatial information directly into representation space of the language model.<n>We further propose a medical region codebook into the LLM vocabulary, allowing the model to generate discrete region codes as symbolic representations of anatomical and pathological regions.
• Score: 42.44822236388223
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Medical large vision-language models (Med-LVLMs) have recently achieved remarkable progress in vision-language comprehension and medical image segmentation. However, existing models still struggle to unify these two capabilities, which is essential for achieving clinically reasoning that connects visual findings with semantic interpretation. We present MedSIGHT, a unified framework that equips Med-LVLMs with structured, pixel-level understanding for grounded visual comprehension. MedSIGHT introduces a novel Region Perceiver module that produces region-centric tokens, encoding spatial information directly into representation space of the language model. We further propose a medical region codebook into the LLM vocabulary, allowing the model to generate discrete region codes as symbolic representations of anatomical and pathological regions. These codes are decoded through the Region Perceiver to reconstruct segmentation mask, achieving end-to-end spatial grounding. Lastly, MedSIGHT combines Region Perceiver, Codebook and LLM using our proposed progressive training strategy to gradually aligns these modules stably. Trained on only 72K multimodal instruction pairs, MedSIGHT achieves state-of-the-art performance across diverse imaging modalities on both medical comprehension and segmentation tasks.

Related papers

• MedPlex: Deep Vision-Language Co-Adaptation for Clinically Grounded Medical Segmentation [8.760474194841908]
  MedPlex is an end-to-end VLM framework that makes text guidance a continuous, clinically grounded component of segmentation learning.<n>MedPlex achieves state-of-the-art performance across CT and MR benchmarks for multi-organ, cardiac substructure, and tumor segmentation.
  arXiv  Detail & Related papers  (2026-08-13T18:37:13Z)
• MedUP: Awakening Unified Understanding and Perception in Medical Vision-Language Models [27.291593841223943]
  We present MedUP, a Med-VLM that unifies perception and understanding within a shared token space.<n>At its core lies UniMedTok, a region tokenizer that encodes masks as discrete tokens in the LLM vocabulary.<n>We curate UniMed-Train, a 1.84M-instance corpus spanning text-guided segmentation, region-grounded understanding, medical VQA and CoT-based segmentation, and introduce UniMed-Bench for unified evaluation.
  arXiv  Detail & Related papers  (2026-08-11T08:22:13Z)
• Location-Aware Fine-Grained Representation Learning for Medical Vision Foundation Models [33.30452735577703]
  We present LoFi, a medical vision foundation model built on location-aware representation learning.<n>We construct MedG, a large-scale medical grounding dataset of 4.48M image-text-box triplets.<n>LoFi consistently outperforms general-purpose and medical vision foundation models as well as state-of-the-art LVLMs.
  arXiv  Detail & Related papers  (2026-08-02T03:59:07Z)
• MedP-CLIP: Medical CLIP with Region-Aware Prompt Integration [22.87996664536728]
  We propose MedP-CLIP, a region-aware medical vision-language model (VLM)<n>We pre-train the model on a meticulously constructed large-scale dataset.<n>Experiments demonstrate that MedP-CLIP significantly outperforms baseline methods in various medical tasks.
  arXiv  Detail & Related papers  (2026-04-13T08:53:36Z)
• Interpretable Bilingual Multimodal Large Language Model for Diverse Biomedical Tasks [13.016940516468674]
  We aim to enhance the capability of medical MLLMs in understanding anatomical regions within entire medical scans.<n>We propose a Region-Aware medical MLLM, MedRegA, which is the first bilingual generalist medical AI system.<n>Our model can not only accomplish powerful performance across various medical vision-language tasks in bilingual settings, but also recognize and detect structures in multimodal medical scans.
  arXiv  Detail & Related papers  (2024-10-24T02:55:41Z)
• LIMIS: Towards Language-based Interactive Medical Image Segmentation [58.553786162527686]
  LIMIS is the first purely language-based interactive medical image segmentation model. We adapt Grounded SAM to the medical domain and design a language-based model interaction strategy. We evaluate LIMIS on three publicly available medical datasets in terms of performance and usability.
  arXiv  Detail & Related papers  (2024-10-22T12:13:47Z)
• Contrastive Localized Language-Image Pre-Training [60.4967533101887]
  Contrastive Language-Image Pre-training (CLIP) has been a celebrated method for training vision encoders to generate image/text representations.<n>We propose Contrastive Localized Language-Image Pre-training (CLOC) by complementing CLIP with region-text contrastive loss and modules.<n>CLOC enables high-quality regional embeddings for image region recognition and retrieval tasks.
  arXiv  Detail & Related papers  (2024-10-03T17:56:09Z)
• ExGra-Med: Extended Context Graph Alignment for Medical Vision-Language Models [95.47808515575382]
  ExGra-Med is a novel framework for vision-language integration in medical AI.<n>It aligns images, instruction responses, and extended captions in the latent space, advancing semantic grounding and cross-modal coherence.<n>It matches LLaVA-Med's performance using just 10% of the pre-training data, achieving a 20.13% gain on VQA-RAD and approaching full-data performance.
  arXiv  Detail & Related papers  (2024-10-03T15:52:03Z)
• Unlocking the Power of Spatial and Temporal Information in Medical Multimodal Pre-training [99.2891802841936]
  We introduce the Med-ST framework for fine-grained spatial and temporal modeling. For spatial modeling, Med-ST employs the Mixture of View Expert (MoVE) architecture to integrate different visual features from both frontal and lateral views. For temporal modeling, we propose a novel cross-modal bidirectional cycle consistency objective by forward mapping classification (FMC) and reverse mapping regression (RMR)
  arXiv  Detail & Related papers  (2024-05-30T03:15:09Z)
• Hierarchical Text-to-Vision Self Supervised Alignment for Improved Histopathology Representation Learning [64.1316997189396]
  We present a novel language-tied self-supervised learning framework, Hierarchical Language-tied Self-Supervision (HLSS) for histopathology images. Our resulting model achieves state-of-the-art performance on two medical imaging benchmarks, OpenSRH and TCGA datasets.
  arXiv  Detail & Related papers  (2024-03-21T17:58:56Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.