GLINT: Sparsely Gated Vision-Language Alignment for Fine-Grained Radiology Representations

• URL: http://arxiv.org/abs/2606.03180v1
• Date: Tue, 02 Jun 2026 05:40:51 GMT
• Title: GLINT: Sparsely Gated Vision-Language Alignment for Fine-Grained Radiology Representations
• Abstract summary: We present GLINT (Gated Language-Image alignmeNT), a framework that explicitly models this sparse correspondence.<n> GLINT enables zero-shot classification, grounding, and segmentation from free-text queries.<n>It is the first to demonstrate zero-shot segmentation on 3D CT volumes without mask supervision.
• Score: 1.2679579476597642
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Vision-language models (VLMs) for radiology have emerged as a scalable paradigm by leveraging image-report pairs naturally produced in clinical workflows. However, this pairing reveals a mismatch in scale: each finding occupies only a small region of the image, yet supervision is provided only at the global image-report level. This poses a central challenge: prior approaches spread weight densely across all patches rather than concentrating on the sparse subset relevant to a given query. To address this, we present GLINT (Gated Language-Image alignmeNT), a framework that explicitly models this sparse correspondence. On the alignment side, we introduce Sparsely Gated Alignment, a novel architecture in which a sigmoid gate over a separate gate embedding space activates only the patches relevant to each textual query, enforcing explicit sparsity. On the representation side, we add Dense Feature Regularization, which anchors the trainable encoder's intermediate features to a frozen self-supervised learning (SSL) teacher, preserving the fine-grained patch features that the gate relies on. The same recipe applies to both 2D chest X-ray (CXR) and 3D chest computed tomography (CT), built with DINOv3 and V-JEPA 2.1, respectively. GLINT enables zero-shot classification, grounding, and segmentation from free-text queries, and to our knowledge is the first to demonstrate zero-shot segmentation on 3D CT volumes without mask supervision. Notably, the most pronounced gains arise on zero-shot grounding and segmentation, where sparse, query-specific localization is required, consistent with our design intent. In downstream evaluation, GLINT outperforms both SSL encoders and medical VLMs on classification, report generation, and segmentation.

Related papers

• GLASS: Graph-Language Alignment with Spherical Scoring for Transferable Graph-Level Anomaly Detection [37.05370485421858]
  GLASS builds a unified representation space by aligning a structure-aware graph encoder with an instruction-aware text embedding.<n>Our framework serializes local, global, and semantic graph properties into a compact Graph Descriptor Prompt (GraphDP)<n>For scoring, we formulate anomaly detection as density estimation on the aligned hypersphere.
  arXiv  Detail & Related papers  (2026-09-04T15:16:16Z)
• LESV: Language Embedded Sparse Voxel Fusion for Open-Vocabulary 3D Scene Understanding [9.377694035678948]
  We present a novel framework that leverages Sparse Voxel Rasterization (SVRaster) as a structured, disjoint geometry representation.<n>This enables a deterministic, confidence-aware feature registration process and suppresses the semantic bleeding artifact common in 3DGS.<n>Our approach achieves state-of-the-art performance on Open Vocabulary 3D Object Retrieval and Point Cloud Understanding benchmarks.
  arXiv  Detail & Related papers  (2026-04-01T20:48:06Z)
• PGOV3D: Open-Vocabulary 3D Semantic Segmentation with Partial-to-Global Curriculum [20.206273757144547]
  PGOV3D is a novel framework that introduces a Partial-to-Global curriculum for improving open-vocabulary 3D semantic segmentation.<n>We pre-train the model on partial scenes that provide dense semantic information but relatively simple geometry.<n>In the second stage, we fine-tune the model on complete scene-level point clouds, which are sparser and structurally more complex.
  arXiv  Detail & Related papers  (2025-06-30T08:13:07Z)
• PRISM-0: A Predicate-Rich Scene Graph Generation Framework for Zero-Shot Open-Vocabulary Tasks [51.31903029903904]
  In Scene Graphs Generation (SGG) one extracts structured representation from visual inputs in the form of objects nodes and predicates connecting them.<n> PRISM-0 is a framework for zero-shot open-vocabulary SGG that bootstraps foundation models in a bottom-up approach.<n> PRIMS-0 generates semantically meaningful graphs that improve downstream tasks such as Image Captioning and Sentence-to-Graph Retrieval.
  arXiv  Detail & Related papers  (2025-04-01T14:29:51Z)
• LESS: Label-Efficient and Single-Stage Referring 3D Segmentation [55.06002976797879]
  Referring 3D is a visual-language task that segments all points of the specified object from a 3D point cloud described by a sentence of query. We propose a novel Referring 3D pipeline, Label-Efficient and Single-Stage, dubbed LESS, which is only under the supervision of efficient binary mask. We achieve state-of-the-art performance on ScanRefer dataset by surpassing the previous methods about 3.7% mIoU using only binary labels.
  arXiv  Detail & Related papers  (2024-10-17T07:47:41Z)
• A Simple Framework for Open-Vocabulary Zero-Shot Segmentation [50.58626342189163]
  SimZSS is a framework for open-vocabulary Zero-Shot segmentation.<n>It exploits the discrete nature of text and linguistic knowledge to pinpoint local concepts within captions.<n>SimZSS achieves state-of-the-art results on 7 out of 8 benchmark datasets in less than 15 minutes.
  arXiv  Detail & Related papers  (2024-06-23T11:57:08Z)
• A Simple Framework for Open-Vocabulary Segmentation and Detection [85.21641508535679]
  We present OpenSeeD, a simple Open-vocabulary and Detection framework that jointly learns from different segmentation and detection datasets. We first introduce a pre-trained text encoder to encode all the visual concepts in two tasks and learn a common semantic space for them. After pre-training, our model exhibits competitive or stronger zero-shot transferability for both segmentation and detection.
  arXiv  Detail & Related papers  (2023-03-14T17:58:34Z)
• Keep Your Friends Close & Enemies Farther: Debiasing Contrastive Learning with Spatial Priors in 3D Radiology Images [11.251405818285331]
  We propose a 3D contrastive framework (Spade) that leverages extracted correspondences to select more effective positive & negative samples for representation learning. Compared to recent state-of-the-art approaches, Spade shows notable improvements on three downstream segmentation tasks.
  arXiv  Detail & Related papers  (2022-11-16T03:36:06Z)
• Two-Stream Graph Convolutional Network for Intra-oral Scanner Image Segmentation [133.02190910009384]
  We propose a two-stream graph convolutional network (i.e., TSGCN) to handle inter-view confusion between different raw attributes. Our TSGCN significantly outperforms state-of-the-art methods in 3D tooth (surface) segmentation.
  arXiv  Detail & Related papers  (2022-04-19T10:41:09Z)
• S3Net: 3D LiDAR Sparse Semantic Segmentation Network [1.330528227599978]
  S3Net is a novel convolutional neural network for LiDAR point cloud semantic segmentation. It adopts an encoder-decoder backbone that consists of Sparse Intra-channel Attention Module (SIntraAM) and Sparse Inter-channel Attention Module (SInterAM)
  arXiv  Detail & Related papers  (2021-03-15T22:15:24Z)
• PGL: Prior-Guided Local Self-supervised Learning for 3D Medical Image Segmentation [87.50205728818601]
  We propose a PriorGuided Local (PGL) self-supervised model that learns the region-wise local consistency in the latent feature space. Our PGL model learns the distinctive representations of local regions, and hence is able to retain structural information.
  arXiv  Detail & Related papers  (2020-11-25T11:03:11Z)
• High-Order Information Matters: Learning Relation and Topology for Occluded Person Re-Identification [84.43394420267794]
  We propose a novel framework by learning high-order relation and topology information for discriminative features and robust alignment. Our framework significantly outperforms state-of-the-art by6.5%mAP scores on Occluded-Duke dataset.
  arXiv  Detail & Related papers  (2020-03-18T12:18:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.