G2D: From Global to Dense Radiography Representation Learning via Vision-Language Pre-training

• URL: http://arxiv.org/abs/2312.01522v4
• Date: Thu, 24 Oct 2024 20:01:44 GMT
• Title: G2D: From Global to Dense Radiography Representation Learning via Vision-Language Pre-training
• Abstract summary: We propose a new medical vision-language pre-training framework, named textbfGlobal to textbfDense level representation learning (G2D) G2D learns dense and semantically-grounded image representations via a pseudo segmentation task parallel with the global vision-language alignment. G2D achieves superior performance across 6 medical imaging tasks and 25 diseases, particularly in semantic segmentation.
• Score: 10.786853837756686
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Recently, medical vision-language pre-training (VLP) has reached substantial progress to learn global visual representation from medical images and their paired radiology reports. However, medical imaging tasks in real world usually require finer granularity in visual features. These tasks include visual localization tasks (e.g., semantic segmentation, object detection) and visual grounding task. Yet, current medical VLP methods face challenges in learning these fine-grained features, as they primarily focus on brute-force alignment between image patches and individual text tokens for local visual feature learning, which is suboptimal for downstream dense prediction tasks. In this work, we propose a new VLP framework, named \textbf{G}lobal to \textbf{D}ense level representation learning (G2D) that achieves significantly improved granularity and more accurate grounding for the learned features, compared to existing medical VLP approaches. In particular, G2D learns dense and semantically-grounded image representations via a pseudo segmentation task parallel with the global vision-language alignment. Notably, generating pseudo segmentation targets does not incur extra trainable parameters: they are obtained on the fly during VLP with a parameter-free processor. G2D achieves superior performance across 6 medical imaging tasks and 25 diseases, particularly in semantic segmentation, which necessitates fine-grained, semantically-grounded image features. In this task, G2D surpasses peer models even when fine-tuned with just 1\% of the training data, compared to the 100\% used by these models. The code can be found in https://github.com/cheliu-computation/G2D-NeurIPS24/tree/main.

Related papers

• RADA: Region-Aware Dual-encoder Auxiliary learning for Barely-supervised Medical Image Segmentation [33.17607546116405]
  Barely-supervised learning reduces annotation burden by using only a few labeled slices per volume.<n>We propose RADA, a novel Region-Aware Dual-encoder Auxiliary learning pipeline.<n>The framework combines image-level fine-grained visual features with text-level semantic guidance, providing region-aware semantic supervision.
  arXiv  Detail & Related papers  (2026-04-13T08:26:26Z)
• VELVET-Med: Vision and Efficient Language Pre-training for Volumetric Imaging Tasks in Medicine [11.993301266706139]
  We propose a vision-language pre-training framework, termed as textbfVELVET-Med, specifically designed for limited volumetric data such as 3D CT and associated radiology reports.<n>Our approach seeks to uncover rich spatial and semantic relationships embedded in volumetric medical images and corresponding clinical narratives.<n>The resulting encoders exhibit strong transferability, achieving state-of-the-art performance across a wide range of downstream tasks.
  arXiv  Detail & Related papers  (2025-08-16T17:08:43Z)
• TULIP: Towards Unified Language-Image Pretraining [60.99500935831526]
  We introduce T, an open-source, drop-in replacement for existing CLIP-like models. Our method leverages generative data augmentation, enhanced image-image and text-text contrastive learning, and image/text reconstruction regularization to learn fine-grained visual features. Our approach, scaling to over 1B parameters, outperforms existing state-of-the-art (SOTA) models across benchmarks.
  arXiv  Detail & Related papers  (2025-03-19T17:58:57Z)
• Barking Up The Syntactic Tree: Enhancing VLM Training with Syntactic Losses [31.85977999591524]
  Vision-Language Models implicitly learn to associate image regions with words from large-scale training data. Rich semantic and syntactic structures within the text modality have been overlooked as sources of supervision. Hierarchically STructured Learning (HIST) enhances spatial vision-language alignment without using additional human annotations.
  arXiv  Detail & Related papers  (2024-12-11T05:36:18Z)
• Talking to DINO: Bridging Self-Supervised Vision Backbones with Language for Open-Vocabulary Segmentation [56.001484215308075]
  We present Talk2DINO, a novel hybrid approach that combines the spatial accuracy of DINOv2 with the language understanding of CLIP.<n>We show that the powerful semantic and localization abilities of Talk2DINO can enhance the segmentation process.<n> Experimental results demonstrate that Talk2DINO achieves state-of-the-art performance across several unsupervised OVS benchmarks.
  arXiv  Detail & Related papers  (2024-11-28T19:00:03Z)
• Tumor segmentation on whole slide images: training or prompting? [0.0]
  We show the efficacy of visual prompting in the context of tumor segmentation for three distinct organs. Our findings reveal that, with appropriate prompt examples, visual prompting can achieve comparable or better performance without extensive fine-tuning.
  arXiv  Detail & Related papers  (2024-02-21T16:59:53Z)
• Exploring Part-Informed Visual-Language Learning for Person Re-Identification [40.725052076983516]
  We propose to enhance fine-grained visual features with part-informed language supervision for visual-based person re-identification tasks. Our $pi$-VL achieves substantial improvements over previous state-of-the-arts on four common-used ReID benchmarks.
  arXiv  Detail & Related papers  (2023-08-04T23:13:49Z)
• Lowis3D: Language-Driven Open-World Instance-Level 3D Scene Understanding [57.47315482494805]
  Open-world instance-level scene understanding aims to locate and recognize unseen object categories that are not present in the annotated dataset. This task is challenging because the model needs to both localize novel 3D objects and infer their semantic categories. We propose to harness pre-trained vision-language (VL) foundation models that encode extensive knowledge from image-text pairs to generate captions for 3D scenes.
  arXiv  Detail & Related papers  (2023-08-01T07:50:14Z)
• Vision Transformers: From Semantic Segmentation to Dense Prediction [139.15562023284187]
  We explore the global context learning potentials of vision transformers (ViTs) for dense visual prediction. Our motivation is that through learning global context at full receptive field layer by layer, ViTs may capture stronger long-range dependency information. We formulate a family of Hierarchical Local-Global (HLG) Transformers, characterized by local attention within windows and global-attention across windows in a pyramidal architecture.
  arXiv  Detail & Related papers  (2022-07-19T15:49:35Z)
• BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation [86.4572981982407]
  We propose BLIP, a new vision-language framework which transfers flexibly to both vision-language understanding and generation tasks. BLIP effectively utilizes the noisy web data by bootstrapping the captions, where a captioner generates synthetic captions and a filter removes the noisy ones. BLIP also demonstrates strong generalization ability when directly transferred to video-language tasks in a zero-shot manner.
  arXiv  Detail & Related papers  (2022-01-28T12:49:48Z)
• Consensus Graph Representation Learning for Better Grounded Image Captioning [48.208119537050166]
  We propose the Consensus Rraph Representation Learning framework (CGRL) for grounded image captioning. We validate the effectiveness of our model, with a significant decline in object hallucination (-9% CHAIRi) on the Flickr30k Entities dataset.
  arXiv  Detail & Related papers  (2021-12-02T04:17:01Z)
• Attention-Guided Supervised Contrastive Learning for Semantic Segmentation [16.729068267453897]
  In a per-pixel prediction task, more than one label can exist in a single image for segmentation. We propose an attention-guided supervised contrastive learning approach to highlight a single semantic object every time as the target.
  arXiv  Detail & Related papers  (2021-06-03T05:01:11Z)
• Semantic Segmentation with Generative Models: Semi-Supervised Learning and Strong Out-of-Domain Generalization [112.68171734288237]
  We propose a novel framework for discriminative pixel-level tasks using a generative model of both images and labels. We learn a generative adversarial network that captures the joint image-label distribution and is trained efficiently using a large set of unlabeled images. We demonstrate strong in-domain performance compared to several baselines, and are the first to showcase extreme out-of-domain generalization.
  arXiv  Detail & Related papers  (2021-04-12T21:41:25Z)
• Group-Wise Semantic Mining for Weakly Supervised Semantic Segmentation [49.90178055521207]
  This work addresses weakly supervised semantic segmentation (WSSS), with the goal of bridging the gap between image-level annotations and pixel-level segmentation. We formulate WSSS as a novel group-wise learning task that explicitly models semantic dependencies in a group of images to estimate more reliable pseudo ground-truths. In particular, we devise a graph neural network (GNN) for group-wise semantic mining, wherein input images are represented as graph nodes.
  arXiv  Detail & Related papers  (2020-12-09T12:40:13Z)
• PGL: Prior-Guided Local Self-supervised Learning for 3D Medical Image Segmentation [87.50205728818601]
  We propose a PriorGuided Local (PGL) self-supervised model that learns the region-wise local consistency in the latent feature space. Our PGL model learns the distinctive representations of local regions, and hence is able to retain structural information.
  arXiv  Detail & Related papers  (2020-11-25T11:03:11Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.