Ultrasound Vision-Language Alignment via Contrastive Learning

• URL: http://arxiv.org/abs/2605.02126v1
• Date: Mon, 04 May 2026 01:12:45 GMT
• Title: Ultrasound Vision-Language Alignment via Contrastive Learning
• Abstract summary: EchoCare-CLIP is a dual-encoder contrastive framework that aligns ultrasound images with clinical text in a shared embedding space.<n>We evaluate model configurations spanning two text encoder families (CLIP, BioClinicalBERT) and two caption strategies (template-based, LLM-generated) against OpenAI CLIP and BiomedCLIP baselines.<n>Our trained models consistently improve cross-modal alignment over baselines, with the best configuration achieving a paired alignment score of 0.682.
• Score: 6.671265771425717
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Ultrasound foundation models have achieved strong performance on structured prediction tasks but remain exclusively vision-based, limiting zero-shot and few-shot transfer to novel tasks where task-specific annotation is scarce. We address this gap with EchoCare-CLIP, a CLIP-style dual-encoder contrastive framework that aligns ultrasound images with clinical text in a shared embedding space. We curate a multi-organ corpus of over 16K image-text pairs spanning breast, liver, lung, and thyroid, with over 78% of captions derived from expert-annotated reports, and complement the remainder with a three-tier template-based and LLM-based caption generation pipeline. We evaluate model configurations spanning two text encoder families (CLIP, BioClinicalBERT) and two caption strategies (template-based, LLM-generated) against OpenAI CLIP and BiomedCLIP baselines. Our trained models consistently improve cross-modal alignment over baselines, with the best configuration achieving a paired alignment score of 0.682. However, stronger alignment does not guarantee better downstream performance: CLIP-based variants with partial fine-tuning achieve the strongest zero-shot classification on external held-out datasets (0.709 on BUSI; 0.626 on AULI), while full end-to-end fine-tuning degrades transfer due to overfitting. On linear probing and few-shot adaptation, model rankings are dataset-dependent, reflecting a trade-off between domain adaptation and representational generalizability. We further show that template-based captions match or outperform LLM-generated captions, suggesting lexical diversity is not a proxy for caption quality. Taken together, our results demonstrate that ultrasound vision-language alignment is achievable from public data alone, but robust clinical transfer requires careful balancing of domain adaptation, encoder capacity, and caption supervision quality.

Related papers

• RADIO1D: Elastic Representations for Condensed Vision Modeling [69.84453279129937]
  This paper challenges the assumption that vision-language models (VLMs) require fixed patch-based 2D vision features.<n>We introduce RADIO1D, which compresses images into a compact, variable-length 1D token sequence using multi-teacher knowledge distillation and an autoencoder design.<n>The resulting representations exhibit strong hierarchical summarization, enabling accurate scene understanding - even with a single token - and support improved composition-aware image retrieval.
  arXiv  Detail & Related papers  (2026-07-03T22:58:54Z)
• SonoCLIP: Mask-Guided Region-Aware Vision-Language Pretraining for Fetal Ultrasound Analysis [51.81845420301345]
  SonoCLIP is a vision-language foundation model for fetal ultrasound analysis.<n>It integrates segmentation masks as mask-channel visual prompts within the vision encoder.<n>It achieves superior zero-shot transfer performance under both global and mask-guided inference.
  arXiv  Detail & Related papers  (2026-06-28T20:04:49Z)
• Decoupling Semantics from Distortions: Multi-Scale Two-Stream Vision-Language Alignment for AI-Generated Image Quality Assessment [0.4790056963046066]
  MST-CLIPIQA is a multi-scale two-stream framework that achieves hierarchical vision-language alignment through explicit representational decoupling.<n>Our architecture leverages dual CLIP encoders with complementary patch granularities, capturing global semantic coherence.<n>Experiments establish new state-of-the-art results, achieving average improvements of 1.11 percent SRCC on quality and 2.35 percent SRCC on text-image correspondence prediction.
  arXiv  Detail & Related papers  (2026-06-15T14:40:30Z)
• Analyzing and Improving Fine-grained Preference Optimization in Medical LVLMs [53.697403481143404]
  Large Vision-Language Models (LVLMs) have achieved strong performance across medical imaging tasks.<n>But they remain prone to factual inconsistencies, poor visual grounding, and misalignment with clinically meaningful feedback.
  arXiv  Detail & Related papers  (2026-06-10T18:35:36Z)
• CapCLIP: A Vision-Language Representation Alignment Approach for Wireless Capsule Endoscopy Analysis [0.4551615447454768]
  Wireless capsule endoscopy (WCE) enables non-invasive visual assessment of the small bowel.<n>Existing learning-based approaches for WCE are predominantly vision-only, often confined to narrow pathology sets.<n>This study introduces CapCLIP, a domain-specific vision-language representation learning framework for WCE.
  arXiv  Detail & Related papers  (2026-05-08T21:14:56Z)
• Multi-View Synergistic Learning with Vision-Language Adaption for Low-Resource Biomedical Image Classification [59.24009931000134]
  Multi-View Synergistic Learning (MVSL) is a unified framework that addresses adaptation paradigms, representation granularity, and disease semantic relationships.<n>MVSL decouples the adaptation of visual and textual encoders to respect their distinct representational characteristics.<n>It further introduces multi-granularity contrastive learning to explicitly model both global image semantics and localized lesion-level evidence.<n>MVSL consistently outperforms state-of-the-art methods in few-shot and zero-shot classification settings.
  arXiv  Detail & Related papers  (2026-04-27T02:41:27Z)
• VIVID-Med: LLM-Supervised Structured Pretraining for Deployable Medical ViTs [24.283989257873085]
  We introduce VIVID-Med, a novel framework that leverages a frozen large language model (LLM) as a structured semantic teacher to pretrain medical vision transformers (ViTs)<n> VIVID-Med translates clinical findings into verifiable field-state pairs via a Unified MedicalNIST (UMS), utilizing answerability-aware masking to focus optimization.<n>It achieves a macro-AUC of 0.8588, outperforming BiomedCLIP by +6.65 points while using 500x less data.
  arXiv  Detail & Related papers  (2026-03-10T02:42:51Z)
• RVLF: A Reinforcing Vision-Language Framework for Gloss-Free Sign Language Translation [44.39679803351263]
  We build a large vision-language model (LVLM) specifically designed for sign language.<n>For a sufficient representation of sign language, RVLF introduces an effective semantic representation learning mechanism.<n>Then, to improve sentence-level semantic misalignment, we introduce a GRPO-based optimization strategy.
  arXiv  Detail & Related papers  (2025-12-08T08:11:53Z)
• Enhancing CLIP Robustness via Cross-Modality Alignment [54.01929554563447]
  We propose Cross-modality Alignment, an optimal transport-based framework for vision-language models.<n> COLA restores global image-text alignment and local structural consistency in the feature space.<n> COLA is training-free and compatible with existing fine-tuned models.
  arXiv  Detail & Related papers  (2025-10-28T03:47:44Z)
• Exploring the Capabilities of LLM Encoders for Image-Text Retrieval in Chest X-rays [8.019362739504087]
  Vision-language pretraining has advanced image-text alignment, yet progress in radiology remains constrained by the heterogeneity of clinical reports.<n>We ask whether large language model (LLM) encoders can provide robust clinical representations that transfer across diverse styles.<n>We introduce LLM2VEC4CXR, a domain-adapted encoder for chest X-ray reports, and LLM2CLIP4CXR, a dual-tower framework that couples this encoder with a vision backbone.
  arXiv  Detail & Related papers  (2025-09-17T09:44:59Z)
• SurgLaVi: Large-Scale Hierarchical Dataset for Surgical Vision-Language Representation Learning [41.95743276961411]
  We present SurgLaVi, the largest and most diverse surgical vision-language dataset to date, comprising nearly 240k clip-caption pairs from more than 200 procedures.<n>At the core of SurgLaVi lies a fully automated pipeline that generates fine-grained transcriptions of surgical videos and segments them into coherent procedural units.<n>To ensure accessibility, we release SurgLaVi-beta, an open-source derivative of 113k clip-caption pairs constructed entirely from public data.
  arXiv  Detail & Related papers  (2025-09-09T21:21:10Z)
• Modeling Caption Diversity in Contrastive Vision-Language Pretraining [48.7603274197994]
  We introduce Llip, Latent Language Image Pretraining, which models the diversity of captions that could match an image.<n>Llip's vision encoder outputs a set of visual features that are mixed into a final representation by conditioning on information derived from the text.<n>We show that Llip outperforms non-contextualized baselines like CLIP and SigLIP on a variety of tasks even with large-scale encoders.
  arXiv  Detail & Related papers  (2024-04-30T01:19:18Z)
• Symmetrical Linguistic Feature Distillation with CLIP for Scene Text Recognition [77.93678598476149]
  We establish a novel Symmetrical Linguistic Feature Distillation framework (named CLIP-OCR) By cascading the CLIP image encoder with the reversed CLIP text encoder, a symmetrical structure is built with an image-to-text feature flow. Extensive experiments demonstrate the effectiveness of CLIP-OCR with 93.8% average accuracy on six popular STR benchmarks.
  arXiv  Detail & Related papers  (2023-10-08T04:00:20Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.