Local-Global Prompt Learning via Sparse Optimal Transport

• URL: http://arxiv.org/abs/2603.08347v1
• Date: Mon, 09 Mar 2026 13:09:55 GMT
• Title: Local-Global Prompt Learning via Sparse Optimal Transport
• Abstract summary: Few-shot adaptation of vision-language models (VLMs) like CLIP typically relies on learning textual prompts matched to global image embeddings.<n>Recent works extend this paradigm by incorporating local image-text alignment to capture fine-grained visual cues.<n>We propose SOT-GLP, which introduces a shared sparse patch support and balanced optimal transport allocation.
• Score: 3.7098626170498643
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Few-shot adaptation of vision-language models (VLMs) like CLIP typically relies on learning textual prompts matched to global image embeddings. Recent works extend this paradigm by incorporating local image-text alignment to capture fine-grained visual cues, yet these approaches often select local regions independently for each prompt, leading to redundant local feature usage and prompt overlap. We propose SOT-GLP, which introduces a shared sparse patch support and balanced optimal transport allocation to explicitly partition salient visual regions among class-specific local prompts while preserving global alignment. Our method learns shared global prompts and class-specific local prompts. The global branch maintains standard image-text matching for robust category-level alignment. The local branch constructs a class-conditioned sparse patch set using V-V attention and aligns it to multiple class-specific prompts via balanced entropic optimal transport, yielding a soft partition of patches that prevents prompt overlap and collapse. We evaluate our method on two complementary objectives: (i) few-shot classification accuracy on 11 standard benchmarks and (ii) out-of-distribution (OOD) detection. On the standard 11-dataset benchmark with 16-shot ViT-B/16, SOT-GLP achieves 85.1% average accuracy, outperforming prior prompt-learning methods. We identify a distinct accuracy-robustness trade-off in prompt learning: while learnable projections optimize in-distribution fit, they alter the foundational feature space. We demonstrate that a projection-free local alignment preserves the native geometry of the CLIP manifold, yielding state-of-the-art OOD detection performance (94.2% AUC) that surpasses fully adapted models. Implementation available at: https://github.com/Deniz2304988/SOT-GLP

Related papers

• LAGO: Language-Guided Adaptive Object-Region Focus for Zero-Shot Visual-Text Alignment [11.785675631199455]
  In fine-grained settings, the relevant evidence often lies in localized parts, attributes, or textures rather than in the full image.<n>Recent localized visual-text alignment methods address this by comparing class descriptions with multiple image regions.<n>We propose LAGO, a framework for efficient and robust zero-shot localized visual-text alignment.
  arXiv  Detail & Related papers  (2026-05-04T09:07:00Z)
• Region Matters: Efficient and Reliable Region-Aware Visual Place Recognition [46.97190804063947]
  We introduce FoL++, a method combining robust discriminative region modeling with adaptive re-ranking.<n>FoL++ achieves state-of-the-art performance with a lightweight memory footprint, improving inference speed by 40% over FoL.
  arXiv  Detail & Related papers  (2026-04-24T09:28:35Z)
• Efficiency Follows Global-Local Decoupling [62.05489838893081]
  We introduce ConvNeur, a two-branch architecture in which a lightweight neural memory branch aggregates global context on a compact set of tokens.<n>A learned gate lets global cues modulate local features without entangling their objectives.<n>On standard classification, detection, and segmentation benchmarks, ConvNeur matches or surpasses comparable alternatives at similar or lower compute.
  arXiv  Detail & Related papers  (2026-03-20T02:20:16Z)
• Towards Fine-Grained Vision-Language Alignment for Few-Shot Anomaly Detection [65.29550320117526]
  We propose a novel framework named FineGrainedAD to improve anomaly localization performance.<n> Experiments demonstrate that the proposed FineGrainedAD achieves superior overall performance in few-shot settings.
  arXiv  Detail & Related papers  (2025-10-30T13:09:00Z)
• Unbiased Region-Language Alignment for Open-Vocabulary Dense Prediction [80.67150791183126]
  Pre-trained vision-language models (VLMs) have demonstrated impressive zero-shot recognition capability, but still underperform in dense prediction tasks.<n>We propose DenseVLM, a framework designed to learn unbiased region-language alignment from powerful pre-trained VLM representations.<n>We show that DenseVLM can directly replace the original VLM in open-vocabulary object detection and image segmentation methods.
  arXiv  Detail & Related papers  (2024-12-09T06:34:23Z)
• GlocalCLIP: Object-agnostic Global-Local Prompt Learning for Zero-shot Anomaly Detection [5.530212768657544]
  We introduce glocal contrastive learning to improve the complementary learning of global and local prompts.<n>The generalization performance of GlocalCLIP in ZSAD was demonstrated on 15 real-world datasets.
  arXiv  Detail & Related papers  (2024-11-09T05:22:13Z)
• GalLoP: Learning Global and Local Prompts for Vision-Language Models [18.61218795834041]
  We introduce Global-Local Prompts (GalLoP), a new prompt learning method that learns multiple diverse prompts leveraging both global and local visual features.<n>GalLoP shows strong robustness performances in both domain generalization and OOD detection, even outperforming dedicated OOD detection methods.
  arXiv  Detail & Related papers  (2024-07-01T15:50:15Z)
• GLC++: Source-Free Universal Domain Adaptation through Global-Local Clustering and Contrastive Affinity Learning [84.54244771470012]
  Source-Free Universal Domain Adaptation (SF-UniDA) aims to accurately classify "known" data belonging to common categories.<n>GLC technique comprises an adaptive one-vs-all global clustering algorithm.<n>GLC++ enhances the novel category clustering accuracy of GLC by 4.1% in open-set scenarios.
  arXiv  Detail & Related papers  (2024-03-21T13:57:45Z)
• Align Your Prompts: Test-Time Prompting with Distribution Alignment for Zero-Shot Generalization [64.62570402941387]
  We use a single test sample to adapt multi-modal prompts at test time by minimizing the feature distribution shift to bridge the gap in the test domain. Our method improves zero-shot top- 1 accuracy beyond existing prompt-learning techniques, with a 3.08% improvement over the baseline MaPLe.
  arXiv  Detail & Related papers  (2023-11-02T17:59:32Z)
• Bootstrap Fine-Grained Vision-Language Alignment for Unified Zero-Shot Anomaly Localization [63.61093388441298]
  Contrastive Language-Image Pre-training models have shown promising performance on zero-shot visual recognition tasks. In this work, we propose AnoCLIP for zero-shot anomaly localization.
  arXiv  Detail & Related papers  (2023-08-30T10:35:36Z)
• VICRegL: Self-Supervised Learning of Local Visual Features [34.92750644059916]
  This paper explores the fundamental trade-off between learning local and global features. A new method called VICRegL is proposed that learns good global and local features simultaneously. We demonstrate strong performance on linear classification and segmentation transfer tasks.
  arXiv  Detail & Related papers  (2022-10-04T12:54:25Z)
• An Entropy-guided Reinforced Partial Convolutional Network for Zero-Shot Learning [77.72330187258498]
  We propose a novel Entropy-guided Reinforced Partial Convolutional Network (ERPCNet) ERPCNet extracts and aggregates localities based on semantic relevance and visual correlations without human-annotated regions. It not only discovers global-cooperative localities dynamically but also converges faster for policy gradient optimization.
  arXiv  Detail & Related papers  (2021-11-03T11:13:13Z)
• Generalized Focal Loss: Learning Qualified and Distributed Bounding Boxes for Dense Object Detection [85.53263670166304]
  One-stage detector basically formulates object detection as dense classification and localization. Recent trend for one-stage detectors is to introduce an individual prediction branch to estimate the quality of localization. This paper delves into the representations of the above three fundamental elements: quality estimation, classification and localization.
  arXiv  Detail & Related papers  (2020-06-08T07:24:33Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.