Looking Beyond the Window: Global-Local Aligned CLIP for Training-free Open-Vocabulary Semantic Segmentation

• URL: http://arxiv.org/abs/2603.23030v1
• Date: Tue, 24 Mar 2026 10:10:12 GMT
• Title: Looking Beyond the Window: Global-Local Aligned CLIP for Training-free Open-Vocabulary Semantic Segmentation
• Abstract summary: A sliding-window inference strategy is commonly adopted in training-free open-vocabulary semantic segmentation methods.<n>This approach introduces a new challenge: each window is processed independently, leading to semantic discrepancy across windows.<n>We propose Global-Local Aligned CLIP(GLA-CLIP), a framework that facilitates comprehensive information exchange across windows.
• Score: 44.32289400317069
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: A sliding-window inference strategy is commonly adopted in recent training-free open-vocabulary semantic segmentation methods to overcome limitation of the CLIP in processing high-resolution images. However, this approach introduces a new challenge: each window is processed independently, leading to semantic discrepancy across windows. To address this issue, we propose Global-Local Aligned CLIP~(GLA-CLIP), a framework that facilitates comprehensive information exchange across windows. Rather than limiting attention to tokens within individual windows, GLA-CLIP extends key-value tokens to incorporate contextual cues from all windows. Nevertheless, we observe a window bias: outer-window tokens are less likely to be attended, since query features are produced through interactions within the inner window patches, thereby lacking semantic grounding beyond their local context. To mitigate this, we introduce a proxy anchor, constructed by aggregating tokens highly similar to the given query from all windows, which provides a unified semantic reference for measuring similarity across both inner- and outer-window patches. Furthermore, we propose a dynamic normalization scheme that adjusts attention strength according to object scale by dynamically scaling and thresholding the attention map to cope with small-object scenarios. Moreover, GLA-CLIP can be equipped on existing methods and broad their receptive field. Extensive experiments validate the effectiveness of GLA-CLIP in enhancing training-free open-vocabulary semantic segmentation performance. Code is available at https://github.com/2btlFe/GLA-CLIP.

Related papers

• Segment Any Events with Language [68.05185562243356]
  We introduce SEAL, the first Semantic-aware Any Events framework that addresses Open-Vocabulary Event Instance (OV-EIS)<n>Given the visual prompt, our model presents a unified framework to support both segmentation event and open-vocabulary mask classification at multiple levels of granularity, including instance-level and part-level.<n>Our SEAL largely outperforms proposed baselines in terms of performance and inference speed with a parameter-efficient architecture.
  arXiv  Detail & Related papers  (2026-01-30T16:42:56Z)
• SuperCLIP: CLIP with Simple Classification Supervision [88.86549733903314]
  Contrastive Language-Image Pretraining achieves strong generalization in vision-language tasks by aligning images and texts in a shared embedding space.<n>Recent findings show that CLIP-like models still underutilize fine-grained semantic signals in text.<n>We propose SuperCLIP, a framework that augments contrastive learning with classification-based supervision.
  arXiv  Detail & Related papers  (2025-12-16T15:11:53Z)
• Generalized Decoupled Learning for Enhancing Open-Vocabulary Dense Perception [71.26728044621458]
  DeCLIP is a novel framework that enhances CLIP by decoupling the self-attention module to obtain content'' and context'' features respectively.<n>It consistently achieves state-of-the-art performance across a broad spectrum of tasks, including 2D detection and segmentation, 3D instance segmentation, video instance segmentation, and 6D object pose estimation.
  arXiv  Detail & Related papers  (2025-08-15T06:43:51Z)
• DeCLIP: Decoupled Learning for Open-Vocabulary Dense Perception [21.87721909270275]
  DeCLIP is a novel framework that enhances CLIP with content'' and context'' features respectively.<n>It significantly outperforms existing methods across multiple open-vocabulary dense prediction tasks.
  arXiv  Detail & Related papers  (2025-05-07T13:46:34Z)
• Rethinking the Global Knowledge of CLIP in Training-Free Open-Vocabulary Semantic Segmentation [5.3499687969383345]
  We propose GCLIP to answer how to extract and utilize beneficial global knowledge of CLIP for TF-OVSS.<n>We aim to equip the last-block attention with image-level properties while not introducing homogeneous attention patterns across patches.
  arXiv  Detail & Related papers  (2025-02-05T03:37:50Z)
• ResCLIP: Residual Attention for Training-free Dense Vision-language Inference [27.551367463011008]
  Cross-correlation of self-attention in CLIP's non-final layers also exhibits localization properties. We propose the Residual Cross-correlation Self-attention (RCS) module, which leverages the cross-correlation self-attention from intermediate layers to remold the attention in the final block. The RCS module effectively reorganizes spatial information, unleashing the localization potential within CLIP for dense vision-language inference.
  arXiv  Detail & Related papers  (2024-11-24T14:14:14Z)
• OpenDAS: Open-Vocabulary Domain Adaptation for 2D and 3D Segmentation [54.98688607911399]
  We propose the task of open-vocabulary domain adaptation to infuse domain-specific knowledge into Vision-Language Models (VLMs) Existing VLM adaptation methods improve performance on base (training) queries, but fail to preserve the open-set capabilities of VLMs on novel queries. Our approach is the only parameter-efficient method that consistently surpasses the original VLM on novel classes.
  arXiv  Detail & Related papers  (2024-05-30T15:16:06Z)
• UMG-CLIP: A Unified Multi-Granularity Vision Generalist for Open-World Understanding [90.74967596080982]
  This paper extends Contrastive Language-Image Pre-training (CLIP) with multi-granularity alignment. We develop a Unified Multi-Granularity learning framework, termed UMG-CLIP, which simultaneously empowers the model with versatile perception abilities. With parameter efficient tuning, UMG-CLIP surpasses current widely used CLIP variants and achieves state-of-the-art performance on diverse image understanding benchmarks.
  arXiv  Detail & Related papers  (2024-01-12T06:35:09Z)
• CLIP-DINOiser: Teaching CLIP a few DINO tricks for open-vocabulary semantic segmentation [31.264574799748903]
  We propose an open-vocabulary semantic segmentation method, which does not require any annotations. We show that the used self-supervised feature properties can directly be learnt from CLIP features. Our method CLIP-DINOiser needs only a single forward pass of CLIP and two light convolutional layers at inference.
  arXiv  Detail & Related papers  (2023-12-19T17:40:27Z)
• Global Knowledge Calibration for Fast Open-Vocabulary Segmentation [124.74256749281625]
  We introduce a text diversification strategy that generates a set of synonyms for each training category. We also employ a text-guided knowledge distillation method to preserve the generalizable knowledge of CLIP. Our proposed model achieves robust generalization performance across various datasets.
  arXiv  Detail & Related papers  (2023-03-16T09:51:41Z)
• Parallel Context Windows for Large Language Models [52.965170346907904]
  We present Parallel Context Windows (PCW), a method that alleviates the context window restriction for any off-the-shelf LLM without further training. Our main results test the PCW approach on in-context learning with models that range in size between 750 million and 178 billion parameters. We show additional benefits in other settings where long context windows may be beneficial: multi-hop questions and retrieval-augmented question answering with multiple retrieved documents.
  arXiv  Detail & Related papers  (2022-12-21T11:38:51Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.