Gram-Anchored Prompt Learning for Vision-Language Models via Second-Order Statistics

• URL: http://arxiv.org/abs/2604.03980v1
• Date: Sun, 05 Apr 2026 06:02:07 GMT
• Title: Gram-Anchored Prompt Learning for Vision-Language Models via Second-Order Statistics
• Abstract summary: We argue that relying solely on first-order information is insufficient for robust adaptation.<n>We propose textbfGram-Anchored Prompt Learning (GAPL) for Vision-Language Models via Second-Order Statistics.
• Score: 6.327904555091201
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Parameter-efficient prompt learning has become the de facto standard for adapting Vision-Language Models (VLMs) to downstream tasks. Existing approaches predominantly focus on aligning text prompts with first-order visual features (i.e., spatial feature maps). While effective for fine-grained semantic discrimination, we argue that relying solely on first-order information is insufficient for robust adaptation, as these spatially entangled features are highly susceptible to domain shifts and local noise. In this work, we propose \textbf{Gram-Anchored Prompt Learning (GAPL)} for Vision-Language Models via Second-Order Statistics, a framework that synergizes local semantic alignment with global structural consistency. Methodologically, we introduce an additional second-order statistical stream via \textbf{Gram matrices} that augments the standard first-order spatial interaction. By anchoring prompts to these second-order priors, our approach enables language representations to dynamically adapt to statistical distribution shifts across diverse domains. Extensive experiments indicate the effectiveness of the second-order features, and show compelling performances of GAPL on various benchmarks.

Related papers

• Semantic-Driven Scale and Spatial Selection for Efficient Cross-Modal Alignment in Referring Remote Sensing Image Segmentation [14.492779733437082]
  Referring Remote Sensing Image seeks to localize and segment the target object or region specified by a natural language expression in a remote sensing image.<n>Existing RRSIS models have benefited from large-scale foundation models, but they predominantly rely on full fine-tuning.<n>We propose a novel framework, Semantic-driven Scale and Spatial Selection for Efficient Cross-modal Alignment.
  arXiv  Detail & Related papers  (2026-06-29T12:54:10Z)
• Dynamic Parsing and Updating Natural Language Specification using VLMs for Robust Vision-Language Tracking [61.91472604294714]
  We propose a novel language dependency parsing mechanism to distill tracking principal components.<n>We exploit the powerful cross-modal understanding capability of the pre-trained vision-language model Qwen-VL.<n>Our method achieves consistent and superior tracking performance on multiple large-scale vision-language tracking benchmarks.
  arXiv  Detail & Related papers  (2026-06-28T12:12:18Z)
• ProVG: Progressive Visual Grounding via Language Decoupling for Remote Sensing Imagery [20.725910735285762]
  Remote sensing visual grounding aims to localize objects in remote sensing imagery according to natural language expressions.<n>We propose textbfProVG, a novel RSVG framework that improves localization accuracy by decoupling language expressions into global context, spatial relations, and object attributes.<n>ProVG consistently outperforms existing methods, achieving new state-of-the-art performance.
  arXiv  Detail & Related papers  (2026-04-02T10:59:19Z)
• dinov3.seg: Open-Vocabulary Semantic Segmentation with DINOv3 [36.6036728217708]
  Open-Vocabulary Semantics (OVSS) assigns pixel-level labels from an open set of text-defined categories, demanding reliable generalization to unseen classes at inference.<n>We introduce dinov3.seg, extending dinov3.txt into a dedicated framework for OVSS.<n>Our contributions are four-fold. First, we design a task-specific architecture tailored to this backbone, systematically adapting established design principles from prior open-vocabulary segmentation work.<n>Second, we jointly leverage text embeddings aligned with both the global [] token and local patch-level visual features from ViT-based encoder
  arXiv  Detail & Related papers  (2026-03-19T23:57:28Z)
• Constrained Prompt Enhancement for Improving Zero-Shot Generalization of Vision-Language Models [57.357091028792325]
  Vision-language models (VLMs) pre-trained on web-scale data exhibit promising zero-shot generalization but often suffer from semantic misalignment.<n>We propose a novel constrained prompt enhancement (CPE) method to improve visual-textual alignment.<n>Our approach consists of two key components: Topology-Guided Synonymous Semantic Generation (TGSSG) and Category-Agnostic Discriminative Region Selection (CADRS)
  arXiv  Detail & Related papers  (2025-08-24T15:45:22Z)
• Weakly-Supervised Image Forgery Localization via Vision-Language Collaborative Reasoning Framework [16.961220047066792]
  ViLaCo is a vision-language collaborative reasoning framework that introduces auxiliary semantic supervision distilled from pre-trained vision-language models.<n>ViLaCo substantially outperforms existing WSIFL methods, achieving state-of-the-art performance in both detection and localization accuracy.
  arXiv  Detail & Related papers  (2025-08-02T12:14:29Z)
• HiLa: Hierarchical Vision-Language Collaboration for Cancer Survival Prediction [55.00788339683146]
  We propose a novel Hierarchical vision-Language collaboration framework for improved survival prediction.<n> Specifically, HiLa employs pretrained feature extractors to generate hierarchical visual features from WSIs at both patch and region levels.<n>This ap-proach enables the comprehensive learning of discriminative visual features cor-responding to different survival-related attributes from prompts.
  arXiv  Detail & Related papers  (2025-07-07T02:06:25Z)
• Interpretable Zero-Shot Learning with Locally-Aligned Vision-Language Model [56.573203512455706]
  Large-scale vision-language models (VLMs) have achieved remarkable success in zero-shot learning (ZSL) by leveraging large-scale visual-text pair datasets.<n>One approach to address this issue is to develop interpretable models by integrating language.<n>We propose LaZSL, a locally-aligned vision-language model for interpretable ZSL.
  arXiv  Detail & Related papers  (2025-06-30T13:14:46Z)
• SemiDAViL: Semi-supervised Domain Adaptation with Vision-Language Guidance for Semantic Segmentation [9.311853182451289]
  We propose a language-guided Semi-supervised Domain Adaptation (SSDA) setting for semantic segmentation.<n>We harness the semantic generalization capabilities inherent in vision-language models (VLMs) to establish a synergistic framework.<n>Our approach demonstrates substantial performance improvements over contemporary state-of-the-art (SoTA) methodologies.
  arXiv  Detail & Related papers  (2025-04-08T19:14:34Z)
• Unseen from Seen: Rewriting Observation-Instruction Using Foundation Models for Augmenting Vision-Language Navigation [63.54377402784965]
  We propose a Rewriting-driven AugMentation (RAM) paradigm for Vision-Language Navigation (VLN)<n>Benefiting from our rewriting mechanism, new observation-instruction pairs can be obtained in both simulator-free and labor-saving manners.<n> Experiments on both the discrete environments (R2R, REVERIE, and R4R dataset) and continuous environments (R2R-CE dataset) show the superior performance and impressive generalization ability of our method.
  arXiv  Detail & Related papers  (2025-03-23T13:18:17Z)
• Language-aware Domain Generalization Network for Cross-Scene Hyperspectral Image Classification [15.842081807249416]
  It is necessary to explore the effectiveness of linguistic mode in assisting hyperspectral image classification. Large-scale pre-training image-text foundation models have demonstrated great performance in a variety of downstream applications. A Language-aware Domain Generalization Network (LDGnet) is proposed to learn cross-domain invariant representation.
  arXiv  Detail & Related papers  (2022-09-06T10:06:10Z)
• HSVA: Hierarchical Semantic-Visual Adaptation for Zero-Shot Learning [74.76431541169342]
  Zero-shot learning (ZSL) tackles the unseen class recognition problem, transferring semantic knowledge from seen classes to unseen ones. We propose a novel hierarchical semantic-visual adaptation (HSVA) framework to align semantic and visual domains. Experiments on four benchmark datasets demonstrate HSVA achieves superior performance on both conventional and generalized ZSL.
  arXiv  Detail & Related papers  (2021-09-30T14:27:50Z)
• Cross-lingual Spoken Language Understanding with Regularized Representation Alignment [71.53159402053392]
  We propose a regularization approach to align word-level and sentence-level representations across languages without any external resource. Experiments on the cross-lingual spoken language understanding task show that our model outperforms current state-of-the-art methods in both few-shot and zero-shot scenarios.
  arXiv  Detail & Related papers  (2020-09-30T08:56:53Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.