Beyond Templates: Revisiting Zero-Shot Remote Sensing through Meta-Prompting

• URL: http://arxiv.org/abs/2606.20702v1
• Date: Mon, 15 Jun 2026 16:04:43 GMT
• Title: Beyond Templates: Revisiting Zero-Shot Remote Sensing through Meta-Prompting
• Abstract summary: Vision-language models (VLMs) have sparked growing interest in zero-shot Earth Observation (EO) downstream tasks.<n>We show that zero-shot performance remains highly sensitive to textual design choices.<n>We explore why semantically rich class descriptions do not translate into consistent gains over simple domain-adapted CLIP-style descriptions.
• Score: 2.816909824193965
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Vision-language models (VLMs) have sparked growing interest in zero-shot Earth Observation (EO) downstream tasks, with further gains enabled by remote-sensing-adapted models. We examine this setting across 17 VLM variants and 12 remote sensing (RS) datasets under Meta-Prompting for Visual Recognition (MPVR), and show that zero-shot performance remains highly sensitive to textual design choices, from the meta-prompts used to guide the LLM in generating class descriptions to the descriptions themselves. We explore why semantically rich LLM-generated class descriptions do not translate into consistent gains over simple domain-adapted CLIP-style descriptions. While LLM descriptions are more semantically expressive, they can also introduce noise in the text embedding space, reducing robustness in downstream tasks. We support this observation through a text log-likelihood analysis in the whitened CLIP feature space, comparing LLM-generated and template-based descriptions. Building on this finding, we study query embedding calibration and show that lightweight calibration of the query space consistently yields strong improvements in zero-shot classification and retrieval. Overall, our results provide practical insight into the trade-off between semantic richness and robustness, and identify embedding calibration as a simple and effective tool for improving zero-shot remote sensing performance.

Related papers

• Task-Adaptive Embedding Refinement via Test-time LLM Guidance [3.6468227618496627]
  We explore the effectiveness of an LLM-guided query refinement paradigm for extending the usability of embedding models to challenging zero-shot search and classification tasks.<n>Our approach refines the embedding representation of a user query using feedback from a generative LLM on a small set of documents.<n>We conduct extensive experiments with state-of-the-art text embedding models across a diverse set of challenging search and classification benchmarks.
  arXiv  Detail & Related papers  (2026-05-12T17:58:27Z)
• Interpretable Zero-shot Referring Expression Comprehension with Query-driven Scene Graphs [18.414159451507153]
  Zero-shot referring expression comprehension (REC) aims to locate target objects in images given natural language queries.<n>Existing Vision-Language Models (VLMs) commonly address zero-shot REC by measuring feature similarities between textual queries and image regions.<n>We propose textbfSGREC, an interpretable zero-shot REC method leveraging query-driven scene graphs as structured intermediaries.
  arXiv  Detail & Related papers  (2026-03-26T04:05:30Z)
• Spatial Preference Rewarding for MLLMs Spatial Understanding [92.25703021388142]
  Multimodal large language models (MLLMs) have demonstrated promising spatial understanding capabilities.<n>Despite their successes, MLLMs still fall short in fine-grained spatial perception abilities.<n>We propose a Spatial Preference Rewarding(SPR) approach that enhances MLLMs' spatial capabilities.
  arXiv  Detail & Related papers  (2025-10-16T07:16:18Z)
• MLLM-Guided VLM Fine-Tuning with Joint Inference for Zero-Shot Composed Image Retrieval [50.062817677022586]
  Zero-Shot Image Retrieval (ZS-CIR) methods typically train adapters that convert reference images into pseudo-text tokens.<n>We propose MLLM-Guided VLM Fine-Tuning with Joint Inference (MVFT-JI) to construct two complementary training tasks using only unlabeled images.
  arXiv  Detail & Related papers  (2025-05-26T08:56:59Z)
• Steered Generation via Gradient Descent on Sparse Features [1.534667887016089]
  We modify the internal structure of large language models (LLMs) by training sparse autoencoders to learn a sparse representation of the query embedding.<n>We demonstrate that manipulating this sparse representation effectively transforms the output toward different stylistic and cognitive targets.
  arXiv  Detail & Related papers  (2025-02-25T21:06:14Z)
• SPARC: Score Prompting and Adaptive Fusion for Zero-Shot Multi-Label Recognition in Vision-Language Models [74.40683913645731]
  Zero-shot multi-label recognition (MLR) with Vision-Language Models (VLMs) faces significant challenges without training data, model tuning, or architectural modifications.<n>Our work proposes a novel solution treating VLMs as black boxes, leveraging scores without training data or ground truth.<n>Analysis of these prompt scores reveals VLM biases and AND''/OR' signal ambiguities, notably that maximum scores are surprisingly suboptimal compared to second-highest scores.
  arXiv  Detail & Related papers  (2025-02-24T07:15:05Z)
• AlignVLM: Bridging Vision and Language Latent Spaces for Multimodal Document Understanding [79.43306110124875]
  AlignVLM is a vision-text alignment method that maps visual features to a weighted average of text embeddings.<n>Our experiments show that AlignVLM achieves state-of-the-art performance compared to prior alignment methods.
  arXiv  Detail & Related papers  (2025-02-03T13:34:51Z)
• Elevating Visual Perception in Multimodal LLMs with Visual Embedding Distillation [109.5893580175657]
  In recent times, the standard practice for developing MLLMs is to feed features from vision encoder(s) into the LLM and train with natural language supervision.<n>This approach often causes models to lean towards language comprehension and undermine the rich visual perception signals present in the data.<n>We propose VisPer-LM, the first approach that infuses visual perception knowledge from expert vision encoders into the LLM's hidden representations.
  arXiv  Detail & Related papers  (2024-12-12T18:55:18Z)
• Rethinking Semantic Parsing for Large Language Models: Enhancing LLM Performance with Semantic Hints [20.844061807562436]
  We propose SENSE, a novel prompting approach that embeds semantic hints within the prompt.<n> Experiments show that SENSE consistently improves LLMs' performance across various tasks.
  arXiv  Detail & Related papers  (2024-09-22T14:35:09Z)
• Zero-Shot Video Moment Retrieval from Frozen Vision-Language Models [58.17315970207874]
  We propose a zero-shot method for adapting generalisable visual-textual priors from arbitrary VLM to facilitate moment-text alignment. Experiments conducted on three VMR benchmark datasets demonstrate the notable performance advantages of our zero-shot algorithm.
  arXiv  Detail & Related papers  (2023-09-01T13:06:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.