Criterion-Conditional In-Context Learning: Evaluating Criterion-Shift Adaptation in Vision-Language Models

• URL: http://arxiv.org/abs/2607.02575v1
• Date: Tue, 30 Jun 2026 17:22:20 GMT
• Title: Criterion-Conditional In-Context Learning: Evaluating Criterion-Shift Adaptation in Vision-Language Models
• Abstract summary: Vision-language models can perform new tasks without parameter updates through in-context learning (ICL)<n>In real-world applications, many tasks exhibit a stable high-level intent, while their decision criteria shift according to specific requirements.<n>We introduce a new setting, denoted as Criterion-Conditional In-Context Learning (CC-ICL), where models must infer the latent criterion from context.
• Score: 2.2633326936241196
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Vision-language models can perform new tasks without parameter updates through in-context learning (ICL), whose core mechanism is utilizing the support set for task induction. In the standard ICL setting, once the task is induced, its decision criterion remains fixed. However, in real-world applications, many tasks exhibit a stable high-level intent, while their decision criteria shift according to specific requirements. Thus, we introduce a new setting, denoted as Criterion-Conditional In-Context Learning (CC-ICL), where models must infer the latent criterion from context and adjust predictions accordingly under fixed task semantics. To evaluate this capability, we propose two complementary metrics, Criterion Invariance and Criterion Sensitivity, capturing the model's robustness and adaptability under criterion shifts. We further construct CC-Bench, a multi-domain benchmark that supports evaluation under the CC-ICL setting. By employing a dual-level data hierarchy, CC-Bench enables legitimate ground-truth variation conditioned on the active criterion even when the task remains fixed. Experiments on CC-Bench reveal that most models exhibit a rigid boundary bias, struggling to align their decisions with the latent criterion. We also find that even a simple multi-criterion training strategy can significantly reduce this bias, improving Criterion Sensitivity and enabling 7B-scale models to surpass proprietary models without degrading general multimodal performance.

Related papers

• Evaluating Criterion-Conditioned Behaviour of Large Language Models in Content Moderation [1.3629559589878444]
  We study whether large language models (LLMs) exhibit criterion-conditioned behaviour.<n>We find that strong benchmark performance can hide substantial failures at the criterion level.<n>These findings call for the development of evaluation methods that explicitly measure criterion-conditioned behaviour.
  arXiv  Detail & Related papers  (2026-09-03T13:17:38Z)
• CritICL: Inference-Time Weak-to-Strong Generalization from Small Language Model Failure Modes [8.494161214945963]
  We introduce CritICL, a novel inference-time framework that improves reasoning while maintaining high efficiency.<n>We utilize failure modes derived from weaker models and incorporate them into inference through critique-based in-context examples.<n>CRITICL consistently outperforms standard in-context learning and achieves performance competitive with or superior to test-time scaling methods.
  arXiv  Detail & Related papers  (2026-08-27T17:59:30Z)
• Capturing LLM Capabilities via Evidence-Calibrated Query Clustering [12.71199389824453]
  We propose ECC, an algorithm that calibrates prior semantic embeddings to bridge the gap between surface-level semantics and latent capability requirements.<n> ECC characterizes each cluster through a capability profile parameterized by a Bradley-Terry model and uses trainable mixture weights to accommodate queries with mixed capability demands.
  arXiv  Detail & Related papers  (2026-05-16T18:30:37Z)
• Multimodal Learning on Low-Quality Data with Conformal Predictive Self-Calibration [72.0672328514289]
  Multimodal learning often grapples with the challenge of low-quality data.<n>We propose a unified framework, termed Conformal Predictive Self-Calibration.<n>Our framework consistently outperforms existing state-of-the-art methods.
  arXiv  Detail & Related papers  (2026-05-05T14:48:52Z)
• A Sanity Check on Composed Image Retrieval [91.95275287747499]
  Composed Image Retrieval (CIR) aims to retrieve a target image based on a query composed of a reference image, and a relative caption that specifies the desired modification.<n>We introduce FISD, a Fully-Informed Semantically-Diverse benchmark, which employs generative models to precisely control the variables of reference-target image pairs.<n>We propose an automatic multi-round agentic evaluation framework to probe the potential of the existing models in the interactive scenarios.
  arXiv  Detail & Related papers  (2026-04-14T15:52:22Z)
• CLSGen: A Dual-Head Fine-Tuning Framework for Joint Probabilistic Classification and Verbalized Explanation [9.80652710081054]
  CLSGen is a novel fine-tuning framework for binary classification tasks.<n>It enables robust probability estimation without sacrificing the model's inherent explanation-generation capabilities.<n>Results show that models fine-tuned with CLSGen outperform existing baselines in classification metrics.
  arXiv  Detail & Related papers  (2026-04-13T17:58:43Z)
• CDRRM: Contrast-Driven Rubric Generation for Reliable and Interpretable Reward Modeling [61.75914342638658]
  CDRRM (Contrast-Driven Reward Model) is a framework for high-quality rubric generation and guided preference judgment.<n>This work offers a scalable, interpretable, and data-efficient path for reward modeling.
  arXiv  Detail & Related papers  (2026-03-09T07:15:23Z)
• CE-RM: A Pointwise Generative Reward Model Optimized via Two-Stage Rollout and Unified Criteria [48.70940362676624]
  We propose CE-RM-4B, a pointwise generative reward model trained with a dedicated two-stage rollout method.<n>Using only about 5.7K high-quality data curated from the open-source preference dataset, our CE-RM-4B achieves superior performance on diverse reward model benchmarks.
  arXiv  Detail & Related papers  (2026-01-28T07:46:13Z)
• Cost-Sensitive Evaluation for Binary Classifiers [0.013048920509133805]
  Weighted Accuracy (WA) is an evaluation metric for binary classifiers with a straightforward interpretation as a weighted version of the well-known accuracy metric.<n>We clarify the conceptual framework for handling class imbalance in cost-sensitive scenarios.
  arXiv  Detail & Related papers  (2025-10-24T20:34:18Z)
• LLM4Rerank: LLM-based Auto-Reranking Framework for Recommendations [51.76373105981212]
  Reranking is a critical component in recommender systems, playing an essential role in refining the output of recommendation algorithms.<n>We introduce a comprehensive reranking framework, designed to seamlessly integrate various reranking criteria.<n>A customizable input mechanism is also integrated, enabling the tuning of the language model's focus to meet specific reranking needs.
  arXiv  Detail & Related papers  (2024-06-18T09:29:18Z)
• Mitigating Catastrophic Forgetting in Task-Incremental Continual Learning with Adaptive Classification Criterion [50.03041373044267]
  We propose a Supervised Contrastive learning framework with adaptive classification criterion for Continual Learning. Experiments show that CFL achieves state-of-the-art performance and has a stronger ability to overcome compared with the classification baselines.
  arXiv  Detail & Related papers  (2023-05-20T19:22:40Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.