OpenworldAUC: Towards Unified Evaluation and Optimization for Open-world Prompt Tuning

• URL: http://arxiv.org/abs/2505.05180v1
• Date: Thu, 08 May 2025 12:31:40 GMT
• Title: OpenworldAUC: Towards Unified Evaluation and Optimization for Open-world Prompt Tuning
• Authors: Cong Hua, Qianqian Xu, Zhiyong Yang, Zitai Wang, Shilong Bao, Qingming Huang,
• Abstract summary: Real-world scenarios require models to handle inputs without prior domain knowledge.<n>We propose OpenworldAUC, a metric that assesses detection and classification through pairwise instance comparisons.<n> Experiments on 15 benchmarks in open-world scenarios show OpenworldAUC achieves SOTA performance on OpenworldAUC and other metrics.
• Score: 86.20909814421748
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Prompt tuning adapts Vision-Language Models like CLIP to open-world tasks with minimal training costs. In this direction, one typical paradigm evaluates model performance separately on known classes (i.e., base domain) and unseen classes (i.e., new domain). However, real-world scenarios require models to handle inputs without prior domain knowledge. This practical challenge has spurred the development of open-world prompt tuning, which demands a unified evaluation of two stages: 1) detecting whether an input belongs to the base or new domain (P1), and 2) classifying the sample into its correct class (P2). What's more, as domain distributions are generally unknown, a proper metric should be insensitive to varying base/new sample ratios (P3). However, we find that current metrics, including HM, overall accuracy, and AUROC, fail to satisfy these three properties simultaneously. To bridge this gap, we propose OpenworldAUC, a unified metric that jointly assesses detection and classification through pairwise instance comparisons. To optimize OpenworldAUC effectively, we introduce Gated Mixture-of-Prompts (GMoP), which employs domain-specific prompts and a gating mechanism to dynamically balance detection and classification. Theoretical guarantees ensure generalization of GMoP under practical conditions. Experiments on 15 benchmarks in open-world scenarios show GMoP achieves SOTA performance on OpenworldAUC and other metrics. We release the code at https://github.com/huacong/OpenworldAUC

Related papers

• Interactive Classification Metrics: A graphical application to build robust intuition for classification model evaluation [0.0]
  Interactive Classification Metrics (ICM) is an application to visualize and explore the relationships between different evaluation metrics.<n>The user changes the distribution statistics and explores corresponding changes across a suite of evaluation metrics.
  arXiv  Detail & Related papers  (2024-12-22T15:36:15Z)
• Adapting Vision-Language Models to Open Classes via Test-Time Prompt Tuning [50.26965628047682]
  Adapting pre-trained models to open classes is a challenging problem in machine learning. In this paper, we consider combining the advantages of both and come up with a test-time prompt tuning approach. Our proposed method outperforms all comparison methods on average considering both base and new classes.
  arXiv  Detail & Related papers  (2024-08-29T12:34:01Z)
• OpenGCD: Assisting Open World Recognition with Generalized Category Discovery [4.600906853436266]
  A desirable open world recognition (OWR) system requires performing three tasks. We propose OpenGCD that combines three key ideas to solve the above problems sequentially. Experiments on two standard classification benchmarks and a challenging dataset demonstrate that OpenGCD not only offers excellent compatibility but also substantially outperforms other baselines.
  arXiv  Detail & Related papers  (2023-08-14T04:10:45Z)
• Self-Paced Learning for Open-Set Domain Adaptation [50.620824701934]
  Traditional domain adaptation methods presume that the classes in the source and target domains are identical. Open-set domain adaptation (OSDA) addresses this limitation by allowing previously unseen classes in the target domain. We propose a novel framework based on self-paced learning to distinguish common and unknown class samples.
  arXiv  Detail & Related papers  (2023-03-10T14:11:09Z)
• To Adapt or to Annotate: Challenges and Interventions for Domain Adaptation in Open-Domain Question Answering [46.403929561360485]
  We study end-to-end model performance of open-domain question answering (ODQA) We find that not only do models fail to generalize, but high retrieval scores often still yield poor answer prediction accuracy. We propose and evaluate several intervention methods which improve end-to-end answer F1 score by up to 24 points.
  arXiv  Detail & Related papers  (2022-12-20T16:06:09Z)
• Learning Classifiers of Prototypes and Reciprocal Points for Universal Domain Adaptation [79.62038105814658]
  Universal Domain aims to transfer the knowledge between datasets by handling two shifts: domain-shift and categoryshift. Main challenge is correctly distinguishing the unknown target samples while adapting the distribution of known class knowledge from source to target. Most existing methods approach this problem by first training the target adapted known and then relying on the single threshold to distinguish unknown target samples.
  arXiv  Detail & Related papers  (2022-12-16T09:01:57Z)
• Federated Adaptive Prompt Tuning for Multi-Domain Collaborative Learning [44.604485649167216]
  Federated learning (FL) enables multiple clients to collaboratively train a global model without disclosing their data. We propose a federated adaptive prompt tuning algorithm, FedAPT, for multi-domain collaborative image classification.
  arXiv  Detail & Related papers  (2022-11-15T03:10:05Z)
• Exploring the Open World Using Incremental Extreme Value Machines [11.3660790934494]
  Open world recognition is a demanding task that is, to the best of our knowledge, addressed by only a few methods. This work introduces a modification of the widely known Extreme Value Machine to enable open world recognition. The proposed method achieves superior accuracy of about 12 % and computational efficiency in the tasks of image classification and face recognition.
  arXiv  Detail & Related papers  (2022-05-30T07:21:13Z)
• On Universal Black-Box Domain Adaptation [53.7611757926922]
  We study an arguably least restrictive setting of domain adaptation in a sense of practical deployment. Only the interface of source model is available to the target domain, and where the label-space relations between the two domains are allowed to be different and unknown. We propose to unify them into a self-training framework, regularized by consistency of predictions in local neighborhoods of target samples.
  arXiv  Detail & Related papers  (2021-04-10T02:21:09Z)
• Fine-Grained Visual Classification with Efficient End-to-end Localization [49.9887676289364]
  We present an efficient localization module that can be fused with a classification network in an end-to-end setup. We evaluate the new model on the three benchmark datasets CUB200-2011, Stanford Cars and FGVC-Aircraft.
  arXiv  Detail & Related papers  (2020-05-11T14:07:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.