[CLS] is Not Enough: Multi-Label Recognition via Patch-Level Inference and Adaptive Aggregation

• URL: http://arxiv.org/abs/2605.25821v1
• Date: Mon, 25 May 2026 13:19:12 GMT
• Title: [CLS] is Not Enough: Multi-Label Recognition via Patch-Level Inference and Adaptive Aggregation
• Abstract summary: PIAA formulates prediction as Patch-level Inference followed by Adaptive Aggregation.<n>We introduce an adaptive aggregation module that consolidates patch-level scores into the final multi-label prediction.<n>Experiments show that our method achieves strong improvements with minimal extra computation.
• Score: 20.637119409165418
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Vision-Language Models such as CLIP exhibit strong zero-shot recognition capability by aligning images with textual concepts, yet they often underperform on multi-label recognition where multiple objects co-exist. A key bottleneck is that the [CLS] token, as a single global visual representation, is insufficient to faithfully encode diverse targets with varying scales, contexts, and co-occurrence patterns. To address this limitation, we present a new multi-label image recognition framework, termed PIAA, which formulates prediction as Patch-level Inference followed by Adaptive Aggregation. Specifically, we first enhance patch-wise predictions from two complementary perspectives: (i) mitigating semantic entanglement in the visual encoder to obtain more discriminative patch representations, and (ii) learning an unsupervised visual classifier to narrow the vision-language modality gap. We then introduce an adaptive aggregation module that consolidates patch-level scores into the final multi-label prediction. Notably, the entire pipeline is fully training-free, requiring no gradient updates or parameter fine-tuning. Experiments show that our method achieves strong improvements with minimal extra computation, exceeding a 6% mAP gain on the challenging NUS-WIDE benchmark over representative baselines. Code is available at https://github.com/akang-wang/PIAA.

Related papers

• When Semantically Consistent Encoding Meets View-Label Heterogeneity Modeling: A Unified Framework for Incomplete Multi-View Multi-Label Learning [35.935368500144975]
  This paper presents V2L, a unified representation-decision framework for incomplete multi-view multi-label classification.<n>On the representation side, V2L constructs semantically consistent variational posteriors from incomplete views.<n>On the decision side, V2L introduces an active view-label relevance modeling strategy.
  arXiv  Detail & Related papers  (2026-09-07T14:07:10Z)
• Unified Multimodal Autoregressive Modeling with Shared Context-Visual Tokenizer is Key to Unification [80.62512020268626]
  UniAR is a unified autoregressive framework where a single visual tokenizer serves as the key bridge between understanding and generation.<n>UniAR adapts a pretrained vision encoder with multi-level feature fusion and a lookup-free bitwise quantization scheme.<n>A diffusion-based visual decoder operates on discrete visual tokens to decode high-fidelity images.
  arXiv  Detail & Related papers  (2026-06-16T17:59:22Z)
• Adapting Vision-Language Models from Iconic to Inclusive for Multi-Label Recognition Without Labels [14.373553090930612]
  Vision-language models (VLMs) enable zero-shot recognition without labeled data.<n>These models often prioritize the most iconic object and omit other contextual positives.<n>We propose an unsupervised framework that adapts VLMs from iconic recognition toward inclusive understanding.
  arXiv  Detail & Related papers  (2026-06-10T03:41:50Z)
• Not All Tokens See Equally: Perception-Grounded Policy Optimization for Large Vision-Language Models [38.47027398567909]
  Perception-Grounded Policy Optimization (PGPO) is a novel fine-grained credit assignment framework that dynamically reshapes advantages at the token level.<n>We show that PGPO actively amplifies learning signals for visually-dependent tokens while suppressing gradient noise from linguistic priors.<n>Both theoretical and empirical analyses confirm that PGPO effectively reduces gradient variance, prevents training collapse, and acts as a potent regularizer for robust, perception-grounded multimodal reasoning.
  arXiv  Detail & Related papers  (2026-04-02T09:53:20Z)
• Test-Time Conditioning with Representation-Aligned Visual Features [9.262325724962485]
  We introduce Representation-Aligned Guidance (REPA-G), a framework that leverages aligned representations with rich semantic properties.<n>We steer the denoising process toward a conditioned representation extracted from a pre-trained feature extractor.<n>Our method provides versatile control at multiple scales, ranging from fine-grained texture matching via single patches to broad semantic guidance.
  arXiv  Detail & Related papers  (2026-02-03T17:15:03Z)
• Unleashing the Power of Vision-Language Models for Long-Tailed Multi-Label Visual Recognition [55.189113121465816]
  We propose a novel correlation adaptation prompt network (CAPNET) for long-tailed multi-label visual recognition.<n>CAPNET explicitly models correlations from CLIP's textual encoder.<n>It improves generalization through test-time ensembling and realigns visual-textual modalities.
  arXiv  Detail & Related papers  (2025-11-25T18:57:28Z)
• Semantic-Aligned Learning with Collaborative Refinement for Unsupervised VI-ReID [82.12123628480371]
  Unsupervised person re-identification (USL-VI-ReID) seeks to match pedestrian images of the same individual across different modalities without human annotations for model learning.<n>Previous methods unify pseudo-labels of cross-modality images through label association algorithms and then design contrastive learning framework for global feature learning.<n>We propose a Semantic-Aligned Learning with Collaborative Refinement (SALCR) framework, which builds up objective for specific fine-grained patterns emphasized by each modality.
  arXiv  Detail & Related papers  (2025-04-27T13:58:12Z)
• LPOSS: Label Propagation Over Patches and Pixels for Open-vocabulary Semantic Segmentation [16.021683473678515]
  We propose a training-free method for semantic segmentation using Vision-and-Language Models (VLMs)<n>Our approach enhances the initial per-patch predictions of VLMs through label propagation.<n>Our method, called LPOSS+, performs inference over the entire image, avoiding window-based processing.
  arXiv  Detail & Related papers  (2025-03-25T15:47:13Z)
• Subobject-level Image Tokenization [60.80949852899857]
  Patch-based image tokenization ignores the morphology of the visual world.<n>Inspired by subword tokenization, we introduce subobject-level adaptive token segmentation.<n>We show that subobject tokenization enables faster convergence and better generalization while using fewer visual tokens.
  arXiv  Detail & Related papers  (2024-02-22T06:47:44Z)
• Noise-Tolerant Few-Shot Unsupervised Adapter for Vision-Language Models [8.59772105902647]
  We design NtUA, a Noise-tolerant Unsupervised Adapter that allows the learning of effective target models with few unlabelled target samples. NtUA works as a key-value cache that formulates visual features and predicted pseudo-labels of the few unlabelled target samples as key-value pairs. NtUA achieves superior performance consistently across multiple widely adopted benchmarks.
  arXiv  Detail & Related papers  (2023-09-26T13:35:31Z)
• UIA-ViT: Unsupervised Inconsistency-Aware Method based on Vision Transformer for Face Forgery Detection [52.91782218300844]
  We propose a novel Unsupervised Inconsistency-Aware method based on Vision Transformer, called UIA-ViT. Due to the self-attention mechanism, the attention map among patch embeddings naturally represents the consistency relation, making the vision Transformer suitable for the consistency representation learning.
  arXiv  Detail & Related papers  (2022-10-23T15:24:47Z)
• Patch-level Representation Learning for Self-supervised Vision Transformers [68.8862419248863]
  Vision Transformers (ViTs) have gained much attention recently as a better architectural choice, often outperforming convolutional networks for various visual tasks. Inspired by this, we design a simple yet effective visual pretext task, coined SelfPatch, for learning better patch-level representations. We demonstrate that SelfPatch can significantly improve the performance of existing SSL methods for various visual tasks.
  arXiv  Detail & Related papers  (2022-06-16T08:01:19Z)
• Cluster-to-Conquer: A Framework for End-to-End Multi-Instance Learning for Whole Slide Image Classification [7.876654642325896]
  We propose an end-to-end framework that clusters the patches from a Whole Slide Images (WSI) into $k$-groups, samples $k'$ patches from each group for training, and uses an adaptive attention mechanism for slide level prediction. The framework is optimized end-to-end on slide-level cross-entropy, patch-level cross-entropy, and KL-divergence loss.
  arXiv  Detail & Related papers  (2021-03-19T04:24:01Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.