Multimodal Multilabel Classification by CLIP

• URL: http://arxiv.org/abs/2406.16141v1
• Date: Sun, 23 Jun 2024 15:28:07 GMT
• Title: Multimodal Multilabel Classification by CLIP
• Authors: Yanming Guo,
• Abstract summary: Multimodal multilabel classification (MMC) is a challenging task that aims to design a learning algorithm to handle two data sources. We leverage a novel technique that utilise the Contrastive Language-Image Pre-training (CLIP) as the feature extractor.
• Score: 3.1002416427168304
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Multimodal multilabel classification (MMC) is a challenging task that aims to design a learning algorithm to handle two data sources, the image and text, and learn a comprehensive semantic feature presentation across the modalities. In this task, we review the extensive number of state-of-the-art approaches in MMC and leverage a novel technique that utilises the Contrastive Language-Image Pre-training (CLIP) as the feature extractor and fine-tune the model by exploring different classification heads, fusion methods and loss functions. Finally, our best result achieved more than 90% F_1 score in the public Kaggle competition leaderboard. This paper provides detailed descriptions of novel training methods and quantitative analysis through the experimental results.

Related papers

• Masked Image Modeling: A Survey [73.21154550957898]
  Masked image modeling emerged as a powerful self-supervised learning technique in computer vision. We construct a taxonomy and review the most prominent papers in recent years. We aggregate the performance results of various masked image modeling methods on the most popular datasets.
  arXiv  Detail & Related papers  (2024-08-13T07:27:02Z)
• Investigating Self-Supervised Methods for Label-Efficient Learning [27.029542823306866]
  We study different self supervised pretext tasks, namely contrastive learning, clustering, and masked image modelling for their low-shot capabilities. We introduce a framework involving both mask image modelling and clustering as pretext tasks, which performs better across all low-shot downstream tasks. When testing the model on full scale datasets, we show performance gains in multi-class classification, multi-label classification and semantic segmentation.
  arXiv  Detail & Related papers  (2024-06-25T10:56:03Z)
• MM1: Methods, Analysis & Insights from Multimodal LLM Pre-training [103.72844619581811]
  We build performant Multimodal Large Language Models (MLLMs) In particular, we study the importance of various architecture components and data choices. We demonstrate that for large-scale multimodal pre-training using a careful mix of image-caption, interleaved image-text, and text-only data.
  arXiv  Detail & Related papers  (2024-03-14T17:51:32Z)
• Data-free Multi-label Image Recognition via LLM-powered Prompt Tuning [23.671999163027284]
  This paper proposes a novel framework for multi-label image recognition without any training data. It uses knowledge of pre-trained Large Language Model to learn prompts to adapt pretrained Vision-Language Model like CLIP to multilabel classification. Our framework presents a new way to explore the synergies between multiple pre-trained models for novel category recognition.
  arXiv  Detail & Related papers  (2024-03-02T13:43:32Z)
• Federated Multi-Task Learning on Non-IID Data Silos: An Experimental Study [17.555311732095483]
  The FMTL approach consolidates the benefits of Federated Learning (FL) and Multi-Task Learning (MTL) This paper introduces a novel framework, FMTL-Bench, for systematic evaluation of the FMTL paradigm.
  arXiv  Detail & Related papers  (2024-02-20T10:13:44Z)
• Convolutional autoencoder-based multimodal one-class classification [80.52334952912808]
  One-class classification refers to approaches of learning using data from a single class only. We propose a deep learning one-class classification method suitable for multimodal data.
  arXiv  Detail & Related papers  (2023-09-25T12:31:18Z)
• Partner-Assisted Learning for Few-Shot Image Classification [54.66864961784989]
  Few-shot Learning has been studied to mimic human visual capabilities and learn effective models without the need of exhaustive human annotation. In this paper, we focus on the design of training strategy to obtain an elemental representation such that the prototype of each novel class can be estimated from a few labeled samples. We propose a two-stage training scheme, which first trains a partner encoder to model pair-wise similarities and extract features serving as soft-anchors, and then trains a main encoder by aligning its outputs with soft-anchors while attempting to maximize classification performance.
  arXiv  Detail & Related papers  (2021-09-15T22:46:19Z)
• Multimodal Contrastive Training for Visual Representation Learning [45.94662252627284]
  We develop an approach to learning visual representations that embraces multimodal data. Our method exploits intrinsic data properties within each modality and semantic information from cross-modal correlation simultaneously. By including multimodal training in a unified framework, our method can learn more powerful and generic visual features.
  arXiv  Detail & Related papers  (2021-04-26T19:23:36Z)
• The Labeled Multiple Canonical Correlation Analysis for Information Fusion [25.23035811685684]
  We introduce a new method for multimodal information fusion and representation based on the Labeled Multiple Canonical Correlation Analysis (LMCCA) We implement a prototype of LMCCA to demonstrate its effectiveness on handwritten digit recognition,face recognition and object recognition.
  arXiv  Detail & Related papers  (2021-02-28T00:13:36Z)
• Multi-Task Learning for Dense Prediction Tasks: A Survey [87.66280582034838]
  Multi-task learning (MTL) techniques have shown promising results w.r.t. performance, computations and/or memory footprint. We provide a well-rounded view on state-of-the-art deep learning approaches for MTL in computer vision.
  arXiv  Detail & Related papers  (2020-04-28T09:15:50Z)
• Unpaired Multi-modal Segmentation via Knowledge Distillation [77.39798870702174]
  We propose a novel learning scheme for unpaired cross-modality image segmentation. In our method, we heavily reuse network parameters, by sharing all convolutional kernels across CT and MRI. We have extensively validated our approach on two multi-class segmentation problems.
  arXiv  Detail & Related papers  (2020-01-06T20:03:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.