MaskCLIP++: A Mask-Based CLIP Fine-tuning Framework for Open-Vocabulary Image Segmentation

• URL: http://arxiv.org/abs/2412.11464v2
• Date: Tue, 24 Dec 2024 04:13:08 GMT
• Title: MaskCLIP++: A Mask-Based CLIP Fine-tuning Framework for Open-Vocabulary Image Segmentation
• Authors: Quan-Sheng Zeng, Yunheng Li, Daquan Zhou, Guanbin Li, Qibin Hou, Ming-Ming Cheng,
• Abstract summary: Open-vocabulary image segmentation has been advanced through the synergy between mask generators and vision-language models. We present a new fine-tuning framework, named MaskCLIP++, which uses ground-truth masks instead of generated masks. We achieve performance improvements of +1.7, +2.3, +2.1, +3.1, and +0.3 mIoU on the A-847, PC-459, A-150, PC-59, and PAS-20 datasets.
• Score: 109.19165503929992
• License:
• Abstract: Open-vocabulary image segmentation has been advanced through the synergy between mask generators and vision-language models like Contrastive Language-Image Pre-training (CLIP). Previous approaches focus on generating masks while aligning mask features with text embeddings during training. In this paper, we observe that relying on generated low-quality masks can weaken the alignment of vision and language in regional representations. This motivates us to present a new fine-tuning framework, named MaskCLIP++, which uses ground-truth masks instead of generated masks to enhance the mask classification capability of CLIP. Due to the limited diversity of image segmentation datasets with mask annotations, we propose incorporating a consistency alignment constraint during fine-tuning, which alleviates categorical bias toward the fine-tuning dataset. After low-cost fine-tuning, combining with the mask generator in previous state-of-the-art mask-based open vocabulary segmentation methods, we achieve performance improvements of +1.7, +2.3, +2.1, +3.1, and +0.3 mIoU on the A-847, PC-459, A-150, PC-59, and PAS-20 datasets, respectively. Code is released at https://github.com/HVision-NKU/MaskCLIPpp .

Related papers

• Mask-Adapter: The Devil is in the Masks for Open-Vocabulary Segmentation [39.73550543404763]
  We introduce Mask-Adapter, a simple yet effective method to address these challenges in open-vocabulary segmentation. Compared to directly using proposal masks, our proposed Mask-Adapter extracts semantic activation maps from proposal masks. Mask-Adapter integrates seamlessly into open-vocabulary segmentation methods based on mask pooling in a plug-and-play manner.
  arXiv  Detail & Related papers  (2024-12-05T17:42:37Z)
• MTA-CLIP: Language-Guided Semantic Segmentation with Mask-Text Alignment [53.235290505274676]
  Large-scale vision-language models such as CLIP can improve semantic segmentation performance. We introduce MTA-CLIP, a novel framework employing mask-level vision-language alignment. MTA-CLIP achieves state-of-the-art, surpassing prior works by an average of 2.8% and 1.3% on benchmark datasets.
  arXiv  Detail & Related papers  (2024-07-31T14:56:42Z)
• Variance-insensitive and Target-preserving Mask Refinement for Interactive Image Segmentation [68.16510297109872]
  Point-based interactive image segmentation can ease the burden of mask annotation in applications such as semantic segmentation and image editing. We introduce a novel method, Variance-Insensitive and Target-Preserving Mask Refinement to enhance segmentation quality with fewer user inputs. Experiments on GrabCut, Berkeley, SBD, and DAVIS datasets demonstrate our method's state-of-the-art performance in interactive image segmentation.
  arXiv  Detail & Related papers  (2023-12-22T02:31:31Z)
• Learning Mask-aware CLIP Representations for Zero-Shot Segmentation [120.97144647340588]
  Mask-awareProposals CLIP (IP-CLIP) is proposed to handle arbitrary numbers of image and mask proposals simultaneously. mask-aware loss and self-distillation loss are designed to fine-tune IP-CLIP, ensuring CLIP is responsive to different mask proposals. We conduct extensive experiments on the popular zero-shot benchmarks.
  arXiv  Detail & Related papers  (2023-09-30T03:27:31Z)
• Automatic Generation of Semantic Parts for Face Image Synthesis [7.728916126705043]
  We describe a network architecture to address the problem of automatically manipulating or generating the shape of object classes in semantic segmentation masks. Our proposed model allows embedding the mask class-wise into a latent space where each class embedding can be independently edited. We report quantitative and qualitative results on the Celeb-MaskHQ dataset, which show our model can both faithfully reconstruct and modify a segmentation mask at the class level.
  arXiv  Detail & Related papers  (2023-07-11T15:01:42Z)
• CoupAlign: Coupling Word-Pixel with Sentence-Mask Alignments for Referring Image Segmentation [104.5033800500497]
  Referring image segmentation aims at localizing all pixels of the visual objects described by a natural language sentence. Previous works learn to straightforwardly align the sentence embedding and pixel-level embedding for highlighting the referred objects. We propose CoupAlign, a simple yet effective multi-level visual-semantic alignment method.
  arXiv  Detail & Related papers  (2022-12-04T08:53:42Z)
• Open-Vocabulary Semantic Segmentation with Mask-adapted CLIP [45.81698881151867]
  Open-vocabulary semantic segmentation aims to segment an image into semantic regions according to text descriptions, which may not have been seen during training. Recent two-stage methods first generate class-agnostic mask proposals and then leverage pre-trained vision-language models, e.g., CLIP, to classify masked regions. We propose to finetune CLIP on a collection of masked image regions and their corresponding text descriptions. In particular, when trained on COCO and evaluated on ADE20K-150, our best model achieves 29.6% mIoU, which is +8.5% higher than the previous state-
  arXiv  Detail & Related papers  (2022-10-09T02:57:32Z)
• Open-Vocabulary Universal Image Segmentation with MaskCLIP [24.74805434602145]
  We tackle an emerging computer vision task, open-vocabulary universal image segmentation. We first build a baseline method by directly adopting pre-trained CLIP models. We then develop MaskCLIP, a Transformer-based approach with a MaskCLIP Visual.
  arXiv  Detail & Related papers  (2022-08-18T17:55:37Z)
• Open-Vocabulary Instance Segmentation via Robust Cross-Modal Pseudo-Labeling [61.03262873980619]
  Open-vocabulary instance segmentation aims at segmenting novel classes without mask annotations. We propose a cross-modal pseudo-labeling framework, which generates training pseudo masks by aligning word semantics in captions with visual features of object masks in images. Our framework is capable of labeling novel classes in captions via their word semantics to self-train a student model.
  arXiv  Detail & Related papers  (2021-11-24T18:50:47Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.