Adapt CLIP as Aggregation Instructor for Image Dehazing

• URL: http://arxiv.org/abs/2408.12317v1
• Date: Thu, 22 Aug 2024 11:51:50 GMT
• Title: Adapt CLIP as Aggregation Instructor for Image Dehazing
• Authors: Xiaozhe Zhang, Fengying Xie, Haidong Ding, Linpeng Pan, Zhenwei Shi,
• Abstract summary: Most dehazing methods suffer from limited receptive field and do not explore the rich semantic prior encapsulated in vision-language models. We introduce CLIPHaze, a pioneering hybrid framework that synergizes the efficient global modeling of Mamba with the prior knowledge and zero-shot capabilities of CLIP. Our method employs parallel state space model and window-based self-attention to obtain global contextual dependency and local fine-grained perception.
• Score: 17.29370328189668
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Most dehazing methods suffer from limited receptive field and do not explore the rich semantic prior encapsulated in vision-language models, which have proven effective in downstream tasks. In this paper, we introduce CLIPHaze, a pioneering hybrid framework that synergizes the efficient global modeling of Mamba with the prior knowledge and zero-shot capabilities of CLIP to address both issues simultaneously. Specifically, our method employs parallel state space model and window-based self-attention to obtain global contextual dependency and local fine-grained perception, respectively. To seamlessly aggregate information from both paths, we introduce CLIP-instructed Aggregation Module (CAM). For non-homogeneous and homogeneous haze, CAM leverages zero-shot estimated haze density map and high-quality image embedding without degradation information to explicitly and implicitly determine the optimal neural operation range for each pixel, thereby adaptively fusing two paths with different receptive fields. Extensive experiments on various benchmarks demonstrate that CLIPHaze achieves state-of-the-art (SOTA) performance, particularly in non-homogeneous haze. Code will be publicly after acceptance.

Related papers

• Continual Panoptic Perception: Towards Multi-modal Incremental Interpretation of Remote Sensing Images [16.0258685984844]
  Continual learning (CL) breaks off the one-way training manner and enables a model to adapt to new data, semantics and tasks continuously. We propose a unified continual learning model that leverages multi-task joint learning covering pixel-level classification, instance-level segmentation and image-level perception.
  arXiv  Detail & Related papers  (2024-07-19T12:22:32Z)
• Enhancing Fine-Grained Image Classifications via Cascaded Vision Language Models [0.0]
  This paper introduces CascadeVLM, an innovative framework that overcomes the constraints of previous CLIP-based methods. Experiments across various fine-grained image datasets demonstrate that CascadeVLM significantly outperforms existing models.
  arXiv  Detail & Related papers  (2024-05-18T14:12:04Z)
• Dual-Image Enhanced CLIP for Zero-Shot Anomaly Detection [58.228940066769596]
  We introduce a Dual-Image Enhanced CLIP approach, leveraging a joint vision-language scoring system. Our methods process pairs of images, utilizing each as a visual reference for the other, thereby enriching the inference process with visual context. Our approach significantly exploits the potential of vision-language joint anomaly detection and demonstrates comparable performance with current SOTA methods across various datasets.
  arXiv  Detail & Related papers  (2024-05-08T03:13:20Z)
• PosSAM: Panoptic Open-vocabulary Segment Anything [58.72494640363136]
  PosSAM is an open-vocabulary panoptic segmentation model that unifies the strengths of the Segment Anything Model (SAM) with the vision-native CLIP model in an end-to-end framework. We introduce a Mask-Aware Selective Ensembling (MASE) algorithm that adaptively enhances the quality of generated masks and boosts the performance of open-vocabulary classification during inference for each image.
  arXiv  Detail & Related papers  (2024-03-14T17:55:03Z)
• Dual-Context Aggregation for Universal Image Matting [16.59886660634162]
  We propose a simple and universal matting framework, named Dual-Context Aggregation Matting (DCAM) Specifically, DCAM first adopts a semantic backbone network to extract low-level features and context features from the input image and guidance. By performing both global contour segmentation and local boundary refinement, DCAM exhibits robustness to diverse types of guidance and objects.
  arXiv  Detail & Related papers  (2024-02-28T06:56:24Z)
• High-fidelity Pseudo-labels for Boosting Weakly-Supervised Segmentation [17.804090651425955]
  Image-level weakly-supervised segmentation (WSSS) reduces the usually vast data annotation cost by surrogate segmentation masks during training. Our work is based on two techniques for improving CAMs; importance sampling, which is a substitute for GAP, and the feature similarity loss. We reformulate both techniques based on binomial posteriors of multiple independent binary problems. This has two benefits; their performance is improved and they become more general, resulting in an add-on method that can boost virtually any WSSS method.
  arXiv  Detail & Related papers  (2023-04-05T17:43:57Z)
• CLIPood: Generalizing CLIP to Out-of-Distributions [73.86353105017076]
  Contrastive language-image pre-training (CLIP) models have shown impressive zero-shot ability, but the further adaptation of CLIP on downstream tasks undesirably degrades OOD performances. We propose CLIPood, a fine-tuning method that can adapt CLIP models to OOD situations where both domain shifts and open classes may occur on unseen test data. Experiments on diverse datasets with different OOD scenarios show that CLIPood consistently outperforms existing generalization techniques.
  arXiv  Detail & Related papers  (2023-02-02T04:27:54Z)
• One-Shot Adaptation of GAN in Just One CLIP [51.188396199083336]
  We present a novel single-shot GAN adaptation method through unified CLIP space manipulations. Specifically, our model employs a two-step training strategy: reference image search in the source generator using a CLIP-guided latent optimization. We show that our model generates diverse outputs with the target texture and outperforms the baseline models both qualitatively and quantitatively.
  arXiv  Detail & Related papers  (2022-03-17T13:03:06Z)
• Semi-supervised Domain Adaptive Structure Learning [72.01544419893628]
  Semi-supervised domain adaptation (SSDA) is a challenging problem requiring methods to overcome both 1) overfitting towards poorly annotated data and 2) distribution shift across domains. We introduce an adaptive structure learning method to regularize the cooperation of SSL and DA.
  arXiv  Detail & Related papers  (2021-12-12T06:11:16Z)
• SSA: Semantic Structure Aware Inference for Weakly Pixel-Wise Dense Predictions without Cost [36.27226683586425]
  The semantic structure aware inference (SSA) is proposed to explore the semantic structure information hidden in different stages of the CNN-based network to generate high-quality CAM in the model inference. The proposed method has the advantage of no parameters and does not need to be trained. Therefore, it can be applied to a wide range of weakly-supervised pixel-wise dense prediction tasks.
  arXiv  Detail & Related papers  (2021-11-05T11:07:21Z)
• Inter-class Discrepancy Alignment for Face Recognition [55.578063356210144]
  We propose a unified framework calledInter-class DiscrepancyAlignment(IDA) IDA-DAO is used to align the similarity scores considering the discrepancy between the images and its neighbors. IDA-SSE can provide convincing inter-class neighbors by introducing virtual candidate images generated with GAN.
  arXiv  Detail & Related papers  (2021-03-02T08:20:08Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.