CLAMP-ViT: Contrastive Data-Free Learning for Adaptive Post-Training Quantization of ViTs

• URL: http://arxiv.org/abs/2407.05266v2
• Date: Mon, 9 Sep 2024 00:08:36 GMT
• Title: CLAMP-ViT: Contrastive Data-Free Learning for Adaptive Post-Training Quantization of ViTs
• Authors: Akshat Ramachandran, Souvik Kundu, Tushar Krishna,
• Abstract summary: We present CLAMP-ViT, a data-free post-training quantization method for vision transformers (ViTs) We identify the limitations of recent techniques, notably their inability to leverage meaningful inter-patch relationships. CLAMP-ViT employs a two-stage approach, cyclically adapting between data generation and model quantization.
• Score: 6.456189487006878
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We present CLAMP-ViT, a data-free post-training quantization method for vision transformers (ViTs). We identify the limitations of recent techniques, notably their inability to leverage meaningful inter-patch relationships, leading to the generation of simplistic and semantically vague data, impacting quantization accuracy. CLAMP-ViT employs a two-stage approach, cyclically adapting between data generation and model quantization. Specifically, we incorporate a patch-level contrastive learning scheme to generate richer, semantically meaningful data. Furthermore, we leverage contrastive learning in layer-wise evolutionary search for fixed- and mixed-precision quantization to identify optimal quantization parameters while mitigating the effects of a non-smooth loss landscape. Extensive evaluations across various vision tasks demonstrate the superiority of CLAMP-ViT, with performance improvements of up to 3% in top-1 accuracy for classification, 0.6 mAP for object detection, and 1.5 mIoU for segmentation at similar or better compression ratio over existing alternatives. Code is available at https://github.com/georgia-tech-synergy-lab/CLAMP-ViT.git

Related papers

• Zero-Shot Embeddings Inform Learning and Forgetting with Vision-Language Encoders [6.7181844004432385]
  The Inter-Intra Modal Measure (IIMM) functions as a strong predictor of performance changes with fine-tuning. Fine-tuning on tasks with higher IIMM scores produces greater in-domain performance gains but also induces more severe out-of-domain performance degradation. With only a single forward pass of the target data, practitioners can leverage this key insight to evaluate the degree to which a model can be expected to improve following fine-tuning.
  arXiv  Detail & Related papers  (2024-07-22T15:35:09Z)
• Zero-Shot Class Unlearning in CLIP with Synthetic Samples [0.0]
  We focus on unlearning within CLIP, a dual vision-language model trained on a massive dataset of image-text pairs. We apply Lipschitz regularization to the multimodal context of CLIP. Our forgetting procedure is iterative, where we track accuracy on a synthetic forget set and stop when accuracy falls below a chosen threshold.
  arXiv  Detail & Related papers  (2024-07-10T09:16:14Z)
• LRP-QViT: Mixed-Precision Vision Transformer Quantization via Layer-wise Relevance Propagation [0.0]
  We introduce LRP-QViT, an explainability-based method for assigning mixed-precision bit allocations to different layers based on their importance during classification. Our experimental findings demonstrate that both our fixed-bit and mixed-bit post-training quantization methods surpass existing models in the context of 4-bit and 6-bit quantization.
  arXiv  Detail & Related papers  (2024-01-20T14:53:19Z)
• Boosting Transformer's Robustness and Efficacy in PPG Signal Artifact Detection with Self-Supervised Learning [0.0]
  This study addresses the underutilization of abundant unlabeled data by employing self-supervised learning (SSL) to extract latent features from this data. Our experiments demonstrate that SSL significantly enhances the Transformer model's ability to learn representations. This approach holds promise for broader applications in PICU environments, where annotated data is often limited.
  arXiv  Detail & Related papers  (2024-01-02T04:00:48Z)
• The effect of data augmentation and 3D-CNN depth on Alzheimer's Disease detection [51.697248252191265]
  This work summarizes and strictly observes best practices regarding data handling, experimental design, and model evaluation. We focus on Alzheimer's Disease (AD) detection, which serves as a paradigmatic example of challenging problem in healthcare. Within this framework, we train predictive 15 models, considering three different data augmentation strategies and five distinct 3D CNN architectures.
  arXiv  Detail & Related papers  (2023-09-13T10:40:41Z)
• Hallucination Improves the Performance of Unsupervised Visual Representation Learning [9.504503675097137]
  We propose Hallucinator that could efficiently generate additional positive samples for further contrast. The Hallucinator is differentiable and creates new data in the feature space. Remarkably, we empirically prove that the proposed Hallucinator generalizes well to various contrastive learning models.
  arXiv  Detail & Related papers  (2023-07-22T21:15:56Z)
• ViT-Calibrator: Decision Stream Calibration for Vision Transformer [49.60474757318486]
  We propose a new paradigm dubbed Decision Stream that boosts the performance of general Vision Transformers. We shed light on the information propagation mechanism in the learning procedure by exploring the correlation between different tokens and the relevance coefficient of multiple dimensions.
  arXiv  Detail & Related papers  (2023-04-10T02:40:24Z)
• Vision Transformers are Robust Learners [65.91359312429147]
  We study the robustness of the Vision Transformer (ViT) against common corruptions and perturbations, distribution shifts, and natural adversarial examples. We present analyses that provide both quantitative and qualitative indications to explain why ViTs are indeed more robust learners.
  arXiv  Detail & Related papers  (2021-05-17T02:39:22Z)
• Improving Calibration for Long-Tailed Recognition [68.32848696795519]
  We propose two methods to improve calibration and performance in such scenarios. For dataset bias due to different samplers, we propose shifted batch normalization. Our proposed methods set new records on multiple popular long-tailed recognition benchmark datasets.
  arXiv  Detail & Related papers  (2021-04-01T13:55:21Z)
• Adversarial Feature Augmentation and Normalization for Visual Recognition [109.6834687220478]
  Recent advances in computer vision take advantage of adversarial data augmentation to ameliorate the generalization ability of classification models. Here, we present an effective and efficient alternative that advocates adversarial augmentation on intermediate feature embeddings. We validate the proposed approach across diverse visual recognition tasks with representative backbone networks.
  arXiv  Detail & Related papers  (2021-03-22T20:36:34Z)
• Dense Contrastive Learning for Self-Supervised Visual Pre-Training [102.15325936477362]
  We present dense contrastive learning, which implements self-supervised learning by optimizing a pairwise contrastive (dis)similarity loss at the pixel level between two views of input images. Compared to the baseline method MoCo-v2, our method introduces negligible computation overhead (only 1% slower)
  arXiv  Detail & Related papers  (2020-11-18T08:42:32Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.