Rethinking Pruning for Vision-Language Models: Strategies for Effective Sparsity and Performance Restoration

• URL: http://arxiv.org/abs/2404.02424v2
• Date: Mon, 24 Jun 2024 21:37:45 GMT
• Title: Rethinking Pruning for Vision-Language Models: Strategies for Effective Sparsity and Performance Restoration
• Authors: Shwai He, Ang Li, Tianlong Chen,
• Abstract summary: Vision-Language Models (VLMs) integrate information from multiple modalities and have shown remarkable success across various tasks. This study addresses two key questions: how to distribute sparsity across different modality-specific models, and how to restore the performance of pruned sparse VLMs.
• Score: 41.714059733079374
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Vision-Language Models (VLMs) integrate information from multiple modalities and have shown remarkable success across various tasks. However, deploying large-scale VLMs in resource-constrained scenarios is challenging. Pruning followed by finetuning offers a potential solution but remains underexplored for VLMs. This study addresses two key questions: how to distribute sparsity across different modality-specific models, and how to restore the performance of pruned sparse VLMs. Our preliminary studies identified two effective pruning settings: applying the same sparsity to both vision and language models, and pruning only the language models. While LoRA finetuning aims to restore sparse models, it faces challenges due to incompatibility with sparse models, disrupting the pruned sparsity. To overcome these issues, we propose SparseLoRA, which applies sparsity directly to LoRA weights. Our experimental results demonstrate significant improvements, including an 11.3\% boost under 2:4 sparsity and a 47.6\% enhancement under unstructured 70\% sparsity. Code is released at: \url{https://github.com/Shwai-He/VLM-Compression}.

Related papers

• LLaVE: Large Language and Vision Embedding Models with Hardness-Weighted Contrastive Learning [76.82159851648711]
  We propose a framework that dynamically improves the embedding model's representation learning for negative pairs. LLaVE establishes stronger baselines that achieve state-of-the-art (SOTA) performance. LLaVE can generalize to text-video retrieval tasks in a zero-shot manner and achieve strong performance.
  arXiv  Detail & Related papers  (2025-03-04T10:21:57Z)
• LoRA-GGPO: Mitigating Double Descent in LoRA Fine-Tuning via Gradient-Guided Perturbation Optimization [12.504723188498]
  Large Language Models (LLMs) have achieved remarkable success in natural language processing. Low-Rank Adaptation (LoRA) has emerged as a practical solution by approximating parameter updates with low-rank matrices. LoRA-GGPO is a novel method that leverages gradient and weight norms to generate targeted perturbations.
  arXiv  Detail & Related papers  (2025-02-20T13:14:41Z)
• Adaptive Pruning for Large Language Models with Structural Importance Awareness [66.2690963378878]
  Large language models (LLMs) have significantly improved language understanding and generation capabilities. LLMs are difficult to deploy on resource-constrained edge devices due to their high computational and storage resource demands. We propose structurally-aware adaptive pruning (SAAP) to significantly reduce the computational and memory costs while maintaining model performance.
  arXiv  Detail & Related papers  (2024-12-19T18:08:04Z)
• LLAVADI: What Matters For Multimodal Large Language Models Distillation [77.73964744238519]
  In this work, we do not propose a new efficient model structure or train small-scale MLLMs from scratch. Our studies involve training strategies, model choices, and distillation algorithms in the knowledge distillation process. By evaluating different benchmarks and proper strategy, even a 2.7B small-scale model can perform on par with larger models with 7B or 13B parameters.
  arXiv  Detail & Related papers  (2024-07-28T06:10:47Z)
• RAVEN: Multitask Retrieval Augmented Vision-Language Learning [5.1583788731239455]
  The scaling of large language models to encode all the world's knowledge is unsustainable and has exacerbated resource barriers. Retrieval-Augmented Generation (RAG) presents a potential solution, yet its application to vision-language models (VLMs) is under explored. This paper introduces RAVEN, a retrieval augmented VLM framework that enhances base VLMs through efficient, task specific fine-tuning.
  arXiv  Detail & Related papers  (2024-06-27T13:08:35Z)
• Advancing the Robustness of Large Language Models through Self-Denoised Smoothing [50.54276872204319]
  Large language models (LLMs) have achieved significant success, but their vulnerability to adversarial perturbations has raised considerable concerns. We propose to leverage the multitasking nature of LLMs to first denoise the noisy inputs and then to make predictions based on these denoised versions. Unlike previous denoised smoothing techniques in computer vision, which require training a separate model to enhance the robustness of LLMs, our method offers significantly better efficiency and flexibility.
  arXiv  Detail & Related papers  (2024-04-18T15:47:00Z)
• Prismatic VLMs: Investigating the Design Space of Visually-Conditioned Language Models [73.40350756742231]
  Visually-conditioned language models (VLMs) have seen growing adoption in applications such as visual dialogue, scene understanding, and robotic task planning. Despite the volume of new releases, key design decisions around image preprocessing, architecture, and optimization are under-explored.
  arXiv  Detail & Related papers  (2024-02-12T18:21:14Z)
• Investigating Training Strategies and Model Robustness of Low-Rank Adaptation for Language Modeling in Speech Recognition [27.515920408920216]
  Low-rank adaptation (LoRA) with frozen pretrained language models (PLMs) is a resource-efficient modeling approach for memory-constrained hardware. In this study, we explore how to enhance model performance by introducing various LoRA training strategies. To further characterize the stability of LoRA-based second-pass speech recognition models, we examine against input perturbations.
  arXiv  Detail & Related papers  (2024-01-19T01:30:16Z)
• VILA: On Pre-training for Visual Language Models [74.08039416548209]
  We study the design options for VLM pre-training through step-by-step controllable comparisons. We build VILA, a Visual Language model family that consistently outperforms the state-of-the-art models.
  arXiv  Detail & Related papers  (2023-12-12T18:58:18Z)
• LoRAShear: Efficient Large Language Model Structured Pruning and Knowledge Recovery [42.018731237153446]
  Large Language Models (LLMs) have transformed the landscape of artificial intelligence. We introduce LoRAShear, a novel efficient approach to structurally prune LLMs and recover knowledge. LoRAShear effectively reduced footprint of LLMs by 20% with only 1.0% performance degradation.
  arXiv  Detail & Related papers  (2023-10-24T00:47:26Z)
• Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruning [52.29522018586365]
  We study structured pruning as an effective means to develop smaller LLMs from pre-trained, larger models. Our approach employs two key techniques: (1) targeted structured pruning, which prunes a larger model to a specified target shape by removing layers, heads, and intermediate and hidden dimensions in an end-to-end manner, and (2) dynamic batch loading, which dynamically updates the composition of sampled data in each training batch based on varying losses across different domains.
  arXiv  Detail & Related papers  (2023-10-10T15:13:30Z)
• Less is More -- Towards parsimonious multi-task models using structured sparsity [4.874780144224057]
  This work focuses on creating sparse models optimized for multiple tasks with fewer parameters. We introduce channel-wise l1/l2 group sparsity in the shared convolutional layers parameters (or weights) of the multi-task learning model. We analyzed the results of group sparsity in both single-task and multi-task settings on two widely-used Multi-Task Learning (MTL) datasets.
  arXiv  Detail & Related papers  (2023-08-23T13:09:03Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.