LLaVA-FA: Learning Fourier Approximation for Compressing Large Multimodal Models

• URL: http://arxiv.org/abs/2602.00135v1
• Date: Wed, 28 Jan 2026 09:39:10 GMT
• Title: LLaVA-FA: Learning Fourier Approximation for Compressing Large Multimodal Models
• Authors: Pengcheng Zheng, Chaoning Zhang, Jiarong Mo, GuoHui Li, Jiaquan Zhang, Jiahao Zhang, Sihan Cao, Sheng Zheng, Caiyan Qin, Guoqing Wang, Yang Yang,
• Abstract summary: Large multimodal models (LMMs) have achieved impressive performance on various vision-language tasks.<n>Existing compression methods often decouple low-rank decomposition and quantization, leading to compounded reconstruction errors.<n>We propose LLaVA-FA, a novel efficient LMM that performs joint low-rank plus quantization approximation in the frequency domain.
• Score: 23.184422544852108
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Large multimodal models (LMMs) have achieved impressive performance on various vision-language tasks, but their substantial computational and memory costs hinder their practical deployment. Existing compression methods often decouple low-rank decomposition and quantization, leading to compounded reconstruction errors, especially in multimodal architectures with cross-modal redundancy. To address this issue, we propose LLaVA-FA, a novel efficient LMM that performs joint low-rank plus quantization approximation in the frequency domain. By leveraging the de-correlation and conjugate symmetry properties of Fourier transform, LLaVA-FA achieves more compact and accurate weight representations. Furthermore, we introduce PolarQuant, a polar-coordinate quantization method tailored for complex matrices, and an optional diagonal calibration (ODC) scheme that eliminates the need for large-scale calibration data. Extensive experimental results demonstrate that our proposed LLaVA-FA outperforms existing efficient multimodal models across multiple benchmarks while maintaining minimal activated parameters and low computational costs, validating its effectiveness as a powerful solution for compressing LMMs.

Related papers

• Beyond Real Weights: Hypercomplex Representations for Stable Quantization [6.708338010963415]
  Multimodal language models (MLLMs) require large parameter capacity to align high-dimensional visual features with linguistic representations.<n>We introduce a progressive re parameterization strategy that compresses these models by gradually replacing dense feed-forward network blocks.<n>A residual schedule, together with lightweight reconstruction and knowledge distillation losses, ensures that the PHM modules inherit the functional behavior of their dense counterparts during training.
  arXiv  Detail & Related papers  (2025-12-09T12:10:57Z)
• SingleQuant: Efficient Quantization of Large Language Models in a Single Pass [17.504732263852876]
  We propose SingleQuant, a single-pass quantization framework that decouples from quantization truncation.<n>Specifically, SingleQuant constructs Alignment Rotation Transformation (ART) and Uniformity Rotation Transformation (URT) targeting distinct activation outliers.<n> Experimental results demonstrate SingleQuant's superiority over the selected baselines across diverse tasks.
  arXiv  Detail & Related papers  (2025-11-27T10:46:39Z)
• QuantVSR: Low-Bit Post-Training Quantization for Real-World Video Super-Resolution [53.13952833016505]
  We propose a low-bit quantization model for real-world video super-resolution (VSR)<n>We use a calibration dataset to measure both spatial and temporal complexity for each layer.<n>We refine the FP and low-bit branches to achieve simultaneous optimization.
  arXiv  Detail & Related papers  (2025-08-06T14:35:59Z)
• Highly Efficient and Effective LLMs with Multi-Boolean Architectures [5.346271362401715]
  Weight binarization has emerged as a promising strategy to reduce the complexity of large language models (LLMs)<n>Existing approaches fall into post-training binarization, which is simple but causes severe performance loss, and training-aware methods, which depend on full-precision latent weights, adding complexity and limiting efficiency.<n>We propose a novel framework that represents LLMs with multi- kernel Boolean parameters and, for the first time, enables direct finetuning LMMs in the Boolean domain, eliminating the need for latent weights.
  arXiv  Detail & Related papers  (2025-05-28T19:40:34Z)
• LatentLLM: Attention-Aware Joint Tensor Compression [50.33925662486034]
  Large language models (LLMs) and large multi-modal models (LMMs) require a massive amount of computational and memory resources.<n>We propose a new framework to convert such LLMs/LMMs into a reduced-dimension latent structure.
  arXiv  Detail & Related papers  (2025-05-23T22:39:54Z)
• RoSTE: An Efficient Quantization-Aware Supervised Fine-Tuning Approach for Large Language Models [53.571195477043496]
  We propose an algorithm named Rotated Straight-Through-Estimator (RoSTE)<n>RoSTE combines quantization-aware supervised fine-tuning (QA-SFT) with an adaptive rotation strategy to reduce activation outliers.<n>Our findings reveal that the prediction error is directly proportional to the quantization error of the converged weights, which can be effectively managed through an optimized rotation configuration.
  arXiv  Detail & Related papers  (2025-02-13T06:44:33Z)
• CoMMIT: Coordinated Multimodal Instruction Tuning [90.1532838391285]
  Multimodal large language models (MLLMs) generally involve cooperative learning between a backbone LLM and a feature encoder of non-text input modalities.<n>In this paper, we analyze the MLLM instruction tuning from both theoretical and empirical perspectives.<n>We propose a Multimodal Balance Coefficient that enables quantitative measurement of the balance of learning.
  arXiv  Detail & Related papers  (2024-07-29T23:18:55Z)
• Pruning Large Language Models to Intra-module Low-rank Architecture with Transitional Activations [21.229296254354878]
  We introduce a task-agnostic structured pruning approach coupled with a compact Transformer architecture design. The proposed approach, named TransAct, reduces transitional activations inside multi-head attention (MHA) and multi-layer perceptron (MLP) modules. Results verify the optimality of our approach at high compression with respect to both efficiency and performance.
  arXiv  Detail & Related papers  (2024-07-08T07:45:38Z)
• Tender: Accelerating Large Language Models via Tensor Decomposition and Runtime Requantization [0.6445087473595953]
  Large language models (LLMs) demonstrate outstanding performance in various tasks in machine learning. deploying LLM inference poses challenges due to the high compute and memory requirements. We present Tender, an algorithm-hardware co-design solution that enables efficient deployment of LLM inference at low precision.
  arXiv  Detail & Related papers  (2024-06-16T09:51:55Z)
• Delta-CoMe: Training-Free Delta-Compression with Mixed-Precision for Large Language Models [79.46938238953916]
  Fine-tuning large language models (LLMs) to diverse applications is crucial to meet complex demands. Recent studies suggest decomposing a fine-tuned LLM into a base model and corresponding delta weights, which are then compressed using low-rank or low-bit approaches to reduce costs. In this work, we observe that existing low-rank and low-bit compression methods can significantly harm the model performance for task-specific fine-tuned LLMs.
  arXiv  Detail & Related papers  (2024-06-13T07:57:27Z)
• DB-LLM: Accurate Dual-Binarization for Efficient LLMs [83.70686728471547]
  Large language models (LLMs) have significantly advanced the field of natural language processing. Existing ultra-low-bit quantization always causes severe accuracy drops. We propose a novel Dual-Binarization method for LLMs, namely DB-LLM.
  arXiv  Detail & Related papers  (2024-02-19T09:04:30Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.