Frequency Matters: Fast Model-Agnostic Data Curation for Pruning and Quantization

• URL: http://arxiv.org/abs/2603.16105v1
• Date: Tue, 17 Mar 2026 04:12:08 GMT
• Title: Frequency Matters: Fast Model-Agnostic Data Curation for Pruning and Quantization
• Abstract summary: Post-training model compression is essential for enhancing the portability of Large Language Models (LLMs)<n>We introduce texttttextbfZipCal, a model-agnostic data curation strategy that maximizes lexical diversity based on Zipfian power laws.
• Score: 7.283573413303817
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Post-training model compression is essential for enhancing the portability of Large Language Models (LLMs) while preserving their performance. While several compression approaches have been proposed, less emphasis has been placed on selecting the most suitable set of data (the so-called \emph{calibration data}) for finding the compressed model configuration. The choice of calibration data is a critical step in preserving model capabilities both intra- and inter-tasks. In this work, we address the challenge of identifying high-performance calibration sets for both pruning and quantization by analyzing intrinsic data properties rather than model-specific signals. We introduce \texttt{\textbf{ZipCal}}, a model-agnostic data curation strategy that maximizes lexical diversity based on Zipfian power laws. Experiments demonstrate that our method consistently outperforms standard uniform random sampling across various pruning benchmarks. Notably, it also performs on par, in terms of downstream performance, with a state-of-the-art method that relies on model perplexity. The latter becomes prohibitively expensive at large-scale models and datasets, while \texttt{\textbf{ZipCal}} is on average $\sim$240$\times$ faster due to its tractable linear complexity\footnote{We make the code and the experiments available at https://anonymous.4open.science/r/zipcal-71CD/.}.

Related papers

• Learning Multi-Indicator Weights for Data Selection: A Joint Task-Model Adaptation Framework with Efficient Proxies [50.39041754816285]
  We propose a framework for learning multi-validation weights that jointly adapts data selection to both the downstream task and the specific model.<n>Our approach achieves performance comparable to or exceeding full-dataset tuning while using only 30% of the training samples on GSM8K.
  arXiv  Detail & Related papers  (2026-05-10T17:30:16Z)
• Float8@2bits: Entropy Coding Enables Data-Free Model Compression [4.775539058503235]
  We introduce EntQuant, the first framework to unite the advantages of different post-training compression regimes.<n>Our method decouples numerical precision from storage cost via entropy coding, compressing a 70B parameter model in less than 30 minutes.<n>We demonstrate that EntQuant does not only achieve state-of-the-art results on standard evaluation sets and models, but also retains functional performance on more complex benchmarks.
  arXiv  Detail & Related papers  (2026-01-30T10:08:15Z)
• DISCO: Diversifying Sample Condensation for Efficient Model Evaluation [59.01400190971061]
  Costly evaluation reduces inclusivity, slows the cycle of innovation, and worsens environmental impact.<n>We argue that promoting diversity among samples is not essential; what matters is to select samples thatmaximise diversity in model responses.<n>Our method, $textbfDiversifying Sample Condensation (DISCO)$, selects the top-k samples with the greatest model disagreements.
  arXiv  Detail & Related papers  (2025-10-09T08:53:59Z)
• Diffusion-Driven High-Dimensional Variable Selection [6.993247097440294]
  We propose a resample-aggregate framework that exploits diffusion models' ability to generate high-fidelity synthetic data.<n>We show that the proposed method is selection consistent under mild assumptions.<n>Our method advances variable selection methodology and broadens the toolkit for interpretable, statistically rigorous analysis.
  arXiv  Detail & Related papers  (2025-08-19T14:54:20Z)
• Self-calibration for Language Model Quantization and Pruning [38.00221764773372]
  Quantization and pruning methods require calibration data, a small set of unlabeled examples.<n>In this paper, we propose self-calibration as a solution.<n>Our approach requires no external data, instead leveraging the model itself to generate synthetic calibration data.
  arXiv  Detail & Related papers  (2024-10-22T16:50:00Z)
• Low-Resource Crop Classification from Multi-Spectral Time Series Using Lossless Compressors [6.379065975644869]
  Deep learning has significantly improved the accuracy of crop classification using multispectral temporal data. In low-resource situations with fewer labeled samples, deep learning models perform poorly due to insufficient data. We propose a non-training alternative to deep learning models, aiming to address these situations.
  arXiv  Detail & Related papers  (2024-05-28T12:28:12Z)
• Latent Semantic Consensus For Deterministic Geometric Model Fitting [109.44565542031384]
  We propose an effective method called Latent Semantic Consensus (LSC) LSC formulates the model fitting problem into two latent semantic spaces based on data points and model hypotheses. LSC is able to provide consistent and reliable solutions within only a few milliseconds for general multi-structural model fitting.
  arXiv  Detail & Related papers  (2024-03-11T05:35:38Z)
• Fed-CVLC: Compressing Federated Learning Communications with Variable-Length Codes [54.18186259484828]
  In Federated Learning (FL) paradigm, a parameter server (PS) concurrently communicates with distributed participating clients for model collection, update aggregation, and model distribution over multiple rounds. We show strong evidences that variable-length is beneficial for compression in FL. We present Fed-CVLC (Federated Learning Compression with Variable-Length Codes), which fine-tunes the code length in response to the dynamics of model updates.
  arXiv  Detail & Related papers  (2024-02-06T07:25:21Z)
• TFMQ-DM: Temporal Feature Maintenance Quantization for Diffusion Models [52.454274602380124]
  Diffusion models heavily depend on the time-step $t$ to achieve satisfactory multi-round denoising. We propose a Temporal Feature Maintenance Quantization (TFMQ) framework building upon a Temporal Information Block. Powered by the pioneering block design, we devise temporal information aware reconstruction (TIAR) and finite set calibration (FSC) to align the full-precision temporal features.
  arXiv  Detail & Related papers  (2023-11-27T12:59:52Z)
• Improved Distribution Matching for Dataset Condensation [91.55972945798531]
  We propose a novel dataset condensation method based on distribution matching. Our simple yet effective method outperforms most previous optimization-oriented methods with much fewer computational resources.
  arXiv  Detail & Related papers  (2023-07-19T04:07:33Z)
• Infor-Coef: Information Bottleneck-based Dynamic Token Downsampling for Compact and Efficient language model [0.0]
  Excessive overhead leads to large latency and computational costs. We propose a model accelaration approaches for large language models. Our model achieves an 18x FLOPs speedup with an accuracy degradation of less than 8% compared to BERT.
  arXiv  Detail & Related papers  (2023-05-21T13:30:56Z)
• ClusterQ: Semantic Feature Distribution Alignment for Data-Free Quantization [111.12063632743013]
  We propose a new and effective data-free quantization method termed ClusterQ. To obtain high inter-class separability of semantic features, we cluster and align the feature distribution statistics. We also incorporate the intra-class variance to solve class-wise mode collapse.
  arXiv  Detail & Related papers  (2022-04-30T06:58:56Z)
• Model Compression for Dynamic Forecast Combination [9.281199058905017]
  We show that compressing dynamic forecasting ensembles into an individual model leads to a comparable predictive performance. We also show that the compressed individual model with best average rank is a rule-based regression model.
  arXiv  Detail & Related papers  (2021-04-05T09:55:35Z)
• On the Discrepancy between Density Estimation and Sequence Generation [92.70116082182076]
  log-likelihood is highly correlated with BLEU when we consider models within the same family. We observe no correlation between rankings of models across different families.
  arXiv  Detail & Related papers  (2020-02-17T20:13:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.