PMKLC: Parallel Multi-Knowledge Learning-based Lossless Compression for Large-Scale Genomics Database

• URL: http://arxiv.org/abs/2507.12805v1
• Date: Thu, 17 Jul 2025 05:46:08 GMT
• Title: PMKLC: Parallel Multi-Knowledge Learning-based Lossless Compression for Large-Scale Genomics Database
• Abstract summary: We propose a novel underlineParallel underlineMulti-underlineKnowledge underlineLearning-based underlineCompressor (PMKLC) with four crucial designs.<n>We benchmark PMKLC-S/M and 14 baselines on 15 real-world datasets with different species and data sizes.<n>PMKLC-S/M achieve the average compression ratio improvement up to 73.609% and 73.480%, the average throughput improvement up to 3.036
• Score: 9.687284031748792
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Learning-based lossless compressors play a crucial role in large-scale genomic database backup, storage, transmission, and management. However, their 1) inadequate compression ratio, 2) low compression \& decompression throughput, and 3) poor compression robustness limit their widespread adoption and application in both industry and academia. To solve those challenges, we propose a novel \underline{P}arallel \underline{M}ulti-\underline{K}nowledge \underline{L}earning-based \underline{C}ompressor (PMKLC) with four crucial designs: 1) We propose an automated multi-knowledge learning-based compression framework as compressors' backbone to enhance compression ratio and robustness; 2) we design a GPU-accelerated ($s$,$k$)-mer encoder to optimize compression throughput and computing resource usage; 3) we introduce data block partitioning and Step-wise Model Passing (SMP) mechanisms for parallel acceleration; 4) We design two compression modes PMKLC-S and PMKLC-M to meet the complex application scenarios, where the former runs on a resource-constrained single GPU and the latter is multi-GPU accelerated. We benchmark PMKLC-S/M and 14 baselines (7 traditional and 7 leaning-based) on 15 real-world datasets with different species and data sizes. Compared to baselines on the testing datasets, PMKLC-S/M achieve the average compression ratio improvement up to 73.609\% and 73.480\%, the average throughput improvement up to 3.036$\times$ and 10.710$\times$, respectively. Besides, PMKLC-S/M also achieve the best robustness and competitive memory cost, indicating its greater stability against datasets with different probability distribution perturbations, and its strong ability to run on memory-constrained devices.

Related papers

• Benchmarking Composable Compression Techniques in Mixture-of-Experts LLMs [50.677556944160216]
  MoEXBench is a systematic benchmark for evaluating composable MoE compression as an end-to-end deployment workflow.<n>It measures composable-compression quality, workload and architecture, pruning/quantization/KV cache sensitivity, and deployment efficiency on commodity hardware.
  arXiv  Detail & Related papers  (2026-08-22T00:01:18Z)
• End-to-End Context Compression at Scale [81.70601323130997]
  Long-context language model inference is bottlenecked by memory, as the KV cache grows with context length.<n>Recent techniques to compress the KV cache fall short: they either degrade model quality substantially or require considerable time and compute to compress a single long prompt.<n>Existing approaches are not competitive with KV cache compression on the accuracy-efficiency frontier.
  arXiv  Detail & Related papers  (2026-06-08T15:43:16Z)
• AHC: Meta-Learned Adaptive Compression for Continual Object Detection on Memory-Constrained Microcontrollers [0.14504054468850666]
  Continual object detection on microcontrollers (MCUs) with under 100KB memory requires efficient feature compression.<n>We introduce Adaptive Hierarchical Compression (AHC), a meta-learning framework featuring three key innovations.<n>AHC enables practical continual detection within a 100KB replay budget, achieving competitive accuracy through mean-pooled compressed feature replay.
  arXiv  Detail & Related papers  (2026-02-24T07:06:51Z)
• Arbitrary Ratio Feature Compression via Next Token Prediction [52.10426317889982]
  Arbitrary Ratio Feature Compression (ARFC) framework supports any compression ratio with a single model.<n>ARC is an auto-regressive model that performs compression via next-gressive prediction.<n>MoS module refines the compressed tokens by utilizing multiple compression results.<n>ERGC is integrated into the training process to preserve semantic and structural relationships during compression.
  arXiv  Detail & Related papers  (2026-02-12T02:38:57Z)
• AgentGC: Evolutionary Learning-based Lossless Compression for Genomics Data with LLM-driven Multiple Agent [25.73008424563264]
  AgentGC is the first evolutionary agent-based Genomics Data (GD) management system.<n>AgentGC integrates user-friendly interface, cognitive layer, and compression layer.<n>Compared with 14 baselines on 9 datasets, the average compression ratios gains are 16.66%, 16.11%, and 16.33%, respectively.
  arXiv  Detail & Related papers  (2026-01-20T03:29:45Z)
• Cmprsr: Abstractive Token-Level Question-Agnostic Prompt Compressor [36.57824786347272]
  We present the first comprehensive LLM-as-a-compressor benchmark spanning 25 open- and closed-source models.<n>We improve the performance of the best overall vanilla compressor, with Textgrad-based compression meta-prompt optimization.<n>We call the resulting model Cmprsr and demonstrate its superiority over both extractive and vanilla abstractive compression.
  arXiv  Detail & Related papers  (2025-11-15T16:28:03Z)
• Compressing Many-Shots in In-Context Learning [61.231471139896506]
  We study an approach to improve the memory and computational efficiency of ICL inference by compressing the many-shot prompts.<n>We first show that existing prompt compression methods are ineffective for many-shot compression.<n>We propose MemCom, a layer-wise compression method.
  arXiv  Detail & Related papers  (2025-10-17T16:57:42Z)
• ReCalKV: Low-Rank KV Cache Compression via Head Reordering and Offline Calibration [81.81027217759433]
  Large language models (LLMs) are often constrained by the excessive memory required to store the Key-Value ( KV) cache.<n>Recent methods have explored reducing the hidden dimensions of the KV cache, but many introduce additional computation through projection layers.<n>We propose ReCalKV, a post-training KV cache compression method that reduces the hidden dimensions of the KV cache.
  arXiv  Detail & Related papers  (2025-05-30T08:49:27Z)
• Breaking the Compression Ceiling: Data-Free Pipeline for Ultra-Efficient Delta Compression [53.08742231761896]
  UltraDelta is a data-free delta compression pipeline that achieves both ultra-high compression and strong performance.<n>UltraDelta is designed to minimize redundancy, maximize information, and stabilize performance across inter-layer, intra-layer, and global dimensions.
  arXiv  Detail & Related papers  (2025-05-19T10:37:22Z)
• UniPCGC: Towards Practical Point Cloud Geometry Compression via an Efficient Unified Approach [4.754973569457509]
  We propose an efficient unified point cloud geometry compression framework, dubbed as UniPCGC.<n>It supports lossy compression, lossless compression, variable rate and variable complexity.<n>Our method achieves a compression ratio (CR) gain of 8.1% on lossless compression, and a Bjontegaard Delta Rate (BD-Rate) gain of 14.02% on lossy compression.
  arXiv  Detail & Related papers  (2025-03-24T10:51:28Z)
• A Universal Framework for Compressing Embeddings in CTR Prediction [68.27582084015044]
  We introduce a Model-agnostic Embedding Compression (MEC) framework that compresses embedding tables by quantizing pre-trained embeddings.<n>Our approach consists of two stages: first, we apply popularity-weighted regularization to balance code distribution between high- and low-frequency features.<n> Experiments on three datasets reveal that our method reduces memory usage by over 50x while maintaining or improving recommendation performance.
  arXiv  Detail & Related papers  (2025-02-21T10:12:34Z)
• LoRC: Low-Rank Compression for LLMs KV Cache with a Progressive Compression Strategy [59.1298692559785]
  Key-Value ( KV) cache is crucial component in serving transformer-based autoregressive large language models (LLMs) Existing approaches to mitigate this issue include: (1) efficient attention variants integrated in upcycling stages; (2) KV cache compression at test time; and (3) KV cache compression at test time. We propose a low-rank approximation of KV weight matrices, allowing plug-in integration with existing transformer-based LLMs without model retraining. Our method is designed to function without model tuning in upcycling stages or task-specific profiling in test stages.
  arXiv  Detail & Related papers  (2024-10-04T03:10:53Z)
• MoDeGPT: Modular Decomposition for Large Language Model Compression [59.361006801465344]
  This paper introduces textbfModular bfDecomposition (MoDeGPT), a novel structured compression framework.<n>MoDeGPT partitions the Transformer block into modules comprised of matrix pairs and reduces the hidden dimensions.<n>Our experiments show MoDeGPT, without backward propagation, matches or surpasses previous structured compression methods.
  arXiv  Detail & Related papers  (2024-08-19T01:30:14Z)
• Long Context Compression with Activation Beacon [22.054232261437186]
  Activation Beacon is a plug-in module for transformer-based LLMs. It targets effective, efficient, and flexible compression of long contexts. It achieves a 2x acceleration in inference time and an 8x reduction of memory costs for KV cache.
  arXiv  Detail & Related papers  (2024-01-07T11:57:40Z)
• GraVAC: Adaptive Compression for Communication-Efficient Distributed DL Training [0.0]
  Distributed data-parallel (DDP) training improves overall application throughput as multiple devices train on a subset of data and aggregate updates to produce a globally shared model. GraVAC is a framework to dynamically adjust compression factor throughout training by evaluating model progress and assessing information loss associated with compression. As opposed to using a static compression factor, GraVAC reduces end-to-end training time for ResNet101, VGG16 and LSTM by 4.32x, 1.95x and 6.67x respectively.
  arXiv  Detail & Related papers  (2023-05-20T14:25:17Z)
• Exploring Autoencoder-based Error-bounded Compression for Scientific Data [14.724393511470225]
  We develop an error-bounded autoencoder-based framework in terms of the SZ model. We optimize the compression quality for the main stages in our designed AE-based error-bounded compression framework.
  arXiv  Detail & Related papers  (2021-05-25T07:53:32Z)
• Towards Compact CNNs via Collaborative Compression [166.86915086497433]
  We propose a Collaborative Compression scheme, which joints channel pruning and tensor decomposition to compress CNN models. We achieve 52.9% FLOPs reduction by removing 48.4% parameters on ResNet-50 with only a Top-1 accuracy drop of 0.56% on ImageNet 2012.
  arXiv  Detail & Related papers  (2021-05-24T12:07:38Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.