LogFold: Compressing Logs with Structured Tokens and Hybrid Encoding

• URL: http://arxiv.org/abs/2603.20618v1
• Date: Sat, 21 Mar 2026 03:27:37 GMT
• Title: LogFold: Compressing Logs with Structured Tokens and Hybrid Encoding
• Abstract summary: We introduce LogFold, a novel log compression method consisting of four components.<n>A token analyzer classifies tokens as structured, unstructured, or static types; a processor mines recurring patterns within structured tokens based on their skeletons; a hybrid encoder tailors data representation according to token types.<n>Experiments on 16 public log datasets demonstrate that LogFold surpasses state-of-the-art baselines, achieving average compression ratio improvements by 11.11%, with a compression speed of 9.842 MB/s.
• Score: 36.603581308724465
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Logs are essential for diagnosing failures and conducting retrospective studies, leading many software organizations to retain log messages for a long time. Nevertheless, the volume of generated log data grows rapidly as software systems grow, necessitating an effective compression method. Apart from general-purpose compressors (e.g., Gzip, Bzip2), many recent studies developed log-specific compression algorithms, but they offer suboptimal performance because of (1) overlooking redundancies within certain complex tokens, and (2) lacking a fine-grained encoding strategy for diverse token types. This work uncovers a new redundancy pattern in structured tokens and proposes a new type-aware encoding strategy to improve log compression. Building on this insight, we introduce LogFold, a novel log compression method consisting of four components: a token analyzer to classifies tokens as structured, unstructured, or static types; a processor that mines recurring patterns within structured tokens based on their delimiter skeletons; a hybrid encoder that tailors data representation according to token types; and a packer that compresses the output into an archive file. Extensive experiments on 16 public log datasets demonstrate that LogFold surpasses state-of-the-art baselines, achieving average compression ratio improvements by 11.11%, with a compression speed of 9.842 MB/s. Ablation studies further indicate the importance of each component. We also conduct sensitivity analyses to verify LogFold's robustness and stability across various internal settings.

Related papers

• Forget, Then Recall: Learnable Compression and Selective Unfolding via Gist Sparse Attention [7.694795562178033]
  Scaling large language models to long contexts is challenging due to the quadratic computational cost of full attention.<n>We introduce selective unfolding via GSA, which first compresses the context into gist tokens, then selects the most relevant gists.<n>This yields a simple coarse-to-fine mechanism that combines compact global representations with targeted access to fine-leaved evidence.
  arXiv  Detail & Related papers  (2026-04-22T04:22:32Z)
• Arbitrary Ratio Feature Compression via Next Token Prediction [52.10426317889982]
  Arbitrary Ratio Feature Compression (ARFC) framework supports any compression ratio with a single model.<n>ARC is an auto-regressive model that performs compression via next-gressive prediction.<n>MoS module refines the compressed tokens by utilizing multiple compression results.<n>ERGC is integrated into the training process to preserve semantic and structural relationships during compression.
  arXiv  Detail & Related papers  (2026-02-12T02:38:57Z)
• LogPrism: Unifying Structure and Variable Encoding for Effective Log Compression [41.26407915252864]
  "parse-then-compress" paradigm limits effectiveness by treating log parsing and compression as isolated objectives.<n>We propose LogPrism, a framework that bridges the gap via unified redundancy encoding.<n> experiments on 16 benchmark datasets confirm that LogPrism establishes a new state-of-the-art.
  arXiv  Detail & Related papers  (2026-01-24T15:12:31Z)
• DeLog: An Efficient Log Compression Framework with Pattern Signature Synthesis [30.690382753027137]
  A novel log compressor implements a Pattern Signature Synthesis mechanism to achieve efficient pattern-based grouping.<n>On 16 public and 10 production datasets, DeLog achieves state-of-the-art compression ratio and speed.
  arXiv  Detail & Related papers  (2026-01-21T15:26:09Z)
• Transformers from Compressed Representations [74.48571451824569]
  TEMPEST (TransformErs froM comPressed rEpreSenTations) is a method that exploits the inherent byte-stream structure of compressed files to design an effective tokenization and encoding strategy.<n>Our proposal substantially reduces the number of tokens required for semantic classification, thereby lowering both computational complexity and memory usage.
  arXiv  Detail & Related papers  (2025-10-26T13:48:03Z)
• UniGist: Towards General and Hardware-aligned Sequence-level Long Context Compression [86.33995240043936]
  UniGist is a sequence-level long-context compression framework for large language models.<n>It efficiently preserves context information by replacing raw tokens with special compression tokens (gists) in a fine-grained manner.<n>Our scheme also supports flexible inference by allowing the actual removal of compressed tokens, resulting in real-time memory savings.
  arXiv  Detail & Related papers  (2025-09-19T08:47:37Z)
• Compressive Meta-Learning [49.300635370079874]
  Compressive learning is a framework that enables efficient processing by using random, non-linear features.<n>We propose a framework that meta-learns both the encoding and decoding stages of compressive learning methods.<n>We explore multiple applications -- including neural network-based compressive PCA, compressive ridge regression, compressive k-means, and autoencoders.
  arXiv  Detail & Related papers  (2025-08-14T22:08:06Z)
• Lightweight Correlation-Aware Table Compression [58.50312417249682]
  $texttVirtual$ is a framework that integrates seamlessly with existing open formats. Experiments on data-gov datasets show that $texttVirtual$ reduces file sizes by up to 40% compared to Apache Parquet.
  arXiv  Detail & Related papers  (2024-10-17T22:28:07Z)
• Unlocking the Power of Numbers: Log Compression via Numeric Token Parsing [18.419996813182163]
  We propose Denum, a simple, general log compressor with high compression ratio and speed. Specifically, Denum contains a Token Parsing module, which extracts all numeric tokens and applies tailored processing methods. It has been evaluated on 16 log datasets and it achieves an 8.7% average compression ratio and 2.6x faster average compression speed.
  arXiv  Detail & Related papers  (2024-08-11T12:30:41Z)
• LogShrink: Effective Log Compression by Leveraging Commonality and Variability of Log Data [35.5712445690333]
  We propose LogShrink, a novel and effective log compression method by leveraging commonality and variability of log data. An analyzer based on longest common subsequence and entropy techniques is proposed to identify the latent commonality and variability in log messages. The key idea behind this is that the commonality and variability can be exploited to shrink log data with a shorter representation.
  arXiv  Detail & Related papers  (2023-09-18T04:27:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.