DeLog: An Efficient Log Compression Framework with Pattern Signature Synthesis

• URL: http://arxiv.org/abs/2601.15084v2
• Date: Thu, 22 Jan 2026 11:59:24 GMT
• Title: DeLog: An Efficient Log Compression Framework with Pattern Signature Synthesis
• Abstract summary: A novel log compressor implements a Pattern Signature Synthesis mechanism to achieve efficient pattern-based grouping.<n>On 16 public and 10 production datasets, DeLog achieves state-of-the-art compression ratio and speed.
• Score: 30.690382753027137
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Parser-based log compression, which separates static templates from dynamic variables, is a promising approach to exploit the unique structure of log data. However, its performance on complex production logs is often unsatisfactory. This performance gap coincides with a known degradation in the accuracy of its core log parsing component on such data, motivating our investigation into a foundational yet unverified question: does higher parsing accuracy necessarily lead to better compression ratio? To answer this, we conduct the first empirical study quantifying this relationship and find that a higher parsing accuracy does not guarantee a better compression ratio. Instead, our findings reveal that compression ratio is dictated by achieving effective pattern-based grouping and encoding, i.e., the partitioning of tokens into low entropy, highly compressible groups. Guided by this insight, we design DeLog, a novel log compressor that implements a Pattern Signature Synthesis mechanism to achieve efficient pattern-based grouping. On 16 public and 10 production datasets, DeLog achieves state-of-the-art compression ratio and speed.

Related papers

• Beyond Single-Dimensional Compression: The Compound Sparsity Frontier of Large Language Models [6.94102129809152]
  Large language models (LLMs) are often compressed through static parameter pruning or dynamic token-level computation.<n>We study a minimalist compound sparsity framework that first applies low-rank approximation and channel pruning to obtain a statically compressed backbone.
  arXiv  Detail & Related papers  (2026-06-29T01:33:30Z)
• Density-aware Soft Context Compression with Semi-Dynamic Compression Ratio [19.375601589555327]
  We introduce Semi-Dynamic Context Compression framework.<n>Our approach features a Discrete Ratio Selector, which predicts a compression target based on intrinsic information density.<n>Our framework consistently outperforms static baselines.
  arXiv  Detail & Related papers  (2026-03-26T21:36:55Z)
• LogFold: Compressing Logs with Structured Tokens and Hybrid Encoding [36.603581308724465]
  We introduce LogFold, a novel log compression method consisting of four components.<n>A token analyzer classifies tokens as structured, unstructured, or static types; a processor mines recurring patterns within structured tokens based on their skeletons; a hybrid encoder tailors data representation according to token types.<n>Experiments on 16 public log datasets demonstrate that LogFold surpasses state-of-the-art baselines, achieving average compression ratio improvements by 11.11%, with a compression speed of 9.842 MB/s.
  arXiv  Detail & Related papers  (2026-03-21T03:27:37Z)
• Arbitrary Ratio Feature Compression via Next Token Prediction [52.10426317889982]
  Arbitrary Ratio Feature Compression (ARFC) framework supports any compression ratio with a single model.<n>ARC is an auto-regressive model that performs compression via next-gressive prediction.<n>MoS module refines the compressed tokens by utilizing multiple compression results.<n>ERGC is integrated into the training process to preserve semantic and structural relationships during compression.
  arXiv  Detail & Related papers  (2026-02-12T02:38:57Z)
• Proxy Compression for Language Modeling [58.904023114033954]
  proxy compression is an alternative training scheme that preserves the efficiency benefits of compressed inputs.<n>Experiments on code language modeling demonstrate that proxy compression substantially improves training efficiency.<n>As model scale increases, proxy-trained models eventually match or rival tokenizer approaches.
  arXiv  Detail & Related papers  (2026-02-04T07:36:46Z)
• LogPrism: Unifying Structure and Variable Encoding for Effective Log Compression [41.26407915252864]
  "parse-then-compress" paradigm limits effectiveness by treating log parsing and compression as isolated objectives.<n>We propose LogPrism, a framework that bridges the gap via unified redundancy encoding.<n> experiments on 16 benchmark datasets confirm that LogPrism establishes a new state-of-the-art.
  arXiv  Detail & Related papers  (2026-01-24T15:12:31Z)
• Optimized Log Parsing with Syntactic Modifications [2.2917707112773593]
  We compare syntax- and semantic-based log parsing architectures.<n>We propose SynLog+, a template identification module that acts as the second phase in a two-phase log parsing architecture.
  arXiv  Detail & Related papers  (2025-10-30T17:59:22Z)
• Compressing Many-Shots in In-Context Learning [61.231471139896506]
  We study an approach to improve the memory and computational efficiency of ICL inference by compressing the many-shot prompts.<n>We first show that existing prompt compression methods are ineffective for many-shot compression.<n>We propose MemCom, a layer-wise compression method.
  arXiv  Detail & Related papers  (2025-10-17T16:57:42Z)
• Are We Using the Right Benchmark: An Evaluation Framework for Visual Token Compression Methods [54.4711434793961]
  We show that simple image downsampling consistently outperforms many advanced compression methods across multiple widely used benchmarks.<n>Motivated by these findings, we introduce VTC-Bench, an evaluation framework that incorporates a data filtering mechanism to denoise existing benchmarks.
  arXiv  Detail & Related papers  (2025-10-08T15:44:28Z)
• UniGist: Towards General and Hardware-aligned Sequence-level Long Context Compression [86.33995240043936]
  UniGist is a sequence-level long-context compression framework for large language models.<n>It efficiently preserves context information by replacing raw tokens with special compression tokens (gists) in a fine-grained manner.<n>Our scheme also supports flexible inference by allowing the actual removal of compressed tokens, resulting in real-time memory savings.
  arXiv  Detail & Related papers  (2025-09-19T08:47:37Z)
• Plug it and Play on Logs: A Configuration-Free Statistic-Based Log Parser [2.342181121922312]
  Log parsing is an essential task in log analysis, and many tools have been designed to accomplish it.<n>In comparison to semantic-baseds, existing statistic-baseds tend to be more efficient, require lower computational costs, and be more privacy-preserving thanks to on-premise deployment.<n>It became a common belief that statistic-baseds cannot be as effective as semantic-baseds.<n>Our work challenges this belief with a novel statistic-based, PIPLUP.
  arXiv  Detail & Related papers  (2025-08-12T21:47:27Z)
• Efficient Token Compression for Vision Transformer with Spatial Information Preserved [59.79302182800274]
  Token compression is essential for reducing the computational and memory requirements of transformer models.<n>We propose an efficient and hardware-compatible token compression method called Prune and Merge.
  arXiv  Detail & Related papers  (2025-03-30T14:23:18Z)
• Optimizing Singular Spectrum for Large Language Model Compression [95.7621116637755]
  We introduce SoCo, a novel compression framework that learns to rescale the decomposed components of SVD in a data-driven manner.<n>Thanks to the learnable singular spectrum, SoCo adaptively prunes components according to the sparsified importance scores.<n> Experimental evaluations across multiple LLMs and benchmarks demonstrate that SoCo surpasses the state-of-the-art methods in model compression.
  arXiv  Detail & Related papers  (2025-02-20T23:18:39Z)
• HELP: Hierarchical Embeddings-based Log Parsing [0.25112747242081457]
  Logs are a first-hand source of information for software maintenance and failure diagnosis. Log parsing is a prerequisite for automated log analysis tasks such as anomaly detection, troubleshooting, and root cause analysis. Existing online parsing algorithms are susceptible to log drift, where slight log changes create false positives that drown out real anomalies.
  arXiv  Detail & Related papers  (2024-08-15T17:54:31Z)
• Unlocking the Power of Numbers: Log Compression via Numeric Token Parsing [18.419996813182163]
  We propose Denum, a simple, general log compressor with high compression ratio and speed. Specifically, Denum contains a Token Parsing module, which extracts all numeric tokens and applies tailored processing methods. It has been evaluated on 16 log datasets and it achieves an 8.7% average compression ratio and 2.6x faster average compression speed.
  arXiv  Detail & Related papers  (2024-08-11T12:30:41Z)
• Fundamental Limits of Prompt Compression: A Rate-Distortion Framework for Black-Box Language Models [21.025001473355996]
  We formalize the problem of prompt compression for large language models (LLMs)<n>We present a framework to unify token-level prompt compression methods which create hard prompts for black-box models.<n>We show that there is a large gap between the performance of current prompt compression methods and the optimal strategy.
  arXiv  Detail & Related papers  (2024-07-22T09:40:13Z)
• LogShrink: Effective Log Compression by Leveraging Commonality and Variability of Log Data [35.5712445690333]
  We propose LogShrink, a novel and effective log compression method by leveraging commonality and variability of log data. An analyzer based on longest common subsequence and entropy techniques is proposed to identify the latent commonality and variability in log messages. The key idea behind this is that the commonality and variability can be exploited to shrink log data with a shorter representation.
  arXiv  Detail & Related papers  (2023-09-18T04:27:05Z)
• Self-Supervised Log Parsing [59.04636530383049]
  Large-scale software systems generate massive volumes of semi-structured log records. Existing approaches rely on log-specifics or manual rule extraction. We propose NuLog that utilizes a self-supervised learning model and formulates the parsing task as masked language modeling.
  arXiv  Detail & Related papers  (2020-03-17T19:25:25Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.