Plug it and Play on Logs: A Configuration-Free Statistic-Based Log Parser

• URL: http://arxiv.org/abs/2508.09366v1
• Date: Tue, 12 Aug 2025 21:47:27 GMT
• Title: Plug it and Play on Logs: A Configuration-Free Statistic-Based Log Parser
• Abstract summary: Log parsing is an essential task in log analysis, and many tools have been designed to accomplish it.<n>In comparison to semantic-baseds, existing statistic-baseds tend to be more efficient, require lower computational costs, and be more privacy-preserving thanks to on-premise deployment.<n>It became a common belief that statistic-baseds cannot be as effective as semantic-baseds.<n>Our work challenges this belief with a novel statistic-based, PIPLUP.
• Score: 2.342181121922312
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Log parsing is an essential task in log analysis, and many tools have been designed to accomplish it. Existing log parsers can be categorized into statistic-based and semantic-based approaches. In comparison to semantic-based parsers, existing statistic-based parsers tend to be more efficient, require lower computational costs, and be more privacy-preserving thanks to on-premise deployment, but often fall short in their accuracy (e.g., grouping or parsing accuracy) and generalizability. Therefore, it became a common belief that statistic-based parsers cannot be as effective as semantic-based parsers since the latter could take advantage of external knowledge supported by pretrained language models. Our work, however, challenges this belief with a novel statistic-based parser, PIPLUP. PIPLUP eliminates the pre-assumption of the position of constant tokens for log grouping and relies on data-insensitive parameters to overcome the generalizability challenge, allowing "plug and play" on given log files. According to our experiments on an open-sourced large log dataset, PIPLUP shows promising accuracy and generalizability with the data-insensitive default parameter set. PIPLUP not only outperforms the state-of-the-art statistic-based log parsers, Drain and its variants, but also obtains a competitive performance compared to the best unsupervised semantic-based log parser (i.e., LUNAR). Further, PIPLUP exhibits low time consumption without GPU acceleration and external API usage; our simple, efficient, and effective approach makes it more practical in real-world adoptions, especially when costs and privacy are of major concerns.

Related papers

• VarParser: Unleashing the Neglected Power of Variables for LLM-based Log Parsing [10.173320661352257]
  Logs serve as a primary source of information for engineers to diagnose failures in large-scale online service systems.<n>With advances in large language models (LLMs), leveraging their strong text understanding capabilities has proven effective for accurate log parsing.<n>Existing LLM-based logs all focus on the constant part of logs, ignoring the potential contribution of the variable part to log parsing.
  arXiv  Detail & Related papers  (2026-01-30T07:49:54Z)
• Accelerated Test-Time Scaling with Model-Free Speculative Sampling [58.69141724095398]
  We introduce STAND (STochastic Adaptive N-gram Drafting), a novel model-free speculative decoding approach.<n>We show that STAND reduces inference latency by 60-65% compared to standard autoregressive decoding.<n>As a model-free approach, STAND can be applied to any existing language model without additional training.
  arXiv  Detail & Related papers  (2025-06-05T07:31:18Z)
• Log-Augmented Generation: Scaling Test-Time Reasoning with Reusable Computation [80.69067017594709]
  Large language models (LLMs) and their agentic counterparts struggle to retain reasoning from previous tasks.<n>We propose a novel framework, log-augmented generation (LAG) that directly reuses prior computation and reasoning from past logs at test time.<n>Our method significantly outperforms standard agentic systems that do not utilize logs.
  arXiv  Detail & Related papers  (2025-05-20T14:14:38Z)
• Preprocessing is All You Need: Boosting the Performance of Log Parsers With a General Preprocessing Framework [2.342181121922312]
  Prior work has proposed many statistic-based logs (e.g., Drain) which are highly efficient.<n>We developed a general preprocessing framework with our findings and evaluated its impact on existing parsings.<n>Our experiments show that the preprocessing framework significantly boosts the performance of four state-of-the-art statistic-baseds.
  arXiv  Detail & Related papers  (2024-12-06T18:39:34Z)
• LogParser-LLM: Advancing Efficient Log Parsing with Large Language Models [19.657278472819588]
  We introduce Log-LLM, a novel log integrated with LLM capabilities. We address the intricate challenge of parsing granularity, proposing a new metric to allow users to calibrate granularity to their specific needs. Our method's efficacy is empirically demonstrated through evaluations on the Loghub-2k and the large-scale LogPub benchmark.
  arXiv  Detail & Related papers  (2024-08-25T05:34:24Z)
• HELP: Hierarchical Embeddings-based Log Parsing [0.25112747242081457]
  Logs are a first-hand source of information for software maintenance and failure diagnosis. Log parsing is a prerequisite for automated log analysis tasks such as anomaly detection, troubleshooting, and root cause analysis. Existing online parsing algorithms are susceptible to log drift, where slight log changes create false positives that drown out real anomalies.
  arXiv  Detail & Related papers  (2024-08-15T17:54:31Z)
• LUNAR: Unsupervised LLM-based Log Parsing [34.344687402936835]
  We propose LUNAR, an unsupervised-based method for efficient and off-the-shelf log parsing. Our key insight is that while LLMs may struggle with direct log parsing, their performance can be significantly enhanced through comparative analysis. Experiments on large-scale public datasets demonstrate that LUNAR significantly outperforms state-of-the-art log crafts in terms of accuracy and efficiency.
  arXiv  Detail & Related papers  (2024-06-11T11:32:01Z)
• Stratified Prediction-Powered Inference for Hybrid Language Model Evaluation [62.2436697657307]
  Prediction-powered inference (PPI) is a method that improves statistical estimates based on limited human-labeled data.<n>We propose a method called Stratified Prediction-Powered Inference (StratPPI)<n>We show that the basic PPI estimates can be considerably improved by employing simple data stratification strategies.
  arXiv  Detail & Related papers  (2024-06-06T17:37:39Z)
• ECLIPSE: Semantic Entropy-LCS for Cross-Lingual Industrial Log Parsing [30.093766305800568]
  We propose ECLIPSE, Enhanced Cross-Lingual Industrial log Parsing with Semantic Entropy-LCS. It integrates two efficient data-driven template-matching algorithms and Faiss indexing. Notably, we launch a Chinese and English cross-platform industrial log parsing benchmark ECLIPSE- BENCH to evaluate the performance mainstreams in industrial scenarios.
  arXiv  Detail & Related papers  (2024-05-22T11:33:29Z)
• LogFormer: A Pre-train and Tuning Pipeline for Log Anomaly Detection [73.69399219776315]
  We propose a unified Transformer-based framework for Log anomaly detection (LogFormer) to improve the generalization ability across different domains. Specifically, our model is first pre-trained on the source domain to obtain shared semantic knowledge of log data. Then, we transfer such knowledge to the target domain via shared parameters.
  arXiv  Detail & Related papers  (2024-01-09T12:55:21Z)
• Log Parsing Evaluation in the Era of Modern Software Systems [47.370291246632114]
  We focus on one integral part of automated log analysis, log parsing, which is the prerequisite to deriving any insights from logs. Our investigation reveals problematic aspects within the log parsing field, particularly its inefficiency in handling heterogeneous real-world logs. We propose a tool, Logchimera, that enables estimating log parsing performance in industry contexts.
  arXiv  Detail & Related papers  (2023-08-17T14:19:22Z)
• Self-Supervised Log Parsing [59.04636530383049]
  Large-scale software systems generate massive volumes of semi-structured log records. Existing approaches rely on log-specifics or manual rule extraction. We propose NuLog that utilizes a self-supervised learning model and formulates the parsing task as masked language modeling.
  arXiv  Detail & Related papers  (2020-03-17T19:25:25Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.