LogLSHD: Fast Log Parsing with Locality-Sensitive Hashing and Dynamic Time Warping

• URL: http://arxiv.org/abs/2504.02172v1
• Date: Wed, 02 Apr 2025 23:08:04 GMT
• Title: LogLSHD: Fast Log Parsing with Locality-Sensitive Hashing and Dynamic Time Warping
• Authors: Shu-Wei Huang, Xingfang Wu, Heng Li,
• Abstract summary: Large-scale software systems generate vast volumes of system logs that are essential for monitoring, diagnosing, and performance optimization.<n>LogLSHD demonstrates exceptional efficiency in parsing time, significantly outperforming state-of-the-art methods.<n>For example, compared to Drain, LogLSHD reduces the average parsing time by 73% while increasing the average parsing accuracy by 15% on the LogHub 2.0 benchmark.
• Score: 2.415288727960745
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Large-scale software systems generate vast volumes of system logs that are essential for monitoring, diagnosing, and performance optimization. However, the unstructured nature and ever-growing scale of these logs present significant challenges for manual analysis and automated downstream tasks such as anomaly detection. Log parsing addresses these challenges by converting raw logs into structured formats, enabling efficient log analysis. Despite its importance, existing log parsing methods suffer from limitations in efficiency and scalability, due to the large size of log data and their heterogeneous formats. To overcome these challenges, this study proposes a log parsing approach, LogLSHD, which leverages Locality-Sensitive Hashing (LSH) to group similar logs and integrates Dynamic Time Warping (DTW) to enhance the accuracy of template extraction. LogLSHD demonstrates exceptional efficiency in parsing time, significantly outperforming state-of-the-art methods. For example, compared to Drain, LogLSHD reduces the average parsing time by 73% while increasing the average parsing accuracy by 15% on the LogHub 2.0 benchmark.

Related papers

• Adaptive and Efficient Log Parsing as a Cloud Service [11.096357194371421]
  ByteBrain-Log is an innovative log parsing framework designed specifically for cloud environments. It processes 229,000 logs per second on average, achieving an 840% speedup over the fastest baseline.
  arXiv  Detail & Related papers  (2025-04-12T07:53:19Z)
• LogParser-LLM: Advancing Efficient Log Parsing with Large Language Models [19.657278472819588]
  We introduce Log-LLM, a novel log integrated with LLM capabilities. We address the intricate challenge of parsing granularity, proposing a new metric to allow users to calibrate granularity to their specific needs. Our method's efficacy is empirically demonstrated through evaluations on the Loghub-2k and the large-scale LogPub benchmark.
  arXiv  Detail & Related papers  (2024-08-25T05:34:24Z)
• HELP: Hierarchical Embeddings-based Log Parsing [0.25112747242081457]
  Logs are a first-hand source of information for software maintenance and failure diagnosis. Log parsing is a prerequisite for automated log analysis tasks such as anomaly detection, troubleshooting, and root cause analysis. Existing online parsing algorithms are susceptible to log drift, where slight log changes create false positives that drown out real anomalies.
  arXiv  Detail & Related papers  (2024-08-15T17:54:31Z)
• LogFormer: A Pre-train and Tuning Pipeline for Log Anomaly Detection [73.69399219776315]
  We propose a unified Transformer-based framework for Log anomaly detection (LogFormer) to improve the generalization ability across different domains. Specifically, our model is first pre-trained on the source domain to obtain shared semantic knowledge of log data. Then, we transfer such knowledge to the target domain via shared parameters.
  arXiv  Detail & Related papers  (2024-01-09T12:55:21Z)
• A Large-Scale Evaluation for Log Parsing Techniques: How Far Are We? [42.56249610409624]
  We provide a new collection of annotated log datasets, denoted Loghub-2.0, which can better reflect the characteristics of log data in real-world software systems. We conduct a thorough re-evaluation of 15 state-of-the-art logs in a more rigorous and practical setting. Particularly, we introduce a new evaluation metric to mitigate the sensitivity of existing metrics to imbalanced data distributions.
  arXiv  Detail & Related papers  (2023-08-21T16:24:15Z)
• Log Parsing Evaluation in the Era of Modern Software Systems [47.370291246632114]
  We focus on one integral part of automated log analysis, log parsing, which is the prerequisite to deriving any insights from logs. Our investigation reveals problematic aspects within the log parsing field, particularly its inefficiency in handling heterogeneous real-world logs. We propose a tool, Logchimera, that enables estimating log parsing performance in industry contexts.
  arXiv  Detail & Related papers  (2023-08-17T14:19:22Z)
• ClusterLog: Clustering Logs for Effective Log-based Anomaly Detection [3.3196401064045014]
  This study proposes ClusterLog, a log pre-processing method that clusters the temporal sequence of log keys based on their semantic similarity. By grouping semantically and sentimentally similar logs, this approach aims to represent log sequences with the smallest amount of unique log keys, intending to improve the ability of a downstream sequence-based model to effectively learn the log patterns.
  arXiv  Detail & Related papers  (2023-01-19T01:54:48Z)
• LogLAB: Attention-Based Labeling of Log Data Anomalies via Weak Supervision [63.08516384181491]
  We present LogLAB, a novel modeling approach for automated labeling of log messages without requiring manual work by experts. Our method relies on estimated failure time windows provided by monitoring systems to produce precise labeled datasets in retrospect. Our evaluation shows that LogLAB consistently outperforms nine benchmark approaches across three different datasets and maintains an F1-score of more than 0.98 even at large failure time windows.
  arXiv  Detail & Related papers  (2021-11-02T15:16:08Z)
• Robust and Transferable Anomaly Detection in Log Data using Pre-Trained Language Models [59.04636530383049]
  Anomalies or failures in large computer systems, such as the cloud, have an impact on a large number of users. We propose a framework for anomaly detection in log data, as a major troubleshooting source of system information.
  arXiv  Detail & Related papers  (2021-02-23T09:17:05Z)
• Self-Attentive Classification-Based Anomaly Detection in Unstructured Logs [59.04636530383049]
  We propose Logsy, a classification-based method to learn log representations. We show an average improvement of 0.25 in the F1 score, compared to the previous methods.
  arXiv  Detail & Related papers  (2020-08-21T07:26:55Z)
• Self-Supervised Log Parsing [59.04636530383049]
  Large-scale software systems generate massive volumes of semi-structured log records. Existing approaches rely on log-specifics or manual rule extraction. We propose NuLog that utilizes a self-supervised learning model and formulates the parsing task as masked language modeling.
  arXiv  Detail & Related papers  (2020-03-17T19:25:25Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.