AL-Bench: A Benchmark for Automatic Logging

• URL: http://arxiv.org/abs/2502.03160v2
• Date: Fri, 07 Feb 2025 13:46:57 GMT
• Title: AL-Bench: A Benchmark for Automatic Logging
• Authors: Boyin Tan, Junjielong Xu, Zhouruixing Zhu, Pinjia He,
• Abstract summary: This paper introduces AL-Bench, a comprehensive benchmark designed specifically for automatic logging tools. The codes with log statements generated by the state-of-the-art tools fail to compile in 20.1%-83.6% cases. Even the best-performing tool only achieves 0.213 cosine similarity between the runtime logs produced by the generated log statements and the ground-truth log statements.
• Score: 3.8293110324859505
• License:
• Abstract: Logging, the practice of inserting log statements into source code, is critical for improving software reliability. Recently, language model-based techniques have been developed to automate log statement generation based on input code. These tools show promising results in their own evaluation. However, current evaluation practices in log statement generation face significant challenges. The lack of a unified, large-scale dataset forces studies to rely on ad-hoc data, hindering consistency and reproducibility. Additionally, assessments based solely on metrics like code similarity fail to reflect real-world effectiveness. These limitations underscore the need for a comprehensive public benchmark to standardize evaluation. This paper introduces AL-Bench, a comprehensive benchmark designed specifically for automatic logging tools. AL-Bench includes a high-quality, diverse dataset collected from 10 widely recognized projects with varying logging requirements and introduces a novel dynamic evaluation approach. Different from the existing evaluations that focus only on components of log statements like code similarity, AL-Bench assesses both the compilability of the code with inserted log statements and the effectiveness of the logs generated by them during runtime, which we believe can better reflect the effectiveness of logging techniques in practice. AL-Bench reveals significant limitations in the state-of-the-art tools. The codes with log statements generated by the state-of-the-art tools fail to compile in 20.1%-83.6% cases. In addition, even the best-performing tool only achieves 0.213 cosine similarity between the runtime logs produced by the generated log statements and the ground-truth log statements. The results reveal substantial opportunities to further enhance the development of automatic logging tools.

Related papers

• Codev-Bench: How Do LLMs Understand Developer-Centric Code Completion? [60.84912551069379]
  We present the Code-Development Benchmark (Codev-Bench), a fine-grained, real-world, repository-level, and developer-centric evaluation framework. Codev-Agent is an agent-based system that automates repository crawling, constructs execution environments, extracts dynamic calling chains from existing unit tests, and generates new test samples to avoid data leakage.
  arXiv  Detail & Related papers  (2024-10-02T09:11:10Z)
• HELP: Hierarchical Embeddings-based Log Parsing [0.25112747242081457]
  Logs are a first-hand source of information for software maintenance and failure diagnosis. Log parsing is a prerequisite for automated log analysis tasks such as anomaly detection, troubleshooting, and root cause analysis. Existing online parsing algorithms are susceptible to log drift, where slight log changes create false positives that drown out real anomalies.
  arXiv  Detail & Related papers  (2024-08-15T17:54:31Z)
• A Large-Scale Evaluation for Log Parsing Techniques: How Far Are We? [42.56249610409624]
  We provide a new collection of annotated log datasets, denoted Loghub-2.0, which can better reflect the characteristics of log data in real-world software systems. We conduct a thorough re-evaluation of 15 state-of-the-art logs in a more rigorous and practical setting. Particularly, we introduce a new evaluation metric to mitigate the sensitivity of existing metrics to imbalanced data distributions.
  arXiv  Detail & Related papers  (2023-08-21T16:24:15Z)
• Log Parsing Evaluation in the Era of Modern Software Systems [47.370291246632114]
  We focus on one integral part of automated log analysis, log parsing, which is the prerequisite to deriving any insights from logs. Our investigation reveals problematic aspects within the log parsing field, particularly its inefficiency in handling heterogeneous real-world logs. We propose a tool, Logchimera, that enables estimating log parsing performance in industry contexts.
  arXiv  Detail & Related papers  (2023-08-17T14:19:22Z)
• Exploring the Effectiveness of LLMs in Automated Logging Generation: An Empirical Study [32.53659676826846]
  This paper performs the first study on exploring large language models (LLMs) for logging statement generation. We first build a logging statement generation dataset, LogBench, with two parts: (1) LogBench-O: logging statements collected from GitHub repositories, and (2) LogBench-T: the transformed unseen code from LogBench-O.
  arXiv  Detail & Related papers  (2023-07-12T06:32:51Z)
• SKTR: Trace Recovery from Stochastically Known Logs [7.882975068446842]
  Developments in machine learning together with the increasing usage of sensor data challenge the reliance on deterministic logs. In this work we formulate the task of generating a deterministic log fromally known logs that is as faithful to reality as possible. An effective trace recovery algorithm would be a powerful aid for maintaining credible process mining tools for uncertain settings.
  arXiv  Detail & Related papers  (2022-06-25T15:29:20Z)
• Data-Driven Approach for Log Instruction Quality Assessment [59.04636530383049]
  There are no widely adopted guidelines on how to write log instructions with good quality properties. We identify two quality properties: 1) correct log level assignment assessing the correctness of the log level, and 2) sufficient linguistic structure assessing the minimal richness of the static text necessary for verbose event description. Our approach correctly assesses log level assignments with an accuracy of 0.88, and the sufficient linguistic structure with an F1 score of 0.99, outperforming the baselines.
  arXiv  Detail & Related papers  (2022-04-06T07:02:23Z)
• Borrowing from Similar Code: A Deep Learning NLP-Based Approach for Log Statement Automation [0.0]
  We introduce an updated and improved log-aware code-clone detection method to predict the location of logging statements. We incorporate natural language processing (NLP) and deep learning methods to automate the log statements' description prediction. Our analysis shows that our hybrid NLP and code-clone detection approach (NLP CC'd) outperforms conventional clone detectors in finding log statement locations.
  arXiv  Detail & Related papers  (2021-12-02T14:03:49Z)
• LogLAB: Attention-Based Labeling of Log Data Anomalies via Weak Supervision [63.08516384181491]
  We present LogLAB, a novel modeling approach for automated labeling of log messages without requiring manual work by experts. Our method relies on estimated failure time windows provided by monitoring systems to produce precise labeled datasets in retrospect. Our evaluation shows that LogLAB consistently outperforms nine benchmark approaches across three different datasets and maintains an F1-score of more than 0.98 even at large failure time windows.
  arXiv  Detail & Related papers  (2021-11-02T15:16:08Z)
• Self-Supervised Log Parsing [59.04636530383049]
  Large-scale software systems generate massive volumes of semi-structured log records. Existing approaches rely on log-specifics or manual rule extraction. We propose NuLog that utilizes a self-supervised learning model and formulates the parsing task as masked language modeling.
  arXiv  Detail & Related papers  (2020-03-17T19:25:25Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.