AUCAD: Automated Construction of Alignment Dataset from Log-Related Issues for Enhancing LLM-based Log Generation

• URL: http://arxiv.org/abs/2412.18835v2
• Date: Wed, 13 Aug 2025 08:07:49 GMT
• Title: AUCAD: Automated Construction of Alignment Dataset from Log-Related Issues for Enhancing LLM-based Log Generation
• Authors: Hao Zhang, Dongjun Yu, Lei Zhang, Guoping Rong, Yongda Yu, Haifeng Shen, He Zhang, Dong Shao, Hongyu Kuang,
• Abstract summary: This paper explores enhancing the performance of LLM-based solutions for automated log statement generation by post-training with a purpose-built dataset.<n>A novel approach called AUCAD automatically constructs such a dataset with information extracting from log-related issues.<n>Both human and experimental evaluations indicate that these models significantly outperform existing LLM-based solutions.
• Score: 19.410504836739058
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Log statements have become an integral part of modern software systems. Prior research efforts have focused on supporting the decisions of placing log statements, such as where/what to log. With the increasing adoption of Large Language Models (LLMs) for code-related tasks such as code completion or generation, automated approaches for generating log statements have gained much momentum. However, the performance of these approaches still has a long way to go. This paper explores enhancing the performance of LLM-based solutions for automated log statement generation by post-training LLMs with a purpose-built dataset. Thus the primary contribution is a novel approach called AUCAD, which automatically constructs such a dataset with information extracting from log-related issues. Researchers have long noticed that a significant portion of the issues in the open-source community are related to log statements. However, distilling this portion of data requires manual efforts, which is labor-intensive and costly, rendering it impractical. Utilizing our approach, we automatically extract log-related issues from 1,537 entries of log data across 88 projects and identify 808 code snippets (i.e., methods) with retrievable source code both before and after modification of each issue (including log statements) to construct a dataset. Each entry in the dataset consists of a data pair representing high-quality and problematic log statements, respectively. With this dataset, we proceed to post-train multiple LLMs (primarily from the Llama series) for automated log statement generation. Both human and experimental evaluations indicate that these models significantly outperform existing LLM-based solutions, thereby validating the efficacy of our method for constructing a post-training dataset to enhance LLM-based log statement generation.

Related papers

• Larger Is Not Always Better: Exploring Small Open-source Language Models in Logging Statement Generation [28.884070374408203]
  Large language models (LLMs) for automated logging statement generation present privacy and resource issues.<n>This paper presents the first large-scale empirical study evaluating small open-source language models (SOLMs) for automated logging statement generation.
  arXiv  Detail & Related papers  (2025-05-22T12:26:53Z)
• SoK: LLM-based Log Parsing [2.2779174914142346]
  This paper systematically reviews 29 large language models (LLMs)-based log parsing methods. We analyze the learning and prompt-engineering paradigms employed, efficiency- and effectiveness-enhancing techniques, and the role of LLMs in the parsing process.
  arXiv  Detail & Related papers  (2025-04-07T09:41:04Z)
• LogLLM: Log-based Anomaly Detection Using Large Language Models [8.03646578793411]
  We propose LogLLM, a log-based anomaly detection framework that leverages large language models (LLMs) LogLLM employs BERT for extracting semantic vectors from log messages, while utilizing Llama, a transformer decoder-based model, for classifying log sequences. Our framework is trained through a novel three-stage procedure designed to enhance performance and adaptability.
  arXiv  Detail & Related papers  (2024-11-13T12:18:00Z)
• Studying and Benchmarking Large Language Models For Log Level Suggestion [49.176736212364496]
  Large Language Models (LLMs) have become a focal point of research across various domains. This paper investigates the impact of characteristics and learning paradigms on the performance of 12 open-source LLMs in log level suggestion.
  arXiv  Detail & Related papers  (2024-10-11T03:52:17Z)
• LogParser-LLM: Advancing Efficient Log Parsing with Large Language Models [19.657278472819588]
  We introduce Log-LLM, a novel log integrated with LLM capabilities. We address the intricate challenge of parsing granularity, proposing a new metric to allow users to calibrate granularity to their specific needs. Our method's efficacy is empirically demonstrated through evaluations on the Loghub-2k and the large-scale LogPub benchmark.
  arXiv  Detail & Related papers  (2024-08-25T05:34:24Z)
• SELF-GUIDE: Better Task-Specific Instruction Following via Self-Synthetic Finetuning [70.21358720599821]
  Large language models (LLMs) hold the promise of solving diverse tasks when provided with appropriate natural language prompts. We propose SELF-GUIDE, a multi-stage mechanism in which we synthesize task-specific input-output pairs from the student LLM. We report an absolute improvement of approximately 15% for classification tasks and 18% for generation tasks in the benchmark's metrics.
  arXiv  Detail & Related papers  (2024-07-16T04:41:58Z)
• DARG: Dynamic Evaluation of Large Language Models via Adaptive Reasoning Graph [70.79413606968814]
  We introduce Dynamic Evaluation of LLMs via Adaptive Reasoning Graph Evolvement (DARG) to dynamically extend current benchmarks with controlled complexity and diversity. Specifically, we first extract the reasoning graphs of data points in current benchmarks and then perturb the reasoning graphs to generate novel testing data. Such newly generated test samples can have different levels of complexity while maintaining linguistic diversity similar to the original benchmarks.
  arXiv  Detail & Related papers  (2024-06-25T04:27:53Z)
• LUNAR: Unsupervised LLM-based Log Parsing [34.344687402936835]
  We propose LUNAR, an unsupervised-based method for efficient and off-the-shelf log parsing. Our key insight is that while LLMs may struggle with direct log parsing, their performance can be significantly enhanced through comparative analysis. Experiments on large-scale public datasets demonstrate that LUNAR significantly outperforms state-of-the-art log crafts in terms of accuracy and efficiency.
  arXiv  Detail & Related papers  (2024-06-11T11:32:01Z)
• Log Parsing with Self-Generated In-Context Learning and Self-Correction [15.93927602769091]
  Despite a variety of log parsing methods that have been proposed, their performance on evolving log data remains unsatisfactory due to reliance on human-crafted rules or learning-based models with limited training data. We propose Ada, an effective and adaptive log parsing framework using LLMs with self-generated in-context learning (SG-ICL) and self-correction.
  arXiv  Detail & Related papers  (2024-06-05T15:31:43Z)
• CodecLM: Aligning Language Models with Tailored Synthetic Data [51.59223474427153]
  We introduce CodecLM, a framework for adaptively generating high-quality synthetic data for instruction-following abilities. We first encode seed instructions into metadata, which are concise keywords generated on-the-fly to capture the target instruction distribution. We also introduce Self-Rubrics and Contrastive Filtering during decoding to tailor data-efficient samples.
  arXiv  Detail & Related papers  (2024-04-08T21:15:36Z)
• Data is all you need: Finetuning LLMs for Chip Design via an Automated design-data augmentation framework [50.02710905062184]
  This paper proposes an automated design-data augmentation framework, which generates high-volume and high-quality natural language aligned with Verilog and EDA scripts. The accuracy of Verilog generation surpasses that of the current state-of-the-art open-source Verilog generation model, increasing from 58.8% to 70.6% with the same benchmark.
  arXiv  Detail & Related papers  (2024-03-17T13:01:03Z)
• Learning Representations on Logs for AIOps [6.47086647390439]
  Large Language Models (LLMs) are trained using self-supervision on a vast amount of unlabeled data. This paper introduces a LLM for log data which is trained on public and proprietary log data. Our proposed LLM, trained on public and proprietary log data, offers superior performance on multiple downstream tasks.
  arXiv  Detail & Related papers  (2023-08-18T20:34:46Z)
• Log Parsing Evaluation in the Era of Modern Software Systems [47.370291246632114]
  We focus on one integral part of automated log analysis, log parsing, which is the prerequisite to deriving any insights from logs. Our investigation reveals problematic aspects within the log parsing field, particularly its inefficiency in handling heterogeneous real-world logs. We propose a tool, Logchimera, that enables estimating log parsing performance in industry contexts.
  arXiv  Detail & Related papers  (2023-08-17T14:19:22Z)
• Exploring the Effectiveness of LLMs in Automated Logging Generation: An Empirical Study [32.53659676826846]
  This paper performs the first study on exploring large language models (LLMs) for logging statement generation. We first build a logging statement generation dataset, LogBench, with two parts: (1) LogBench-O: logging statements collected from GitHub repositories, and (2) LogBench-T: the transformed unseen code from LogBench-O.
  arXiv  Detail & Related papers  (2023-07-12T06:32:51Z)
• LogLAB: Attention-Based Labeling of Log Data Anomalies via Weak Supervision [63.08516384181491]
  We present LogLAB, a novel modeling approach for automated labeling of log messages without requiring manual work by experts. Our method relies on estimated failure time windows provided by monitoring systems to produce precise labeled datasets in retrospect. Our evaluation shows that LogLAB consistently outperforms nine benchmark approaches across three different datasets and maintains an F1-score of more than 0.98 even at large failure time windows.
  arXiv  Detail & Related papers  (2021-11-02T15:16:08Z)
• Self-Supervised Log Parsing [59.04636530383049]
  Large-scale software systems generate massive volumes of semi-structured log records. Existing approaches rely on log-specifics or manual rule extraction. We propose NuLog that utilizes a self-supervised learning model and formulates the parsing task as masked language modeling.
  arXiv  Detail & Related papers  (2020-03-17T19:25:25Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.