Logging Like Humans for LLMs: Rethinking Logging via Execution and Runtime Feedback

• URL: http://arxiv.org/abs/2603.29122v1
• Date: Tue, 31 Mar 2026 01:18:49 GMT
• Title: Logging Like Humans for LLMs: Rethinking Logging via Execution and Runtime Feedback
• Abstract summary: We introduce ReLog, an iterative logging generation framework guided by runtime feedback.<n>We evaluate ReLog through downstream debug tasks, including defect localization and repair.<n>Results show that ReLog consistently outperforms all baselines.
• Score: 17.85879405133239
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Logging statements are essential for software debugging and maintenance. However, existing approaches to automatic logging generation rely on static analysis and produce statements in a single pass without considering runtime behavior. They are also typically evaluated by similarity to developer-written logs, assuming these logs form an adequate gold standard. This assumption is increasingly limiting in the LLM era, where logs are consumed not only by developers but also by LLMs for downstream tasks. As a result, optimizing logs for human similarity does not necessarily reflect their practical utility. To address these limitations, we introduce ReLog, an iterative logging generation framework guided by runtime feedback. ReLog leverages LLMs to generate, execute, evaluate, and refine logging statements so that runtime logs better support downstream tasks. Instead of comparing against developer-written logs, we evaluate ReLog through downstream debugging tasks, including defect localization and repair. We construct a benchmark based on Defects4J under both direct and indirect debugging settings. Results show that ReLog consistently outperforms all baselines, achieving an F1 score of 0.520 and repairing 97 defects in the direct setting, and the best F1 score of 0.408 in the indirect setting where source code is unavailable. Additional experiments across multiple LLMs demonstrate the generality of the framework, while ablations confirm the importance of iterative refinement and compilation repair. Overall, our work reframes logging as a runtime-guided, task-oriented process and advocates evaluating logs by their downstream utility rather than textual similarity.

Related papers

• RepoReasoner: Evaluating Repository-Level Code Reasoning Ability of Long-Context Language Models [54.53236295237077]
  We introduce RepoReasoner, a benchmark for evaluating repository-level code reasoning.<n> Output Prediction measures fine-grained, stateful execution reasoning across files.<n>Call Chain Prediction evaluates high-level architectural dependency understanding under noisy context.
  arXiv  Detail & Related papers  (2026-07-28T17:12:41Z)
• ThinkLog: Leveraging Reasoning for Log Statement Generation [4.372207458643732]
  ThinkLog is an end-to-end log statement generation method.<n>It incorporates reasoning that helps LLMs make decisions about log insertion, severity level assignment, and message generation.<n>It achieves 20.55% log statement generation accuracy, representing a 15.4% improvement over the best existing method.
  arXiv  Detail & Related papers  (2026-07-13T14:37:35Z)
• VeriPilot: An LLM-Powered Verilog Debugging Framework [28.151772567640354]
  We propose VeriPilot, a framework to enable fine-grained bug localization and repair in Verilog designs.<n>VeriPilot goes beyond output-level comparison by aligning internal variable semantics between the Verilog design and its corresponding golden model.<n>It then performs step-by-step signal tracing using Control-Data-Flow Graphs (CDFGs) derived from static analysis.
  arXiv  Detail & Related papers  (2026-06-22T09:15:40Z)
• LLM-SrcLog: Towards Proactive and Unified Log Template Extraction via Large Language Models [19.933913707655467]
  LLM-SrcLog is a proactive and unified framework for log template parsing.<n>It extracts templates directly from source code prior to deployment.<n>It supplements them with data-driven parsing for logs without available code.
  arXiv  Detail & Related papers  (2025-12-04T05:30:15Z)
• InspectCoder: Dynamic Analysis-Enabled Self Repair through interactive LLM-Debugger Collaboration [71.18377595277018]
  Large Language Models (LLMs) frequently generate buggy code with complex logic errors that are challenging to diagnose.<n>We present InspectCoder, the first agentic program repair system that empowers LLMs to actively conduct dynamic analysis via interactive debugger control.
  arXiv  Detail & Related papers  (2025-10-21T06:26:29Z)
• D-LiFT: Improving LLM-based Decompiler Backend via Code Quality-driven Fine-tuning [49.16469288280772]
  Decompilers reconstruct human-readable source code from binaries.<n>Despite recent advances, their outputs often suffer from syntactic and semantic errors and remain difficult to read.<n>With the advent of large language models (LLMs), researchers began to explore the potential of LLMs to refine decompiler output.<n>We present D-LIFT, an enhanced decompiler-LLM pipeline with fine-tuned reinforcement learning.
  arXiv  Detail & Related papers  (2025-06-11T19:09:08Z)
• AL-Bench: A Benchmark for Automatic Logging [3.8293110324859505]
  We introduce AL-Bench, a benchmark designed specifically for automatic logging tools.<n> AL-Bench includes a large-scale, high-quality, diverse dataset collected from 10 widely recognized projects.<n>It provides a run-time perspective of logging quality in addition to the traditional static evaluation at source code level.
  arXiv  Detail & Related papers  (2025-02-05T13:32:39Z)
• HELP: Hierarchical Embeddings-based Log Parsing [0.25112747242081457]
  Logs are a first-hand source of information for software maintenance and failure diagnosis. Log parsing is a prerequisite for automated log analysis tasks such as anomaly detection, troubleshooting, and root cause analysis. Existing online parsing algorithms are susceptible to log drift, where slight log changes create false positives that drown out real anomalies.
  arXiv  Detail & Related papers  (2024-08-15T17:54:31Z)
• LogUpdater: Automated Detection and Repair of Specific Defects in Logging Statements [29.631530836349505]
  Developers use logging statements to track software runtime behaviors and system status.<n>Unclear or misleading logs can hide true execution patterns and hinder software maintenance.<n>We conduct a study to identify four logging statement defect types by analyzing log-centric changes.<n>We introduce LogUpdater, a framework for automatically detecting and updating these log defects.
  arXiv  Detail & Related papers  (2024-08-06T11:04:37Z)
• Generating Benchmarks for Factuality Evaluation of Language Models [61.69950787311278]
  We propose FACTOR: Factual Assessment via Corpus TransfORmation, a scalable approach for evaluating LM factuality. FACTOR automatically transforms a factual corpus of interest into a benchmark evaluating an LM's propensity to generate true facts from the corpus vs. similar but incorrect statements. We show that: (i) our benchmark scores increase with model size and improve when the LM is augmented with retrieval; (ii) benchmark score and perplexity do not always agree on model ranking; (iii) when perplexity and benchmark score disagree, the latter better reflects factuality in open-ended generation.
  arXiv  Detail & Related papers  (2023-07-13T17:14:38Z)
• Exploring the Effectiveness of LLMs in Automated Logging Generation: An Empirical Study [32.53659676826846]
  This paper performs the first study on exploring large language models (LLMs) for logging statement generation. We first build a logging statement generation dataset, LogBench, with two parts: (1) LogBench-O: logging statements collected from GitHub repositories, and (2) LogBench-T: the transformed unseen code from LogBench-O.
  arXiv  Detail & Related papers  (2023-07-12T06:32:51Z)
• LLMs as Factual Reasoners: Insights from Existing Benchmarks and Beyond [135.8013388183257]
  We propose a new protocol for inconsistency detection benchmark creation and implement it in a 10-domain benchmark called SummEdits. Most LLMs struggle on SummEdits, with performance close to random chance. The best-performing model, GPT-4, is still 8% below estimated human performance.
  arXiv  Detail & Related papers  (2023-05-23T21:50:06Z)
• Data-Driven Approach for Log Instruction Quality Assessment [59.04636530383049]
  There are no widely adopted guidelines on how to write log instructions with good quality properties. We identify two quality properties: 1) correct log level assignment assessing the correctness of the log level, and 2) sufficient linguistic structure assessing the minimal richness of the static text necessary for verbose event description. Our approach correctly assesses log level assignments with an accuracy of 0.88, and the sufficient linguistic structure with an F1 score of 0.99, outperforming the baselines.
  arXiv  Detail & Related papers  (2022-04-06T07:02:23Z)
• Self-Supervised Log Parsing [59.04636530383049]
  Large-scale software systems generate massive volumes of semi-structured log records. Existing approaches rely on log-specifics or manual rule extraction. We propose NuLog that utilizes a self-supervised learning model and formulates the parsing task as masked language modeling.
  arXiv  Detail & Related papers  (2020-03-17T19:25:25Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.