LLM-Enhanced Log Anomaly Detection: A Comprehensive Benchmark of Large Language Models for Automated System Diagnostics

• URL: http://arxiv.org/abs/2604.12218v1
• Date: Tue, 14 Apr 2026 02:51:48 GMT
• Title: LLM-Enhanced Log Anomaly Detection: A Comprehensive Benchmark of Large Language Models for Automated System Diagnostics
• Abstract summary: Large Language Models (LLMs) offer promising new approaches to log understanding.<n>LLMs demonstrate remarkablezero-shot capabilities without requiring any labeled training data.<n>Our findings provide actionable guidelines for choosing log anomaly detection methods.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: System log anomaly detection is critical for maintaining the reliability of large-scale software systems, yet traditional methods struggle with the heterogeneous and evolving nature of modern log data. Recent advances in Large Language Models (LLMs) offer promising new approaches to log understanding, but a systematic comparison of LLM-based methods against established techniques remains lacking. In this paper, we present a comprehensive benchmark study evaluating both LLM-based and traditional approaches for log anomaly detection across four widely-used public datasets: HDFS, BGL, Thunderbird, and Spirit. We evaluate three categories of methods: (1) classical log parsers (Drain, Spell, AEL) combined with machine learning classifiers, (2) fine-tuned transformer models (BERT, RoBERTa), and (3) prompt-based LLM approaches (GPT-3.5, GPT-4, LLaMA-3) in zero-shot and few-shot settings. Our experiments reveal that while fine-tuned transformers achieve the highest F1-scores (0.96-0.99), prompt-based LLMs demonstrate remarkablezero-shot capabilities (F1: 0.82-0.91) without requiring any labeled training data -- a significant advantage for real-world deployment where labeled anomalies are scarce. We further analyze the cost-accuracy trade-offs, latency characteristics, and failure modes of each approach. Our findings provide actionable guidelines for practitioners choosing log anomaly detection methods based on their specific constraints regarding accuracy, latency, cost, and label availability. All code and experimental configurations are publicly available to facilitate reproducibility.

Related papers

• The Regularizing Power of Language-Training Deepfake Detectors [57.529452351298495]
  Deepfake detection generally suffers from overfitting to low-level domain-specific artifacts.<n>We use a dual-encoder architecture, pairing a frozen specialist detector with a LoRA-tuned MLLM encoder.<n>We employ a reinforcement learning stage that encourages the model to generate descriptive reasoning before classifying.
  arXiv  Detail & Related papers  (2026-05-29T12:01:17Z)
• AnomalyGen: Enhancing Log-Based Anomaly Detection with Code-Guided Data Augmentation [42.87177529900358]
  AnomalyGen is a novel framework that augments training data by synthesizing labeled log sequences from source code.<n> Evaluations on HDFS and Zookeeper across 12 diverse anomaly detection models show AnomalyGen consistently improves performance.
  arXiv  Detail & Related papers  (2026-04-13T07:24:43Z)
• Next-generation cyberattack detection with large language models: anomaly analysis across heterogeneous logs [0.0]
  This project explores large language models (LLMs) for anomaly detection across heterogeneous log sources.<n>Traditional intrusion detection systems suffer from high false positive rates, semantic blindness, and data scarcity.<n>We address these challenges through three contributions: (1) LogAtlas-Foundation-Sessions and LogAtlas-Defense-Set, balanced and heterogeneous log datasets with explicit attack annotations and privacy preservation.
  arXiv  Detail & Related papers  (2026-02-06T15:31:58Z)
• LLM-Assisted Logic Rule Learning: Scaling Human Expertise for Time Series Anomaly Detection [0.9740025522928777]
  Time series anomaly detection is critical for supply chain management to take proactive operations.<n>We propose a framework that leverages large language models (LLMs) to systematically encode human expertise into interpretable, logic-based rules.
  arXiv  Detail & Related papers  (2026-01-27T06:37:37Z)
• Generality Is Not Enough: Zero-Label Cross-System Log-Based Anomaly Detection via Knowledge-Level Collaboration [10.873294740040912]
  GeneralLog is a novel collaborative method for zero-label cross-system log anomaly detection.<n>GeneralLog achieves over 90% F1-score under a fully zero-label setting, significantly outperforming existing methods.
  arXiv  Detail & Related papers  (2025-11-08T06:47:28Z)
• When Punctuation Matters: A Large-Scale Comparison of Prompt Robustness Methods for LLMs [55.20230501807337]
  We present the first systematic evaluation of 5 methods for improving prompt robustness within a unified experimental framework.<n>We benchmark these techniques on 8 models from Llama, Qwen and Gemma families across 52 tasks from Natural Instructions dataset.
  arXiv  Detail & Related papers  (2025-08-15T10:32:50Z)
• GLiClass: Generalist Lightweight Model for Sequence Classification Tasks [49.2639069781367]
  We propose GLiClass, a novel method that adapts the GLiNER architecture for sequence classification tasks.<n>Our approach achieves strong accuracy and efficiency comparable to embedding-based methods, while maintaining the flexibility needed for zero-shot and few-shot learning scenarios.
  arXiv  Detail & Related papers  (2025-08-11T06:22:25Z)
• Anomaly Detection of Tabular Data Using LLMs [54.470648484612866]
  We show that pre-trained large language models (LLMs) are zero-shot batch-level anomaly detectors. We propose an end-to-end fine-tuning strategy to bring out the potential of LLMs in detecting real anomalies.
  arXiv  Detail & Related papers  (2024-06-24T04:17:03Z)
• Log-based Anomaly Detection based on EVT Theory with feedback [31.949892354842525]
  We present an accurate, lightweight, and adaptive log-based anomaly detection framework, referred to as SeaLog. Our method introduces a Trie-based Detection Agent (TDA) that employs a lightweight, dynamically-growing trie structure for real-time anomaly detection. To enhance TDA's accuracy in response to evolving log data, we enable it to receive feedback from experts.
  arXiv  Detail & Related papers  (2023-06-08T08:34:58Z)
• PULL: Reactive Log Anomaly Detection Based On Iterative PU Learning [58.85063149619348]
  We propose PULL, an iterative log analysis method for reactive anomaly detection based on estimated failure time windows. Our evaluation shows that PULL consistently outperforms ten benchmark baselines across three different datasets.
  arXiv  Detail & Related papers  (2023-01-25T16:34:43Z)
• Prompt-driven efficient Open-set Semi-supervised Learning [52.30303262499391]
  Open-set semi-supervised learning (OSSL) has attracted growing interest, which investigates a more practical scenario where out-of-distribution (OOD) samples are only contained in unlabeled data. We propose a prompt-driven efficient OSSL framework, called OpenPrompt, which can propagate class information from labeled to unlabeled data with only a small number of trainable parameters.
  arXiv  Detail & Related papers  (2022-09-28T16:25:08Z)
• LogLAB: Attention-Based Labeling of Log Data Anomalies via Weak Supervision [63.08516384181491]
  We present LogLAB, a novel modeling approach for automated labeling of log messages without requiring manual work by experts. Our method relies on estimated failure time windows provided by monitoring systems to produce precise labeled datasets in retrospect. Our evaluation shows that LogLAB consistently outperforms nine benchmark approaches across three different datasets and maintains an F1-score of more than 0.98 even at large failure time windows.
  arXiv  Detail & Related papers  (2021-11-02T15:16:08Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.