LLMs in the SOC: An Empirical Study of Human-AI Collaboration in Security Operations Centres

• URL: http://arxiv.org/abs/2508.18947v2
• Date: Fri, 19 Sep 2025 06:37:16 GMT
• Title: LLMs in the SOC: An Empirical Study of Human-AI Collaboration in Security Operations Centres
• Abstract summary: The integration of Large Language Models into Security Operations Centres (SOCs) presents a transformative, yet still evolving, opportunity to reduce analyst workload.<n>We present a longitudinal study of 3,090 analyst queries from 45 SOC analysts over 10 months.<n>Our analysis reveals that analysts use LLMs as on-demand aids for sensemaking and context-building, rather than for making high-stakes determinations.
• Score: 15.150052598613408
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: The integration of Large Language Models (LLMs) into Security Operations Centres (SOCs) presents a transformative, yet still evolving, opportunity to reduce analyst workload through human-AI collaboration. However, their real-world application in SOCs remains underexplored. To address this gap, we present a longitudinal study of 3,090 analyst queries from 45 SOC analysts over 10 months. Our analysis reveals that analysts use LLMs as on-demand aids for sensemaking and context-building, rather than for making high-stakes determinations, preserving analyst decision authority. The majority of queries are related to interpreting low-level telemetry (e.g., commands) and refining technical communication through short (1-3 turn) interactions. Notably, 93% of queries align with established cybersecurity competencies (NICE Framework), underscoring the relevance of LLM use for SOC-related tasks. Despite variations in tasks and engagement, usage trends indicate a shift from occasional exploration to routine integration, with growing adoption and sustained use among a subset of analysts. We find that LLMs function as flexible, on-demand cognitive aids that augment, rather than replace, SOC expertise. Our study provides actionable guidance for designing context-aware, human-centred AI assistance in security operations, highlighting the need for further in-the-wild research on real-world analyst-LLM collaboration, challenges, and impacts.

Related papers

• From Chasing Ghosts to Missed Attacks: Perspectives and Perceptions of SOC Practitioners on LLM Integration, Risks, and Readiness [9.803413284017138]
  Security Operations Centers (SOCs) process large volumes of security events.<n>Recent advances in Large Language Models (LLMs) suggest potential benefits for security operations.<n>Despite concerns, practitioners express a strong willingness to adopt LLMs.
  arXiv  Detail & Related papers  (2026-08-01T13:47:44Z)
• SelfAI: Building a Self-Training AI System with LLM Agents [79.10991818561907]
  SelfAI is a general multi-agent platform that combines a User Agent for translating high-level research objectives into standardized experimental configurations.<n>An Experiment Manager orchestrates parallel, fault-tolerant training across heterogeneous hardware while maintaining a structured knowledge base for continuous feedback.<n>Across regression, computer vision, scientific computing, medical imaging, and drug discovery benchmarks, SelfAI consistently achieves strong performance and reduces redundant trials.
  arXiv  Detail & Related papers  (2025-11-29T09:18:39Z)
• LLM-Assisted Thematic Analysis: Opportunities, Limitations, and Recommendations [5.660100855602864]
  Large Language Models (LLMs) are increasingly used to assist qualitative research in Software Engineering (SE)<n>This study investigates how experienced SE researchers conceptualize the opportunities, risks, and methodological implications of integrating LLMs into thematic analysis.
  arXiv  Detail & Related papers  (2025-11-18T14:32:48Z)
• CTIArena: Benchmarking LLM Knowledge and Reasoning Across Heterogeneous Cyber Threat Intelligence [48.63397742510097]
  Cyber threat intelligence (CTI) is central to modern cybersecurity, providing critical insights for detecting and mitigating evolving threats.<n>With the natural language understanding and reasoning capabilities of large language models (LLMs), there is increasing interest in applying them to CTI.<n>We present CTIArena, the first benchmark for evaluating LLM performance on heterogeneous, multi-source CTI.
  arXiv  Detail & Related papers  (2025-10-13T22:10:17Z)
• Large Language Models for Security Operations Centers: A Comprehensive Survey [0.0]
  Large Language Models (LLMs) have emerged as powerful tools capable of understanding and generating human-like text.<n>This survey systematically explores the integration of generative AI and more specifically LLMs into SOC workflow.
  arXiv  Detail & Related papers  (2025-09-13T15:27:50Z)
• Bridging Expertise Gaps: The Role of LLMs in Human-AI Collaboration for Cybersecurity [17.780795900414716]
  This study investigates whether large language models (LLMs) can function as intelligent collaborators to bridge expertise gaps in cybersecurity decision-making.<n>We find that human-AI collaboration improves task performance,reducing false positives in phishing detection and false negatives in intrusion detection.
  arXiv  Detail & Related papers  (2025-05-06T04:47:52Z)
• An Overview of Large Language Models for Statisticians [109.38601458831545]
  Large Language Models (LLMs) have emerged as transformative tools in artificial intelligence (AI)<n>This paper explores potential areas where statisticians can make important contributions to the development of LLMs.<n>We focus on issues such as uncertainty quantification, interpretability, fairness, privacy, watermarking and model adaptation.
  arXiv  Detail & Related papers  (2025-02-25T03:40:36Z)
• Applications and Implications of Large Language Models in Qualitative Analysis: A New Frontier for Empirical Software Engineering [0.46426852157920906]
  The study emphasizes the need for structured strategies and guidelines to optimize LLM use in qualitative research within software engineering.<n>While LLMs show promise in supporting qualitative analysis, human expertise remains crucial for interpreting data, and ongoing exploration of best practices will be vital for their successful integration into empirical software engineering research.
  arXiv  Detail & Related papers  (2024-12-09T15:17:36Z)
• The LLM Effect: Are Humans Truly Using LLMs, or Are They Being Influenced By Them Instead? [60.01746782465275]
  Large Language Models (LLMs) have shown capabilities close to human performance in various analytical tasks. This paper investigates the efficiency and accuracy of LLMs in specialized tasks through a structured user study focusing on Human-LLM partnership.
  arXiv  Detail & Related papers  (2024-10-07T02:30:18Z)
• The Imperative of Conversation Analysis in the Era of LLMs: A Survey of Tasks, Techniques, and Trends [64.99423243200296]
  Conversation Analysis (CA) strives to uncover and analyze critical information from conversation data. In this paper, we perform a thorough review and systematize CA task to summarize the existing related work. We derive four key steps of CA from conversation scene reconstruction, to in-depth attribution analysis, and then to performing targeted training, finally generating conversations.
  arXiv  Detail & Related papers  (2024-09-21T16:52:43Z)
• Cooperate or Collapse: Emergence of Sustainable Cooperation in a Society of LLM Agents [101.17919953243107]
  GovSim is a generative simulation platform designed to study strategic interactions and cooperative decision-making in large language models (LLMs)<n>We find that all but the most powerful LLM agents fail to achieve a sustainable equilibrium in GovSim, with the highest survival rate below 54%.<n>We show that agents that leverage "Universalization"-based reasoning, a theory of moral thinking, are able to achieve significantly better sustainability.
  arXiv  Detail & Related papers  (2024-04-25T15:59:16Z)
• A Survey on Large Language Model based Autonomous Agents [105.2509166861984]
  Large language models (LLMs) have demonstrated remarkable potential in achieving human-level intelligence.<n>This paper delivers a systematic review of the field of LLM-based autonomous agents from a holistic perspective.<n>We present a comprehensive overview of the diverse applications of LLM-based autonomous agents in the fields of social science, natural science, and engineering.
  arXiv  Detail & Related papers  (2023-08-22T13:30:37Z)
• Sentiment Analysis in the Era of Large Language Models: A Reality Check [69.97942065617664]
  This paper investigates the capabilities of large language models (LLMs) in performing various sentiment analysis tasks. We evaluate performance across 13 tasks on 26 datasets and compare the results against small language models (SLMs) trained on domain-specific datasets.
  arXiv  Detail & Related papers  (2023-05-24T10:45:25Z)
• Can Large Language Models assist in Hazard Analysis? [1.599072005190786]
  Large Language Models (LLMs) have demonstrated remarkable natural language processing and generation capabilities. This paper explores the potential of integrating LLMs in the hazard analysis for safety-critical systems.
  arXiv  Detail & Related papers  (2023-03-25T19:43:27Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.