Agentic and Generative AI for Open-Source Intelligence and Cyber Investigations: Taxonomy, Evaluation, Challenges, and Future Directions

• URL: http://arxiv.org/abs/2607.03233v1
• Date: Fri, 03 Jul 2026 11:42:29 GMT
• Title: Agentic and Generative AI for Open-Source Intelligence and Cyber Investigations: Taxonomy, Evaluation, Challenges, and Future Directions
• Abstract summary: Large language models (LLMs) and agentic AI systems have emerged as promising solutions.<n>This survey systematically reviews 74 studies and makes four contributions.
• Score: 24.28151698249644
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The rapid growth of publicly available digital information has rendered manual open-source intelligence (OSINT) analysis insufficient for modern intelligence, cybersecurity, and cyber investigation. Large language models (LLMs) and agentic AI systems, capable of tool use, multi-step reasoning, and iterative intelligence generation, have emerged as promising solutions, yet evaluation frameworks have not kept pace with reported capabilities. This survey systematically reviews 74 studies and makes four contributions. First, it establishes agentic AI as a distinct analytical category rather than an extension of LLM prompting, organising the literature through an 11-category taxonomy covering LLM foundations, agentic architectures, retrieval-augmented generation (RAG), knowledge graphs, prompt engineering, domain adaptation, evaluation benchmarks, and risk. Second, it identifies the hallucination-validation gap as a corpus-level finding: although hallucination is recognised as a major reliability concern in over twenty studies, end-to-end hallucination is empirically measured in only one OSINT-specific RAG-based system, non-reproducible conditions, while related reasoning and factual-correction studies evaluate general-domain question answering rather than OSINT. Third, it maps existing research to the OSINT lifecycle, showing strong support for collection and analysis but limited coverage of verification, reporting, dissemination, and decision support. Fourth, it derives a ten-point research agenda addressing evaluation, benchmarking, hallucination measurement, adversarial robustness, dark-web coverage, multimodal intelligence, and governance. It concludes that a human-AI co-pilot model, where LLMs assist collection and triage while analysts retain responsibility for verification and decision-making, represents the most defensible near-term deployment architecture.

Related papers

• When Agentic AI Meets Integrated Sensing and Communication [23.084474668808912]
  Agentic artificial intelligence (AI) is transforming Integrated Sensing and Communication (ISAC) into a goal-driven, closed-loop intelligent system.<n>This survey unifies the literature within a six-stage closed-loop framework.<n>It introduces five levels of agentic maturity, ranging from physical-layer primitives to fully closed-loop ISAC.
  arXiv  Detail & Related papers  (2026-08-06T09:26:49Z)
• LLM-Based Scientific Peer Review: Methods, Benchmarks, and Reliability Challenges [4.655159257282136]
  This survey offers a systems-level analysis of large language models (LLMs)-based scientific peer review.<n>We focus on two core evaluative functions: critique generation and score prediction.<n>We identify emerging risks, including prompt injection, data poisoning, retrieval vulnerabilities, and reward hacking.
  arXiv  Detail & Related papers  (2026-06-23T18:12:39Z)
• AI-Driven Test Case Generation from Natural Language Requirements: A Survey of Techniques and Research Gaps [0.0]
  Software testing is critical for verifying that systems meet specified requirements, yet remains among the most time-consuming and expensive activities in development.<n>Recent advances in AI, natural language processing (NLP), and large language models (LLMs) have made automating this pipeline increasingly feasible.<n>This survey addresses four research questions: what AI and NLP techniques have been proposed for generating test cases from natural language requirements; what tools and frameworks support these approaches; and what research gaps remain.
  arXiv  Detail & Related papers  (2026-06-04T15:07:23Z)
• ForeSci: Evaluating LLM Agents for Forward-Looking AI Research Judgment [22.38716170940567]
  We introduce ForeSci, a benchmark for evaluating whether research agents can make forward-looking judgements from historical evidence.<n>ForeSci contains 500 tasks across four fast-moving AI domains and four decision families.<n>We show that explicit evidence organization improves traceability and factual support, but gains depend strongly on the decision family.
  arXiv  Detail & Related papers  (2026-05-30T09:41:26Z)
• AutoResearch AI: Towards AI-Powered Research Automation for Scientific Discovery [184.2309450554096]
  AutoResearch is defined as the developmental spectrum of AI-powered scientific workflow automation.<n>We analyze how research systems redistribute control, evidence, execution, validation, and accountability across and organize the field.<n>We propose five evaluation dimensions--novelty, impact, reliability, and provenance--and show that AutoResearch autonomy is domain-conditioned.
  arXiv  Detail & Related papers  (2026-05-22T03:40:30Z)
• AI for Auto-Research: Roadmap & User Guide [107.0834449839233]
  We present an end-to-end analysis of AI across the complete research lifecycle.<n>We identify a sharp, stage-dependent boundary between reliable assistance and unreliable autonomy.<n>We show that greater automation can obscure rather than eliminate failure modes.
  arXiv  Detail & Related papers  (2026-05-18T17:08:26Z)
• Assistance to Autonomy: A Systematic Literature Review of Agentic AI across the Software Development Life Cycle [4.241377338556251]
  This study contributes a comprehensive characterization of the current literature on agentic systems in software product development.<n>It also provides a methodological contribution in the form of an AI-assisted tool to automate the screening phase in high-volume SLR domains.
  arXiv  Detail & Related papers  (2026-05-14T10:46:51Z)
• AutoResearchBench: Benchmarking AI Agents on Complex Scientific Literature Discovery [55.70879973230979]
  AutoResearchBench is a benchmark for autonomous scientific literature discovery.<n>Compared to previous benchmarks on agentic web browsing, AutoResearchBench is research-oriented.<n>Even the most powerful LLMs, despite having largely conquered general agentic web-browsing benchmarks such as BrowseComp, achieve only 9.39% accuracy on Deep Research and 9.31% IoU on Wide Research.
  arXiv  Detail & Related papers  (2026-04-28T06:05:17Z)
• Demystifying deep search: a holistic evaluation with hint-free multi-hop questions and factorised metrics [89.1999907891494]
  We present WebDetective, a benchmark of hint-free multi-hop questions paired with a controlled Wikipedia sandbox.<n>Our evaluation of 25 state-of-the-art models reveals systematic weaknesses across all architectures.<n>We develop an agentic workflow, EvidenceLoop, that explicitly targets the challenges our benchmark identifies.
  arXiv  Detail & Related papers  (2025-10-01T07:59:03Z)
• Explain Before You Answer: A Survey on Compositional Visual Reasoning [74.27548620675748]
  Compositional visual reasoning has emerged as a key research frontier in multimodal AI.<n>This survey systematically reviews 260+ papers from top venues (CVPR, ICCV, NeurIPS, ICML, ACL, etc.)<n>We then catalog 60+ benchmarks and corresponding metrics that probe compositional visual reasoning along dimensions such as grounding accuracy, chain-of-thought faithfulness, and high-resolution perception.
  arXiv  Detail & Related papers  (2025-08-24T11:01:51Z)
• Survey on Evaluation of LLM-based Agents [28.91672694491855]
  The emergence of LLM-based agents represents a paradigm shift in AI.<n>This paper provides the first comprehensive survey of evaluation methodologies for these increasingly capable agents.
  arXiv  Detail & Related papers  (2025-03-20T17:59:23Z)
• A Literature Review of Literature Reviews in Pattern Analysis and Machine Intelligence [51.26815896167173]
  We present a comprehensive tertiary analysis of PAMI reviews along three complementary dimensions.<n>Our analyses reveal distinctive organizational patterns as well as persistent gaps in current review practices.<n>Finally, our evaluation of state-of-the-art AI-generated reviews indicates encouraging advances in coherence and organization.
  arXiv  Detail & Related papers  (2024-02-20T11:28:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.