ReCon: A Resource-Constrained Benchmark for LLM-Based Cybersecurity Compliance Across Ingestion and Retrieval Pipelines

• URL: http://arxiv.org/abs/2607.22885v1
• Date: Fri, 24 Jul 2026 19:53:00 GMT
• Title: ReCon: A Resource-Constrained Benchmark for LLM-Based Cybersecurity Compliance Across Ingestion and Retrieval Pipelines
• Abstract summary: Governance failure is one of the major reasons for a weakened cybersecurity posture.<n>The policy document must be in compliance with international or national standards.<n>It is often observed that policy documents are often incomplete with respect to industry standards or regulations.
• Score: 0.6773121102591491
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: With the increasingly aggressive cyber threat landscape for governments, businesses, and institutions, as information and/or cybersecurity implementations are increasingly under scrutiny by regulators, it has been pointed out that governance failure is one of the major reasons for a weakened cybersecurity posture. A major component of Cyber/information security governance is the development, adoption, and implementation of a comprehensive information and/or cyber security policy document. The policy document must be in compliance with international or national standards and, if possible, with regulatory guidelines. However, it is often observed that policy documents are often incomplete with respect to industry standards or regulations and require revision when subjected to a thorough audit. Identifying the gaps between the controls and processes documented in the policy and those required in the regulations or standards necessitates extensive manual effort. The advent of Generative AI tools such as Large Language Models (LLMs) led to use of LLMs and Agentic AI tools to automate such compliance checks, as seen in a few research publications in recent times. However, such reported use of LLMs are experimented with high resource environments such as expensive GPUs and memory based servers. For smaller organizations such expensive compute platform may not be easily available. In this article, we benchmark the compliance checking tasks on LLMs that do not require GPU and high memory usage and the effectiveness of such resource constrained LLMs in compliance checking. Our experiments demonstrated that the low resource LLMs can provide good agreement/accuracy in compliance checking of policy documents against standards by experimenting with ISO 27002:2022 controls against multiple policy documents.

Related papers

• Evaluating Large Language Model Performance on International Maritime Dangerous Goods Code Compliance [47.37396434232136]
  This paper introduces DGEval, the first benchmark for evaluating knowledge of IMDG Amendment 42-24.<n>It comprises 1,678 questions across multiple-choice, open-ended, DGL lookup, and regulatory identification tasks.<n>We evaluate 13 models from six providers across multiple thinking configurations.
  arXiv  Detail & Related papers  (2026-08-21T12:32:31Z)
• ContextLens: Modeling Imperfect Privacy and Safety Context for Legal Compliance [49.524070843587594]
  We propose ContextLens, a semi-rule-based framework that leverages large language models (LLMs) to ground input context in the legal domain.<n>We conduct experiments on existing compliance benchmarks that cover the General Data Protection Regulation () and the EU AI Act.<n>The results suggest that our ContextLens can significantly improve LLMs' compliance assessment and surpass existing baselines without any training.
  arXiv  Detail & Related papers  (2026-04-14T05:35:38Z)
• Frontier AI Auditing: Toward Rigorous Third-Party Assessment of Safety and Security Practices at Leading AI Companies [57.521647436515785]
  We define frontier AI auditing as rigorous third-party verification of frontier AI developers' safety and security claims.<n>We introduce AI Assurance Levels (AAL-1 to AAL-4), ranging from time-bounded system audits to continuous, deception-resilient verification.
  arXiv  Detail & Related papers  (2026-01-16T18:44:09Z)
• Towards Verifiably Safe Tool Use for LLM Agents [53.55621104327779]
  Large language model (LLM)-based AI agents extend capabilities by enabling access to tools such as data sources, APIs, search engines, code sandboxes, and even other agents.<n>LLMs may invoke unintended tool interactions and introduce risks, such as leaking sensitive data or overwriting critical records.<n>Current approaches to mitigate these risks, such as model-based safeguards, enhance agents' reliability but cannot guarantee system safety.
  arXiv  Detail & Related papers  (2026-01-12T21:31:38Z)
• Generation of Programmatic Rules for Document Forgery Detection Using Large Language Models [10.32461766065764]
  Document forgery poses a growing threat to legal, economic, and governmental processes.<n>Existing plausibility checks are manually implemented by software engineers, which is time-consuming.<n>Recent advances in code generation with large language models (LLMs) offer new potential for automating and scaling the generation of these checks.
  arXiv  Detail & Related papers  (2025-12-22T10:08:25Z)
• Large Language Models as a (Bad) Security Norm in the Context of Regulation and Compliance [0.0]
  This paper asks how AI can negatively affect cybersecurity and law when used for the design and deployment of security infrastructure.<n>The paper discusses the use of Large Language Models (LLM) in security, either directly or indirectly, and briefly tackles other types of AI.<n>It then lists norms in cybersecurity, then a range of legal cybersecurity obligations from the European Union, to create a frame of reference.
  arXiv  Detail & Related papers  (2025-12-18T11:14:21Z)
• Executable Governance for AI: Translating Policies into Rules Using LLMs [1.388831902854619]
  Policy-to-Tests (P2T) is a framework that converts natural policy documents into normalized, machine-readable rules.<n>To test the framework beyond a single policy, we apply it across general frameworks, sector guidance, and enterprise standards.<n>These AI-generated rules closely match strong human baselines on span-level and rule-level metrics, with robust inter-annotator agreement on the gold set.
  arXiv  Detail & Related papers  (2025-12-04T03:11:54Z)
• A.S.E: A Repository-Level Benchmark for Evaluating Security in AI-Generated Code [49.009041488527544]
  A.S.E is a repository-level evaluation benchmark for assessing the security of AI-generated code.<n>Current large language models (LLMs) still struggle with secure coding.<n>A larger reasoning budget does not necessarily lead to better code generation.
  arXiv  Detail & Related papers  (2025-08-25T15:11:11Z)
• From Legal Text to Tech Specs: Generative AI's Interpretation of Consent in Privacy Law [0.0]
  This study employs a three-step pipeline that involves using an LLM to classify software use cases for compliance.<n>By benchmarking LLMs against real-world use cases, this research provides insights into leveraging AI-driven solutions to enhance legal compliance of software.
  arXiv  Detail & Related papers  (2025-07-05T23:36:05Z)
• Automating Security Audit Using Large Language Model based Agent: An Exploration Experiment [7.039114328812774]
  Security audits help to maintain a strong security posture by ensuring that policies are in place, controls are implemented, gaps are identified for cybersecurity risks mitigation.<n>This paper looks at the possibility of developing a framework to leverage Large Language Models (LLMs) as an autonomous agent to execute part of the security audit.
  arXiv  Detail & Related papers  (2025-05-15T22:22:52Z)
• Distributed Machine Learning and the Semblance of Trust [66.1227776348216]
  Federated Learning (FL) allows the data owner to maintain data governance and perform model training locally without having to share their data. FL and related techniques are often described as privacy-preserving. We explain why this term is not appropriate and outline the risks associated with over-reliance on protocols that were not designed with formal definitions of privacy in mind.
  arXiv  Detail & Related papers  (2021-12-21T08:44:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.