How Compliant Are GitHub Actions Workflows? A Checklist-Based Study with LLM-Assisted Auditing

• URL: http://arxiv.org/abs/2605.02091v1
• Date: Sun, 03 May 2026 23:21:13 GMT
• Title: How Compliant Are GitHub Actions Workflows? A Checklist-Based Study with LLM-Assisted Auditing
• Abstract summary: GitHub Actions (GHA) CI are critical infrastructure, but current tooling offers only syntactic or checks.<n>This paper proposes a novel, documentation-grounded GHA compliance checklist with 30 criteria spanning four workflow sections and eight themes.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: GitHub Actions (GHA) CI workflows are critical infrastructure, but current tooling offers only syntactic or heuristic checks and does not enforce documented best practices for security, maintainability, or performance. Consequently, issues like over-privileged permissions, weak secrets management, and missing failure notifications remain undetected in real-world pipelines. This paper proposes a novel, documentation-grounded GHA compliance checklist with 30 criteria spanning four workflow sections and eight themes, and assesses Large Language Models (LLMs) for scalable compliance auditing. On 95 real-world Java workflows (2,850 assessments) using four open-weight LLMs, we find only fair agreement (Fleiss' kappa = 0.28), with systematic disagreement on structural reasoning and security-sensitive judgments. To address this, we introduce a multi-tier adjudication framework in which GPT 5 resolves model conflicts before targeted manual review, reducing verification effort by 81% while retaining 87% agreement with expert judgment. At scale, it reveals major compliance gaps: overall compliance is 28%, dropping to 4% for permission controls; Security (26%) lags far behind Clarity (68%). Our results show that LLMs enable scalable compliance measurement but cannot replace experts, highlighting the need for hybrid human-AI auditing and providing empirical benchmarks and guidance for defensible GHA workflow audits.

Related papers

• Knowledge Synthesis Review Framework: Task-Level Benchmarking of LLM-Based Systems for Multi-Source Evidence Synthesis [0.0]
  Large language models (LLMs) may accelerate this work, but their reliability across cognitive tasks of a review remains uncertain.<n>We introduce the Knowledge Synthesis Review (KSR), a human-in-the-loop framework that decomposes evidence synthesis into screening, extraction, analysis, and synthesis.<n>We evaluated GPT-5, Claude Sonnet 4, Gemini 2.5 Pro, and NotebookLM on a 244-document benchmark subset drawn from a 1,893-document corpus on AI.
  arXiv  Detail & Related papers  (2026-08-13T02:41:14Z)
• EnterpriseRAG: Benchmarking LLM Instruction Adherence and Robustness under Non-Ideal Enterprise Retrieval [41.53357936032648]
  Existing benchmarks assume clean retrieval with simple queries, failing to capture production conditions where noisy documents and multi-dimensional constraints coexist.<n>We introduce EnterpriseRAG, a benchmark of 983 expert-validated samples across six domains.<n>We will release the benchmark and evaluation framework upon publication.
  arXiv  Detail & Related papers  (2026-08-12T02:51:04Z)
• ACCORD: Action-Conditioned Contextual Grounding for Language Agents [69.73525608707764]
  ACCORD (Action-Conditioned Contextual Grounding) is an agent framework for adaptive grounding.<n>We show that current agents often fail to do so. They act from assumed rather than observed specifics, overlook information they could have gathered, and fail to incorporate evidence that has already been returned.
  arXiv  Detail & Related papers  (2026-06-15T09:05:55Z)
• Decoupled Smart Contract Audits: Lightweight LLM Framework via Distillation and Aggregation [0.5649790777986989]
  We introduce an efficient end-to-end smart contract security audit framework utilizing lightweight, highly optimized open-source LLMs.<n>Our framework decouples comprehensive audit tasks into four interconnected components: vulnerability detection, explanation, severity classification, and remediation recommendation.<n> Experimental results demonstrate that our lightweight pipeline consistently outperforms state-of-the-art open-source coder dense LLMs.
  arXiv  Detail & Related papers  (2026-06-02T04:13:43Z)
• Social Bias in LLM-Generated Code: Benchmark and Mitigation [10.39413030802123]
  Large Language Models (LLMs) are increasingly deployed to generate code for human-centered applications where demographic fairness is critical.<n>Existing evaluations focus almost exclusively on functional correctness, leaving social bias in LLM-generated code largely unexamined.<n>We propose the Fairness Monitor Agent (FMA), a modular component that plugs into any existing code generation pipeline without modifying it.
  arXiv  Detail & Related papers  (2026-05-01T04:06:02Z)
• Measuring LLM Trust Allocation Across Conflicting Software Artifacts [4.974612243038232]
  We present TRACE, a framework that elicits structured artifact-level trust traces over Javadoc, method signatures, implementations, and test prefixes.<n>We evaluate per-artifact quality assessment, inconsistency detection, affected artifact attribution, and source prioritization.
  arXiv  Detail & Related papers  (2026-04-03T20:38:14Z)
• HLE-Verified: A Systematic Verification and Structured Revision of Humanity's Last Exam [63.84155758655084]
  Humanity's Last Exam (HLE) has become a widely used benchmark for evaluating frontier large language models.<n>We introduce HLE-Verified, a verified and revised version of HLE with a transparent verification protocol and fine-grained error taxonomy.<n>We evaluate seven state-of-the-art language models on HLE and HLE-Verified, observing an average absolute accuracy gain of 7--10 percentage points.
  arXiv  Detail & Related papers  (2026-02-15T02:50:15Z)
• SPECA: Specification-to-Checklist Agentic Auditing for Multi-Implementation Systems -- A Case Study on Ethereum Clients [1.711666249985278]
  SPECA is a Specification-to-Checklist framework that turns normative requirements into checklists.<n>We instantiate SPECA in an in-the-wild security audit contest for the Fusaka upgrade, covering 11 production clients.<n>Our improved agent, evaluated against the ground truth of a competitive audit, achieved a strict recall of 27.3 percent on high-impact vulnerabilities.
  arXiv  Detail & Related papers  (2026-02-07T12:19:00Z)
• MMLU-CF: A Contamination-free Multi-task Language Understanding Benchmark [57.999567012489706]
  We propose a contamination-free and more challenging benchmark called MMLU-CF.<n>This benchmark reassesses LLMs' understanding of world knowledge by averting both unintentional and malicious data leakage.<n>Our evaluation of mainstream LLMs reveals that the powerful GPT-4o achieves merely a 5-shot score of 73.4% and a 0-shot score of 71.9% on the test set.
  arXiv  Detail & Related papers  (2024-12-19T18:58:04Z)
• Exploring Automatic Cryptographic API Misuse Detection in the Era of LLMs [60.32717556756674]
  This paper introduces a systematic evaluation framework to assess Large Language Models in detecting cryptographic misuses. Our in-depth analysis of 11,940 LLM-generated reports highlights that the inherent instabilities in LLMs can lead to over half of the reports being false positives. The optimized approach achieves a remarkable detection rate of nearly 90%, surpassing traditional methods and uncovering previously unknown misuses in established benchmarks.
  arXiv  Detail & Related papers  (2024-07-23T15:31:26Z)
• SORRY-Bench: Systematically Evaluating Large Language Model Safety Refusal [64.9938658716425]
  SORRY-Bench is a proposed benchmark for evaluating large language models' (LLMs) ability to recognize and reject unsafe user requests.<n>First, existing methods often use coarse-grained taxonomy of unsafe topics, and are over-representing some fine-grained topics.<n>Second, linguistic characteristics and formatting of prompts are often overlooked, like different languages, dialects, and more -- which are only implicitly considered in many evaluations.
  arXiv  Detail & Related papers  (2024-06-20T17:56:07Z)
• Flames: Benchmarking Value Alignment of LLMs in Chinese [86.73527292670308]
  This paper proposes a value alignment benchmark named Flames. It encompasses both common harmlessness principles and a unique morality dimension that integrates specific Chinese values. Our findings indicate that all the evaluated LLMs demonstrate relatively poor performance on Flames.
  arXiv  Detail & Related papers  (2023-11-12T17:18:21Z)
• LLMs as Factual Reasoners: Insights from Existing Benchmarks and Beyond [135.8013388183257]
  We propose a new protocol for inconsistency detection benchmark creation and implement it in a 10-domain benchmark called SummEdits. Most LLMs struggle on SummEdits, with performance close to random chance. The best-performing model, GPT-4, is still 8% below estimated human performance.
  arXiv  Detail & Related papers  (2023-05-23T21:50:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.