Empirical Software Engineering TerraProbe: A Layered-Oracle Framework for Detecting Deceptive Fixes in LLM-Assisted Terraform

• URL: http://arxiv.org/abs/2606.26590v1
• Date: Thu, 25 Jun 2026 04:21:15 GMT
• Title: Empirical Software Engineering TerraProbe: A Layered-Oracle Framework for Detecting Deceptive Fixes in LLM-Assisted Terraform
• Abstract summary: Security misconfigurations in Terraform Infrastructure-as-Code are a growing risk in cloud deployments.<n>Existing evaluations often treat a repair as successful when the targeted static-analysis finding disappears.<n>This paper presents TerraProbe, a five-layer oracle framework for evaluating LLM-assisted Terraform security repair.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Security misconfigurations in Terraform Infrastructure-as-Code are a growing risk in cloud deployments, and large language models are increasingly used as automated repair agents. Existing evaluations often treat a repair as successful when the targeted static-analysis finding disappears, without checking planning validity, behavioral change, or security intent. This paper presents TerraProbe, a five-layer oracle framework for evaluating LLM-assisted Terraform security repair. We apply TerraProbe to 288 first-pass repairs generated by gemini-2.5-flash-lite, GPT-4o, and Claude 3.5 Sonnet across 68 real-world TerraDS modules and 28 controlled injected-defect modules. The results show that targeted Checkov removal overstates repair success. Although targeted removal reaches 83.3 percent for the primary model, full-scanner cleanliness drops to 10.4 percent, Terraform planning succeeds for 39.6 percent, and plan comparison is reachable for 38.5 percent. Human adjudication further shows that 71.4 percent of plan-compared real-world repairs are deceptive fixes that pass automated checks while leaving the underlying vulnerability in place. This pattern is statistically indistinguishable across the three models, with deceptive-fix rates from 57.1 percent to 71.4 percent and pairwise Fisher exact p-values above 0.10. The paper introduces a four-dimensional taxonomy of deceptive fixes, validated with Cohen kappa of 0.78 and Krippendorff alpha of 0.76. IAM permission analysis confirms that wildcard Resource grants persist in all nine CKV2 AWS 11 deceptive-fix cases. TerraProbe contributes an evaluation methodology, a replication package, and the Multi-Layer Oracle Evaluation framework for distinguishing intent-aligned security repairs from scanner-passing false successes.

Related papers

• BenchShield: Formal Model-Backed Instrumentation for Reward Integrity in LLM-Agent Evaluation Infrastructure [47.61632934139226]
  LM-agent benchmarks increasingly function as interactive evaluation infrastructure.<n>This interactivity makes evaluations vulnerable to reward hacking.<n>This paper presents BenchShield, a model-backed instrumentation layer for reward integrity.
  arXiv  Detail & Related papers  (2026-09-10T03:10:58Z)
• IdeaAMBIG: Benchmarking Implementation-Critical Gaps in Research-Idea Specifications [52.570108663867046]
  A research idea may be novel, coherent, and scientifically plausible, yet its proposed method may remain insufficiently specified for faithful implementation.<n>We study the codification readiness of implementation-facing research-method specifications, defined by whether they provide sufficient methodological information for a competent implementer or coding agent to construct the intended method without unsupported assumptions.<n>We introduce IdeaAMBIG, a benchmark of 660 evidence-grounded instances: 163 real-world gaps from reports and GitHub issues, and 497 controlled synthetic gaps injected into codification-ready references.
  arXiv  Detail & Related papers  (2026-09-09T17:59:04Z)
• Benchmarking Automated Security Patch Backporting: How Far Are We? [15.423124944381081]
  Porting Benchmark is a curated dataset of 1,234 security patch backporting cases.<n>Our results show that aligned evaluation changes the apparent performance landscape.<n>We identify four root-cause categories (missing target API awareness, cross-version semantic mismatch, non-local dependency propagation failure, and patch construction or localization failure)
  arXiv  Detail & Related papers  (2026-08-18T11:43:10Z)
• Apodex Discovery: Reality Benchmarks and Environments for Evaluating and Building Discoverative Artificial Intelligence [86.61561123749011]
  We introduce Apodex Discovery, a framework for building and evaluating discoverative AI through the heavy-duty solver.<n>First, a problem-scouting process surveyed 561 industries across 16 sectors, assembled 423 high-value real-world problems, and selected 20 for the initial release.<n>Second, a common environment-task-episode abstraction provides data, tools, constraints, feedback, trajectory recording, and verification of intermediate artifacts and final submissions.
  arXiv  Detail & Related papers  (2026-08-11T18:48:40Z)
• Efficient Visual Pointing for Embodied AI:Agent-Driven Data Synthesis, Cross-Block Attention, and Iterative Correction [55.11480729304395]
  We describe our PointArena 2026 solution, which achieves 77.2% overall accuracy and ranks second on the benchmark.<n>The ap proach targets three failure modes. First, agent-driven syn thesis builds large semantic and anchor-relative candidate pools.<n>Second, a determinis tic steerable-data pipeline creates a verified 10,000-sample main set, plus reserve samples, using masks, templates, and path verification.
  arXiv  Detail & Related papers  (2026-06-29T06:39:03Z)
• Paired-CSLiDAR: Height-Stratified Registration for Cross-Source Aerial-Ground LiDAR Pose Refinement [46.13526676100776]
  We introduce Paired-CSLiDAR, a cross-source aerial-ground LiDAR benchmark for single-scan pose refinement.<n>The benchmark contains 12,683 ground-aerial pairs across 6 evaluation sites.<n>We propose Residual-Guided Stratified Registration (RGSR), a training-free, geometry-only refinement pipeline.
  arXiv  Detail & Related papers  (2026-05-01T13:14:20Z)
• Towards Optimal Agentic Architectures for Offensive Security Tasks [3.93181912653522]
  We introduce a benchmark of 20 interactive targets (10 web/API and 10 binary)<n>Each exposing one-reachable ground-truth vulnerability, evaluated in whitebox and blackbox modes.<n> MAS-Indep attains the highest validated detection rate (64.2%), while SAS is the strongest efficiency baseline at $0.058 per validated finding.
  arXiv  Detail & Related papers  (2026-04-20T18:17:51Z)
• Broken by Default: A Formal Verification Study of Security Vulnerabilities in AI-Generated Code [0.0]
  Broken by Default is a formal verification study of 3,500 code artifacts generated by seven widely-deployed LLMs across 500 security-critical prompts.<n>Across all models, 55.8% of artifacts contain at least one COBALT-identified vulnerability; of these, 1,055 are formally proven via Z3 satisfiability witnesses.<n>GPT-4o leads at 62.4% (grade F); Gemini 2.5 Flash performs best at 48.4% (grade D)
  arXiv  Detail & Related papers  (2026-04-07T00:55:42Z)
• Your Agent, Their Asset: A Real-World Safety Analysis of OpenClaw [87.97230960702274]
  We present the first real-world safety evaluation of OpenClaw.<n>We introduce the CIK taxonomy, which unifies an agent's persistent state into three dimensions.<n>Our evaluations cover 12 attack scenarios on a live OpenClaw instance.
  arXiv  Detail & Related papers  (2026-04-06T15:27:05Z)
• Why LLMs Fail: A Failure Analysis and Partial Success Measurement for Automated Security Patch Generation [0.0]
  This study analyzes 319 Large Language Models (LLMs)generated security patches across 64 Java vulnerabilities from the Vul4J benchmark.<n>Using tri-axis evaluation, the analysis reveals that only 24.8% of patches achieve full correctness, while 51.4% fail both security and functionality.<n>The proposed Security Repair Score (SRS) quantifies this gap, showing LLMs preserve functionality (mean 0.832) but struggle with security (mean 0.251)
  arXiv  Detail & Related papers  (2026-03-10T05:34:56Z)
• VulnRepairEval: An Exploit-Based Evaluation Framework for Assessing Large Language Model Vulnerability Repair Capabilities [41.85494398578654]
  VulnRepairEval is an evaluation framework anchored in functional Proof-of-Concept exploits.<n>Our framework delivers a comprehensive, containerized evaluation pipeline that enables reproducible differential assessment.
  arXiv  Detail & Related papers  (2025-09-03T14:06:10Z)
• VADER: A Human-Evaluated Benchmark for Vulnerability Assessment, Detection, Explanation, and Remediation [0.8087612190556891]
  VADER comprises 174 real-world software vulnerabilities, each carefully curated from GitHub and annotated by security experts.<n>For each vulnerability case, models are tasked with identifying the flaw, classifying it using Common Weaknession (CWE), explaining its underlying cause, proposing a patch, and formulating a test plan.<n>Using a one-shot prompting strategy, we benchmark six state-of-the-art LLMs (Claude 3.7 Sonnet, Gemini 2.5 Pro, GPT-4.1, GPT-4.5, Grok 3 Beta, and o3) on VADER.<n>Our results show that current state-of-the-
  arXiv  Detail & Related papers  (2025-05-26T01:20:44Z)
• BountyBench: Dollar Impact of AI Agent Attackers and Defenders on Real-World Cybersecurity Systems [62.17474934536671]
  We introduce the first framework to capture offensive and defensive cyber-capabilities in evolving real-world systems.<n>To capture the vulnerability lifecycle, we define three task types: Detect (detecting a new vulnerability), Exploit (exploiting a specific vulnerability), and Patch (patching a specific vulnerability)<n>We evaluate 8 agents: Claude Code, OpenAI Codex CLI with o3-high and o4-mini, and custom agents with o3-high, GPT-4.1, Gemini 2.5 Pro Preview, Claude 3.7 Sonnet Thinking, and DeepSeek-R1.
  arXiv  Detail & Related papers  (2025-05-21T07:44:52Z)
• Certified Defences Against Adversarial Patch Attacks on Semantic Segmentation [44.13336566131961]
  We present Demasked Smoothing, the first approach to certify the robustness of semantic segmentation models against patch attacks. Using different masking strategies, Demasked Smoothing can be applied both for certified detection and certified recovery. In extensive experiments we show that Demasked Smoothing can on average certify 64% of the pixel predictions for a 1% patch in the detection task and 48% against a 0.5% patch for the recovery task on the ADE20K dataset.
  arXiv  Detail & Related papers  (2022-09-13T13:24:22Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.