Knowledge Boundary Probing and Demand-Guided Intervention for LLM-Based Power System Code Generation

• URL: http://arxiv.org/abs/2605.31478v1
• Date: Fri, 29 May 2026 16:06:34 GMT
• Title: Knowledge Boundary Probing and Demand-Guided Intervention for LLM-Based Power System Code Generation
• Abstract summary: We show that first-pass failures in power-system code generation are dominated by structured API-knowledge boundary: hallucinated function names, misused parameters, and mishandled result tables.<n>We introduce PowerCodeBench, an execution-validated benchmark generator that pairs natural-language operator queries with pandapower code and numerical ground truth.<n>We also introduce an L0-L3 documentation-driven probing procedure that measures per-model API knowledge profiles.
• Score: 8.136015158683694
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Large language models (LLMs) are increasingly used to automate power-system analysis, but many utilities and energy-research labs require on-premise serving for confidentiality, regulatory, reproducibility, and cost reasons. This makes the reliability of open-weight models a deployment issue. We show that first-pass failures in power-system code generation are dominated not by reasoning alone, but by structured API-knowledge boundary errors: hallucinated function names, misused parameters, and mishandled result tables in versioned simulation libraries. We introduce PowerCodeBench, an execution-validated benchmark generator that pairs natural-language operator queries with pandapower code and numerical ground truth; an L0-L3 documentation-driven probing procedure that measures per-model API knowledge profiles; and a boundary-aware intervention that combines query-side API demand estimation with targeted proactive documentation injection and routed reactive correction. On a 2,000-task frozen release, we evaluate ten open-weight LLMs (1.5B-480B parameters) and four commercial mid-tier APIs. The intervention improves every evaluated open-weight model of at least 7B parameters and every commercial API by 32 to 56 accuracy points. Open-weight models in the 70B-120B range match the commercial mid-tier accuracy range, while Llama-3.1-405B and Qwen3-Coder-480B lead the panel. The targeted prompts preserve the full-context accuracy ceiling while using 41% of the prompt-token cost. The result is an accuracy-side, deployment-time path toward reliable on-premise LLM assistance for grid-analysis workflows without fine-tuning or cloud inference.

Related papers

• Neuro-Symbolic Software Verification: Hyper-charging Local Language Models with Symbolic Reasoning at Scale [11.088680803534785]
  We present VerIbmc, a neuro-symbolic pipeline that pairs symbolic invariant generation with locally deployable open-weight language models.<n>All inference runs entirely on a single local machine using open-weight models -- no cloud API or proprietary model is required.
  arXiv  Detail & Related papers  (2026-06-15T15:59:10Z)
• LiveFMBench: Unveiling the Power and Limits of Agentic Workflows in Specification Generation [75.05397479715576]
  Large language models (LLMs) and agents have shown promising progress, but their true capabilities and failure modes remain unclear.<n>We present the first systematic and contamination-aware study of LLM- and agent-based formal specification generation for C programs.
  arXiv  Detail & Related papers  (2026-05-02T11:31:33Z)
• Authority Inversion in LLM-Mediated Ubiquitous Systems: When Models Trust Users Over Sensors [6.414826816896125]
  We investigate how large language models implicitly allocate authority when sensor measurements and user claims conflict remain unexamined.<n>We find numerical sensor data fails to integrate into answer-relevant model directions, allowing natural-language claims to dominate the final decision.<n>We propose Geometric Authority (GAC) as an inference-time layer-level intervention to suppress misplaced user authority.
  arXiv  Detail & Related papers  (2026-04-28T04:59:03Z)
• Compiled AI: Deterministic Code Generation for LLM-Based Workflow Automation [3.225273674498579]
  We study compiled AI, a paradigm in which large language models generate executable code artifacts during a compilation phase, after which execute deterministically without further model invocation.<n>Our contribution is a systems-oriented study of its application to high-stakes runtime enterprise.
  arXiv  Detail & Related papers  (2026-04-06T20:25:20Z)
• LLM Readiness Harness: Evaluation, Observability, and CI Gates for LLM/RAG Applications [51.56484100374058]
  We present a readiness harness for LLM and RAG applications that turns evaluation into a deployment decision workflow.<n>The system combines automated benchmarks, OpenTelemetry observability, and CI quality gates under a minimal API contract.<n>We evaluate the harness on ticket-routing and BEIR tasks with full Azure matrix coverage.
  arXiv  Detail & Related papers  (2026-03-28T18:03:32Z)
• From Stochastic Answers to Verifiable Reasoning: Interpretable Decision-Making with LLM-Generated Code [0.0]
  Large language models (LLMs) are increasingly used for high-stakes decisionmaking.<n>Black-box models obscure their reasoning, while recent LLM-based rule systems rely on per-sample evaluation.<n>We propose reframing LLMs as code generators rather than per-instance evaluators.
  arXiv  Detail & Related papers  (2026-02-28T00:27:29Z)
• HuggingR$^{4}$: A Progressive Reasoning Framework for Discovering Optimal Model Companions [50.61510609116118]
  HuggingR$4$ is a novel framework that combines Reasoning, Retrieval, Refinement, and Reflection to efficiently select models.<n>It attains a workability rate of 92.03% and a reasonability rate of 82.46%, surpassing existing method by 26.51% and 33.25% respectively.
  arXiv  Detail & Related papers  (2025-11-24T03:13:45Z)
• Reasoning with Confidence: Efficient Verification of LLM Reasoning Steps via Uncertainty Heads [104.9566359759396]
  We propose a lightweight alternative for step-level reasoning verification based on data-driven uncertainty scores.<n>Our findings suggest that the internal states of LLMs encode their uncertainty and can serve as reliable signals for reasoning verification.
  arXiv  Detail & Related papers  (2025-11-09T03:38:29Z)
• Are You Getting What You Pay For? Auditing Model Substitution in LLM APIs [71.7892165868749]
  Commercial Large Language Model (LLM) APIs create a fundamental trust problem.<n>Users pay for specific models but have no guarantee that providers deliver them faithfully.<n>We formalize this model substitution problem and evaluate detection methods under realistic adversarial conditions.<n>We propose and evaluate the use of Trusted Execution Environments (TEEs) as one practical and robust solution.
  arXiv  Detail & Related papers  (2025-04-07T03:57:41Z)
• Identifying and Mitigating API Misuse in Large Language Models [26.4403427473915]
  API misuse in code generated by large language models (LLMs) represents a serious emerging challenge in software development.<n>This paper presents the first comprehensive study of API misuse patterns in LLM-generated code, analyzing both method selection and parameter usage across Python and Java.<n>We propose Dr.Fix, a novel LLM-based automatic program repair approach for API misuse based on the aforementioned taxonomy.
  arXiv  Detail & Related papers  (2025-03-28T18:43:12Z)
• SOPBench: Evaluating Language Agents at Following Standard Operating Procedures and Constraints [59.645885492637845]
  SOPBench is an evaluation pipeline that transforms each service-specific SOP code program into a directed graph of executable functions.<n>Our approach transforms each service-specific SOP code program into a directed graph of executable functions and requires agents to call these functions based on natural language SOP descriptions.<n>We evaluate 18 leading models, and results show the task is challenging even for top-tier models.
  arXiv  Detail & Related papers  (2025-03-11T17:53:02Z)
• CaLM: Contrasting Large and Small Language Models to Verify Grounded Generation [76.31621715032558]
  Grounded generation aims to equip language models (LMs) with the ability to produce more credible and accountable responses. We introduce CaLM, a novel verification framework. Our framework empowers smaller LMs, which rely less on parametric memory, to validate the output of larger LMs.
  arXiv  Detail & Related papers  (2024-06-08T06:04:55Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.