Constraint Decay: The Fragility of LLM Agents in Backend Code Generation

• URL: http://arxiv.org/abs/2605.06445v1
• Date: Thu, 07 May 2026 15:44:40 GMT
• Title: Constraint Decay: The Fragility of LLM Agents in Backend Code Generation
• Abstract summary: Large Language Model (LLM) agents demonstrate strong performance in autonomous code generation under loose specifications.<n>Production-grade software requires strict adherence to structural constraints, such as architectural patterns, databases, and object-relational mappings.<n>We present a systematic study evaluating how well agents handle structural constraints in backend generation.
• Score: 9.659020624935687
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Large Language Model (LLM) agents demonstrate strong performance in autonomous code generation under loose specifications. However, production-grade software requires strict adherence to structural constraints, such as architectural patterns, databases, and object-relational mappings. Existing benchmarks often overlook these non-functional requirements, rewarding functionally correct but structurally arbitrary solutions. We present a systematic study evaluating how well agents handle structural constraints in multi-file backend generation. By fixing a unified API contract across 80 greenfield generation tasks and 20 feature-implementation tasks spanning eight web frameworks, we isolate the effect of structural complexity using a dual evaluation with end-to-end behavioral tests and static verifiers. Our findings reveal a phenomenon of constraint decay: as structural requirements accumulate, agent performance exhibits a substantial decline. Capable configurations lose 30 points on average in assertion pass rates from baseline to fully specified tasks, while some weaker configurations approach zero. Framework sensitivity analysis exposes significant performance disparities: agents succeed in minimal, explicit frameworks (e.g., Flask) but perform substantially worse on average in convention-heavy environments (e.g., FastAPI, Django). Finally, error analysis identifies data-layer defects (e.g., incorrect query composition and ORM runtime violations) as the leading root causes. This work highlights that jointly satisfying functional and structural requirements remains a key open challenge for coding agents.

Related papers

• SWE-Gate: Passing Functional Tests Is Not Enough for Software Engineering Agents [63.65682461568621]
  We introduce SWE-Gate, a repository-level benchmark for software engineering agents that explicitly evaluates review constraint compliance alongside functional correctness.<n>SWE-Gate derives review constraints from real pull request review comments and synthesizes repository-level repair instances around these constraints.<n>Each instance provides separate functional and constraint tests, together with non-compliant and gold patches, enabling explicit separation between issue resolution capability and review constraint compliance.
  arXiv  Detail & Related papers  (2026-09-03T17:53:34Z)
• Formal Verification of Agentic Systems over Operational Data [59.98246281888422]
  Agentic systems driven by large language models (LLMs) are increasingly deployed in real-world where they act on persistent operational data.<n>Existing approaches do not provide such system-level guarantees.<n>We study here the verification of agentic systems comprising a single LLM and a tool orchestration harness over relational operational data.
  arXiv  Detail & Related papers  (2026-08-04T13:01:30Z)
• RepoReasoner: Evaluating Repository-Level Code Reasoning Ability of Long-Context Language Models [54.53236295237077]
  We introduce RepoReasoner, a benchmark for evaluating repository-level code reasoning.<n> Output Prediction measures fine-grained, stateful execution reasoning across files.<n>Call Chain Prediction evaluates high-level architectural dependency understanding under noisy context.
  arXiv  Detail & Related papers  (2026-07-28T17:12:41Z)
• StructureClaw: Traceable LLM Agents and an Executable Benchmark for Structural Engineering Workflows [15.552044829103373]
  We present StructureClaw, an artifact-centered workbench for structural engineering requests.<n>We also introduce StructureClaw-Bench, an executable benchmark of 150 controlled scenarios.<n>Across ten agent-model configurations, each evaluated on the same 50 standard cases, the average Success Rate rises from 56.8% with the generic-skill baseline to 88.6% with the full automatic workflow.
  arXiv  Detail & Related papers  (2026-07-16T12:13:41Z)
• A Unified Structured Query Understanding Framework for Industrial Semantic Search [9.913119853395205]
  Query understanding in large-scale industrial search systems is typically implemented as a cascade of disparate, task-specific components.<n>We propose and deploy a unified structured query understanding system that consolidates these heterogeneous functions into a single Small Language Model.<n>We validate this approach through extensive offline and online tests within LinkedIn's Job Search system.
  arXiv  Detail & Related papers  (2026-05-22T19:35:15Z)
• LiveFMBench: Unveiling the Power and Limits of Agentic Workflows in Specification Generation [75.05397479715576]
  Large language models (LLMs) and agents have shown promising progress, but their true capabilities and failure modes remain unclear.<n>We present the first systematic and contamination-aware study of LLM- and agent-based formal specification generation for C programs.
  arXiv  Detail & Related papers  (2026-05-02T11:31:33Z)
• TopoPilot: Reliable Conversational Workflow Automation for Topological Data Analysis and Visualization [4.62716665682001]
  TopoPilot is a reliable and agentic framework for automating complex scientific visualization.<n>TopoPilot incorporates systematic guardrails and verification mechanisms to ensure reliable operation.<n>In evaluations, TopoPilot achieves a success rate exceeding 99%, compared to under 50% for baselines without comprehensive guardrails and checks.
  arXiv  Detail & Related papers  (2026-03-26T05:56:53Z)
• Dynamic analysis enhances issue resolution [53.50448142467294]
  DAIRA (Dynamic Analysis-enhanced Issue Resolution Agent) is an automated repair framework that embeds dynamic analysis into the agent's reasoning cycle.<n> Driven by a Test Tracing-Driven methodology, DAIRA utilizes lightweight monitors to extract critical runtime data.<n>When powered by Gemini 3 Flash Preview, DAIRA establishes a new state-of-the-art (SOTA) performance, achieving a 79.4% resolution rate on the SWE-bench Verified dataset.
  arXiv  Detail & Related papers  (2026-03-23T14:48:54Z)
• CCTU: A Benchmark for Tool Use under Complex Constraints [66.87622847854337]
  We introduce CCTU, a benchmark for evaluating large language models (LLMs) under complex constraints.<n>The benchmark comprises 200 carefully curated and challenging test cases across diverse tool-use scenarios.<n>We develop an executable constraint validation module that performs step-level validation and enforces compliance.
  arXiv  Detail & Related papers  (2026-03-16T14:05:13Z)
• Architecture-Aware Multi-Design Generation for Repository-Level Feature Addition [53.50448142467294]
  RAIM is a multi-design and architecture-aware framework for repository-level feature addition.<n>It shifts away from linear patching by generating multiple diverse implementation designs.<n>Experiments on the NoCode-bench Verified dataset demonstrate that RAIM establishes a new state-of-the-art performance.
  arXiv  Detail & Related papers  (2026-03-02T12:50:40Z)
• Data Dependency-Aware Code Generation from Enhanced UML Sequence Diagrams [54.528185120850274]
  We propose a novel step-by-step code generation framework named API2Dep.<n>First, we introduce an enhanced Unified Modeling Language (UML) API diagram tailored for service-oriented architectures.<n>Second, recognizing the critical role of data flow, we introduce a dedicated data dependency inference task.
  arXiv  Detail & Related papers  (2025-08-05T12:28:23Z)
• A Large Language Model-Empowered Agent for Reliable and Robust Structural Analysis [14.754785659805869]
  Large language models (LLMs) have exhibited remarkable capabilities across diverse open-domain tasks, yet their application in specialized domains such as civil engineering remains largely unexplored.<n>This paper starts bridging this gap by evaluating and enhancing the reliability and robustness of LLMs in structural analysis of beams.<n> Experimental results demonstrate that the agent achieves accuracy exceeding 99.0% on the benchmark dataset, exhibiting reliable and robust performance across diverse conditions.
  arXiv  Detail & Related papers  (2025-06-27T04:16:53Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.