Regression Accumulation in Multi-Turn LLM Programming Conversations

• URL: http://arxiv.org/abs/2607.01855v1
• Date: Thu, 02 Jul 2026 08:15:40 GMT
• Title: Regression Accumulation in Multi-Turn LLM Programming Conversations
• Abstract summary: regression accumulation appears across all six models.<n>Cross-Turn Conflict, where later code conflicts with earlier requirements, is the main failure class.<n> Verification Gate is the only strategy that consistently improves all models.
• Score: 25.94870617800846
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: In LLM-assisted software development, coding is often iterative. We study regression accumulation in multi-turn LLM programming conversations, where later code suggestions may break requirements introduced in earlier turns. Reliability therefore depends not only on satisfying the current request, but also on preserving previously satisfied behavior. We construct 542 tasks from HumanEval+ and MBPP+ and extend each task into an 8-turn requirement-evolution chain. We evaluate six LLMs on 26,016 turn instances (542 x 6 x 8). At each turn, we test whether the current code still passes earlier benchmark tests. We also analyze 384 failure cases from the failure population and build a taxonomy of multi-turn regression bugs through independent four-annotator labeling. Our results show that regression accumulation appears across all six models: 40% to 73% of tasks lose previously correct behavior over the full conversation. Final-turn quality is lower than initial-turn quality across models, especially when later turns add input validation or broader input types. Manual analysis shows that Cross-Turn Conflict, where later code conflicts with earlier requirements, is the main failure class. We further find that Verification Gate, which checks new code against prior tests and triggers rollback and retry, is the only strategy that consistently improves all models, raising final-turn quality from 75.8% to 87.9% on DeepSeek-V3 and from 31.6% to 47.3% on Llama-3.1-8B. These findings suggest that strong single-turn performance can overestimate reliability in multi-turn coding conversations. Future evaluation and tool design should test whether later code suggestions preserve earlier requirements and should include Verification Gate mechanisms.

Related papers

• Does Fixing Break Security? An Empirical Study of Security Degradation in Iterative LLM-Driven Infrastructure-as-Code Repair [2.4063592468412276]
  Iterative IaC repair does introduce security regressions, but the conservative, defensible rate is about 3.3% of scenarios.<n>Our findings motivate security-aware feedback-loop design and actionable iteration-budget guidance.
  arXiv  Detail & Related papers  (2026-08-13T16:01:32Z)
• ThinkRetrieve: Retrieval-Augmented Reasoning Traces for Test-Time Scaling [60.48279351544583]
  ThinkRetrieve is a test-time scaling framework for large reasoning models.<n>It retrieves relevant exemplars at each intermediate step and injects them directly into the thinking trace.<n>It consistently improves accuracy over standard test-time scaling.
  arXiv  Detail & Related papers  (2026-08-11T13:58:07Z)
• REVES: REvision and VErification--Augmented Training for Test-Time Scaling [53.197756110943395]
  We propose a two-stage iterative framework that alternates between online data/prompt augmentation and policy optimization.<n>We observe gains of +6.5 points over the RL baseline and +4.0 points over standard multi-turn training.
  arXiv  Detail & Related papers  (2026-06-17T10:37:23Z)
• Understanding and Mitigating Premature Confidence for Better LLM Reasoning [76.16007941549857]
  Long chains of thought (CoT) from current language models frequently contain logical gaps and unjustified leaps.<n>We find such a signal in how the model's confidence evolves during reasoning.<n>We exploit this in progressive confidence shaping, a reinforcement learning objective that trains models to update their confidence as they reason rather than commit early.
  arXiv  Detail & Related papers  (2026-05-23T04:42:45Z)
• Distributional Energy-Based Models for Uncertainty-Aware Structured LLM Reasoning [40.342912574072024]
  Large Language Models produce structured outputs such as travel plans or code solutions.<n>Individual reasoning steps may appear correct while the output as a whole violates budgets, fails test cases, or contradicts earlier deductions.<n>We propose a learned quality scorer with deterministic analytical constraint penalties for verifying structured LLM outputs.
  arXiv  Detail & Related papers  (2026-05-15T17:08:27Z)
• Social Bias in LLM-Generated Code: Benchmark and Mitigation [10.39413030802123]
  Large Language Models (LLMs) are increasingly deployed to generate code for human-centered applications where demographic fairness is critical.<n>Existing evaluations focus almost exclusively on functional correctness, leaving social bias in LLM-generated code largely unexamined.<n>We propose the Fairness Monitor Agent (FMA), a modular component that plugs into any existing code generation pipeline without modifying it.
  arXiv  Detail & Related papers  (2026-05-01T04:06:02Z)
• Examining LLMs Ability to Summarize Code Through Mutation-Analysis [9.187740569361804]
  We present a mutation-based evaluation methodology that directly tests whether a summary truly matches the code's logic.<n>We validate it through three experiments totalling 624 mutation-summary evaluations across 62 programs.<n>We find that summary accuracy decreases sharply with complexity, while mutation type and location exhibit weaker effects.
  arXiv  Detail & Related papers  (2026-02-19T21:00:49Z)
• Teaching LLM to Reason: Reinforcement Learning from Algorithmic Problems without Code [76.80306464249217]
  We propose TeaR, which aims at teaching LLMs to reason better.<n>TeaR leverages careful data curation and reinforcement learning to guide models in discovering optimal reasoning paths through code-related tasks.<n>We conduct extensive experiments using two base models and three long-CoT distillation models, with model sizes ranging from 1.5 billion to 32 billion parameters, and across 17 benchmarks spanning Math, Knowledge, Code, and Logical Reasoning.
  arXiv  Detail & Related papers  (2025-07-10T07:34:05Z)
• A Hierarchical and Evolvable Benchmark for Fine-Grained Code Instruction Following with Multi-Turn Feedback [30.446511584123492]
  Large language models (LLMs) have advanced significantly in code generation, yet their ability to follow complex programming instructions with layered and diverse constraints remains underexplored.<n>We introduce MultiCodeIF, a comprehensive benchmark designed to evaluate instruction-following in code generation across multiple dimensions.<n>We synthesize and evolve 2,021 code tasks sourced from 14 programming languages, supporting multi-turn evaluation through feedback-driven task variants.
  arXiv  Detail & Related papers  (2025-07-01T11:51:40Z)
• A Preliminary Study on the Robustness of Code Generation by Large Language Models [40.01096420024215]
  We present the first empirical study of LLM-generated code robustness using the CoderEval benchmark.<n>We find that 35.2% of their outputs are less robust than human-written code, with over 90% of deficiencies caused by missing conditional checks.<n>To address these issues, we propose RobGen, a model-agnostic framework that improves robustness without retraining.
  arXiv  Detail & Related papers  (2025-03-26T03:44:03Z)
• Preference Optimization for Reasoning with Pseudo Feedback [100.62603571434167]
  We introduce a novel approach to generate pseudo feedback for reasoning tasks by framing the labeling of solutions as an evaluation against associated test cases.<n>We conduct experiments on both mathematical reasoning and coding tasks using pseudo feedback for preference optimization, and observe improvements across both tasks.
  arXiv  Detail & Related papers  (2024-11-25T12:44:02Z)
• PRover: Proof Generation for Interpretable Reasoning over Rules [81.40404921232192]
  We propose a transformer-based model that answers binary questions over rule-bases and generates the corresponding proofs. Our model learns to predict nodes and edges corresponding to proof graphs in an efficient constrained training paradigm. We conduct experiments on synthetic, hand-authored, and human-paraphrased rule-bases to show promising results for QA and proof generation.
  arXiv  Detail & Related papers  (2020-10-06T15:47:53Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.