Consistency Amplifies: How Behavioral Variance Shapes Agent Accuracy

• URL: http://arxiv.org/abs/2603.25764v1
• Date: Thu, 26 Mar 2026 04:39:13 GMT
• Title: Consistency Amplifies: How Behavioral Variance Shapes Agent Accuracy
• Abstract summary: We study consistency in the context of SWE-bench, a challenging software engineering benchmark requiring complex, multi-step reasoning.<n>We find that across models, higher consistency aligns with higher accuracy.<n>Within a model, consistency can amplify both correct and incorrect interpretations.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: As LLM-based agents are deployed in production systems, understanding their behavioral consistency (whether they produce similar action sequences when given identical tasks) becomes critical for reliability. We study consistency in the context of SWE-bench, a challenging software engineering benchmark requiring complex, multi-step reasoning. Comparing Claude~4.5~Sonnet, GPT-5, and Llama-3.1-70B across 50 runs each (10 tasks $\times$ 5 runs), we find that across models, higher consistency aligns with higher accuracy: Claude achieves the lowest variance (CV: 15.2\%) and highest accuracy (58\%), GPT-5 is intermediate (CV: 32.2\%, accuracy: 32\%), and Llama shows the highest variance (CV: 47.0\%) with lowest accuracy (4\%). However, within a model, consistency can amplify both correct and incorrect interpretations. Our analysis reveals a critical nuance: \textbf{consistency amplifies outcomes rather than guaranteeing correctness}. 71\% of Claude's failures stem from "consistent wrong interpretation": making the same incorrect assumption across all runs. Interestingly, GPT-5 achieves similar early strategic agreement as Claude (diverging at step 3.4 vs.\ 3.2) but exhibits 2.1$\times$ higher variance, suggesting that divergence timing alone does not determine consistency. These findings suggest that for production deployment, interpretation accuracy matters more than execution consistency, with implications for agent evaluation and training.

Related papers

• Three Regimes of Context-Parametric Conflict: A Predictive Framework and Empirical Validation [0.0]
  We study how large language models handle conflict between their training knowledge and a contradicting document.<n>We propose a three-regime framework: Regime 1 (single-source updating), Regime 2 (competitive integration), and Regime 3 (task-appropriate selection)<n>We validate the framework across Claude Sonnet 4.6, GPT-5.5, Gemini 2.5 Flash, Llama 4 Maverick, and DeepSeek V3 using 9,970 API calls in three experimental phases.
  arXiv  Detail & Related papers  (2026-05-12T06:00:48Z)
• Teaching LLMs Program Semantics via Symbolic Execution Traces [0.7046782561282057]
  We introduce an evaluation framework of 500 C verification tasks built on SV-COMP 2025.<n>We evaluate 14 models across six families and find high overall accuracy masks a critical weakness.<n>Just $sim$3,000 bug traces combined with chain-of-thought reasoning at inference time improve violation detection by over 17 percentage points.
  arXiv  Detail & Related papers  (2026-05-07T13:01:06Z)
• Compositional Multi-hop Factual Error Correction via Decomposition-and-Injection [60.34509125517578]
  We present CECoR (Compositional Error Correction via Reasoning-aware Synthesis), a reasoning-aware framework for compositional error correction.<n> CECoR decomposes multi-hop claims into interpretable reasoning steps and injects controlled perturbations to synthesize high-quality training pairs.<n>A two-stage learning strategy combining supervised fine-tuning and reinforcement learning improves factual accuracy and robustness.
  arXiv  Detail & Related papers  (2026-05-04T07:08:39Z)
• Correct Answers from Sound Reasoning: Verifiable Process Supervision for Language Models [94.68358825189738]
  We propose a post-training framework for verifiable domains that jointly optimize prediction accuracy and reasoning quality.<n>We evaluate VPS on chess, a controlled testbed where reasoning steps can be deterministically verified against engine signals.<n>VPS preserves accuracy while significantly improving reasoning quality, reducing win-rate error by up to 30% and restoring consistency to near saturation.
  arXiv  Detail & Related papers  (2026-04-03T15:19:46Z)
• Prompt Architecture Determines Reasoning Quality: A Variable Isolation Study on the Car Wash Problem [0.0]
  This study examines which prompt architecture layers in a production system enable correct reasoning.<n>We find that the STAR (Situation-Task-Action-Result) reasoning framework alone raises accuracy from 0% to 85%.
  arXiv  Detail & Related papers  (2026-02-25T11:40:15Z)
• When Agents Disagree With Themselves: Measuring Behavioral Consistency in LLM-Based Agents [0.0]
  ReAct-style agents produce 2.0--4.2 distinct action sequences per 10 runs on average, even with identical inputs.<n>Tasks with consistent behavior achieve 80--92% accuracy, while highly inconsistent tasks achieve only 25--60%.<n>Our results suggest that monitoring behavioral consistency during execution could enable early error detection and improve agent reliability.
  arXiv  Detail & Related papers  (2026-02-12T06:15:14Z)
• ACAR: Adaptive Complexity Routing for Multi-Model Ensembles with Auditable Decision Traces [3.151184728006369]
  We present ACAR, a measurement framework for studying multi-model orchestration under auditable conditions.<n>ACAR uses self-consistency variance (sigma) computed from N=3 probe samples to route tasks across single-model, two-model, and three-model execution modes.<n>We evaluate ACAR on 1,510 tasks spanning four benchmarks, producing more than 7,550 auditable runs.
  arXiv  Detail & Related papers  (2026-02-06T23:27:17Z)
• Outcome Accuracy is Not Enough: Aligning the Reasoning Process of Reward Models [108.26461635308796]
  We introduce Rationale Consistency, a fine-grained metric that quantifies the alignment between the model's reasoning process and human judgment.<n>Our evaluation of frontier models reveals that rationale consistency effectively discriminates among state-of-the-art models.<n>We introduce a hybrid signal that combines rationale consistency with outcome accuracy for GenRM training.
  arXiv  Detail & Related papers  (2026-02-04T15:24:52Z)
• Does Inference Scaling Improve Reasoning Faithfulness? A Multi-Model Analysis of Self-Consistency Tradeoffs [0.0]
  Self-consistency has emerged as a popular technique for improving large language model accuracy on reasoning tasks.<n>We investigate a fundamental question that has not been studied before: does inference scaling improve reasoning faithfulness?<n>We conduct a comprehensive empirical study across four frontier models (GPT-5.2, Claude Opus 4.5, Gemini-3-flash-preview, and DeepSeek-v3.2) on 100 GSM8K mathematical reasoning problems.
  arXiv  Detail & Related papers  (2026-01-10T04:20:00Z)
• CARE What Fails: Contrastive Anchored-REflection for Verifiable Multimodal [84.71254539482369]
  Group-relative reinforcement learning with verifiable rewards (RLVR) often wastes the most informative data it already has the failures.<n>We present CARE, a failure-centric post-training framework for multimodal reasoning that turns errors into supervision.<n> CARE improves accuracy and training smoothness while explicitly increasing the share of learning signal that comes from failures.
  arXiv  Detail & Related papers  (2025-12-22T16:34:21Z)
• Robust Fine-tuning of Zero-shot Models via Variance Reduction [56.360865951192324]
  When fine-tuning zero-shot models, our desideratum is for the fine-tuned model to excel in both in-distribution (ID) and out-of-distribution (OOD) We propose a sample-wise ensembling technique that can simultaneously attain the best ID and OOD accuracy without the trade-offs.
  arXiv  Detail & Related papers  (2024-11-11T13:13:39Z)
• Accurate and Reliable Predictions with Mutual-Transport Ensemble [46.368395985214875]
  We propose a co-trained auxiliary model and adaptively regularizes the cross-entropy loss using Kullback-Leibler (KL) We show that MTE can simultaneously enhance both accuracy and uncertainty calibration. For example, on the CIFAR-100 dataset, our MTE method on ResNet34/50 achieved significant improvements compared to previous state-of-the-art method.
  arXiv  Detail & Related papers  (2024-05-30T03:15:59Z)
• Benchmarking and Improving Generator-Validator Consistency of Language Models [82.73914625520686]
  inconsistency between generating and validating an answer is prevalent in language models (LMs) Even GPT-4, a state-of-the-art LM, is GV-consistent only 76% of the time. We find that this approach improves GV-consistency of Alpaca-30B from 60% to 93%.
  arXiv  Detail & Related papers  (2023-10-03T07:23:22Z)
• Faithful Chain-of-Thought Reasoning [51.21714389639417]
  Chain-of-Thought (CoT) prompting boosts Language Models' (LM) performance on a gamut of reasoning tasks. We propose Faithful CoT, a reasoning framework involving two stages: Translation and Problem Solving. This guarantees that the reasoning chain provides a faithful explanation of the final answer.
  arXiv  Detail & Related papers  (2023-01-31T03:04:26Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.