Benchmarks are Not Enough: RAMP for Runtime Assessing of Agentic Models in Production Systems

• URL: http://arxiv.org/abs/2605.27492v1
• Date: Tue, 26 May 2026 16:28:10 GMT
• Title: Benchmarks are Not Enough: RAMP for Runtime Assessing of Agentic Models in Production Systems
• Abstract summary: We present RAMP, a production-grounded infrastructure for assessing software engineering agents.<n>RAMP provides a unified runtime assessment architecture through standardized orchestration and execution interfaces.
• Score: 2.9725171307292544
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: LLM agents are rapidly evolving from coding assistants into autonomous software engineering systems. However, existing evaluation methodologies remain largely centered on static, isolated, and short-horizon benchmarks that fail to capture the dynamic complexity of real-world production workflows. As a result, benchmark performance may poorly reflect practical capability under realistic runtime environments involving long execution chains, tool interactions, dependency management, and iterative feedback loops. We thus present RAMP, a production-grounded infrastructure for assessing long-horizon software engineering agents. Built upon the YatCC integrated platform, RAMP provides a unified runtime assessment architecture through standardized orchestration and execution interfaces. RAMP introduces realistic compiler-construction workloads with serial dependencies and complex toolchain interactions, together with a staged recovery mechanism for analyzing execution behavior under partial workflow failure. The framework further incorporates utility-oriented multi-dimensional metrics that jointly evaluate outcome quality and process efficiency. We conduct runtime assessments across 15 mainstream models and observe substantial capability degradation that remains largely invisible to conventional isolated benchmarks. Task completion rates progressively collapse across serial workflows, dropping from 100% in the initial stage to only 20% in the final stage, while none of the evaluated models successfully completes the entire pipeline. Runtime analysis reveals systematic failure propagation and significant resource inefficiencies, with computational costs differing by up to three orders of magnitude among comparable models. These findings suggest RAMP advances agentic model evaluation toward continuous, runtime-observable, and production-grounded assessment.

Related papers

• DynAMO:Dynamic Asset Management Orchestration via Topological Multi-Agent Scheduling [0.2333246928962179]
  We present DynAMO, a deployment-ready engine using a Plan-then-Execute architecture to generate verifiable workflow graphs.<n>By dynamically identifying independent tasks, DynAMO preserves structural correctness and safety.<n>Parallel execution reduces end-to-end latency by a median of 1.6x over sequential orchestration.
  arXiv  Detail & Related papers  (2026-06-14T10:54:24Z)
• The Illusion of Multi-Agent Advantage [63.92466143571383]
  We show that expert-architected Multi-Agent Systems (MAS) consistently outperforms automatically generated architectures in raw performance and cost-efficiency.<n>Existing evaluation frameworks mask critical architectural gaps and inefficiencies by failing to account for the marginal utility of increased computational cost.
  arXiv  Detail & Related papers  (2026-06-11T07:39:24Z)
• Identifying and Mitigating Systemic Measurement Bias in Production LLM Inference Benchmarks [5.044646432723069]
  We show that benchmarking utilities rely on single-process, asyncio-driven architectures that introduce fundamental client-side queuing bottlenecks.<n>We propose an unbiased, multi-process evaluation framework that effectively distributes client-side load, ensuring negligible queuing overhead.
  arXiv  Detail & Related papers  (2026-05-22T20:57:26Z)
• ComplexMCP: Evaluation of LLM Agents in Dynamic, Interdependent, and Large-Scale Tool Sandbox [61.862814740220806]
  $textbfComplexMCP$ is a benchmark designed to evaluate agents in rigorous conditions.<n>Built on the Model Context Protocol (MCP), $textbfComplexMCP$ provides over 300 meticulously tested tools.
  arXiv  Detail & Related papers  (2026-05-11T16:20:51Z)
• Beyond Isolated Tasks: A Framework for Evaluating Coding Agents on Sequential Software Evolution [5.10403054516716]
  Existing datasets evaluate performance on isolated, single pull request (PR) tasks in a stateless manner.<n>We introduce an automated coding task generation framework, which helps generate our dataset SWE-STEPS.
  arXiv  Detail & Related papers  (2026-04-03T13:44:40Z)
• Understanding by Reconstruction: Reversing the Software Development Process for LLM Pretraining [66.89012795621349]
  Large Language Models (LLMs) often struggle with the deep, long-horizon reasoning required for complex software engineering.<n>We propose a novel paradigm: understanding via reconstruction.<n>We introduce a framework that synthesizes latent agentic trajectories using a multi-agent simulation.
  arXiv  Detail & Related papers  (2026-03-11T09:23:20Z)
• EmboCoach-Bench: Benchmarking AI Agents on Developing Embodied Robots [68.29056647487519]
  Embodied AI is fueled by high-fidelity simulation and large-scale data collection.<n>However, this scaling capability remains bottlenecked by a reliance on labor-intensive manual oversight.<n>We introduce textscEmboCoach-Bench, a benchmark evaluating the capacity of LLM agents to autonomously engineer embodied policies.
  arXiv  Detail & Related papers  (2026-01-29T11:33:49Z)
• LoCoBench-Agent: An Interactive Benchmark for LLM Agents in Long-Context Software Engineering [90.84806758077536]
  We introduce textbfLoCoBench-Agent, a comprehensive evaluation framework specifically designed to assess large language models (LLMs) agents in realistic, long-context software engineering.<n>Our framework extends LoCoBench's 8,000 scenarios into interactive agent environments, enabling systematic evaluation of multi-turn conversations.<n>Our framework provides agents with 8 specialized tools (file operations, search, code analysis) and evaluates them across context lengths ranging from 10K to 1M tokens.
  arXiv  Detail & Related papers  (2025-11-17T23:57:24Z)
• Flash-Searcher: Fast and Effective Web Agents via DAG-Based Parallel Execution [48.7788770680643]
  Flash-Searcher is a novel parallel agent reasoning framework.<n>It decomposes complex tasks into subtasks with explicit dependencies, enabling concurrent execution of independent reasoning paths.<n>It achieves 67.7% accuracy on BrowseComp and 83% on xbench-DeepSearch, while reducing agent execution steps by up to 35% compared to current frameworks.
  arXiv  Detail & Related papers  (2025-09-29T17:39:30Z)
• AgentCompass: Towards Reliable Evaluation of Agentic Workflows in Production [4.031479494871582]
  We present Agent, the first evaluation framework designed specifically for post-deployment monitoring and reasoning of agentic pipeline.<n>Agent achieves state-of-the-art results on key metrics, while uncovering critical issues missed in human annotations.
  arXiv  Detail & Related papers  (2025-09-18T05:59:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.