PROTEA: Offline Evaluation and Iterative Refinement for Multi-Agent LLM Workflows

• URL: http://arxiv.org/abs/2605.18032v1
• Date: Mon, 18 May 2026 08:22:14 GMT
• Title: PROTEA: Offline Evaluation and Iterative Refinement for Multi-Agent LLM Workflows
• Abstract summary: Multi-agent LLMs often outperform single-prompt baselines, but they remain difficult to debug and refine.<n>We present PROTEA, a unified interface for offline, test-driven improvement of multi-agent systems.
• Score: 1.6429511119132734
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Multi-agent LLM workflows -- systems composed of multiple role-specific LLM calls -- often outperform single-prompt baselines, but they remain difficult to debug and refine. Failures can originate from subtle errors in intermediate outputs that propagate to downstream nodes, requiring developers to inspect long traces and infer which agent to modify. We present PROTEA, a unified interface for offline, test-driven improvement of multi-agent workflows. PROTEA executes a workflow, scores intermediate node outputs with configurable rubrics, and overlays per-node states and rationales on the workflow graph to localize likely bottlenecks. To support complex systems where final-answer references are the primary supervision, PROTEA performs backward node evaluation: it generates candidate node-level expectations from final-answer references and graph context, then compares them with observed node outputs. For selected nodes, PROTEA presents targeted prompt revisions as editable before/after comparisons, then automatically reruns and re-evaluates the workflow to show output changes and score trajectories within the same interface. In two production-adjacent workflows, PROTEA improved document-inspection accuracy from 64.3% to 83.9% and recommendation Hit@5 from 0.30 to 0.38. In a formative study with six experienced LLM developers, participants valued graph-level localization, per-node rationales, and editable before/after prompt revisions.

Related papers

• TROVE: Adaptive Agent Skill Orchestration via Trace-Grounded Route Validation and Editing [14.424308190797683]
  We propose Trace-grounded Route Orchestration via Validation and Editing (TROVE)<n> Offline, TROVE distills evaluated workflow-search traces into atomic and composite skills and an outcome-conditioned transition graph.<n>Online, it treats a planned route as provisional, after committing one top-level skill, the controller retains a valid continuation, inserts a trace-supported local response, or replaces only the invalid suffix.
  arXiv  Detail & Related papers  (2026-09-04T11:38:13Z)
• FlowScout: From Execution Feedback to Reliable Tool-Using Agent Workflows [6.767792769288717]
  FlowScout is an execution-guided framework for generating tool-integrated agentic from historical task-solving records.<n>It first mines a common tool coordination from historical records to construct an initial workflow.<n>It then refines the workflow topology through Monte Carlo tree search guided by execution feedback.
  arXiv  Detail & Related papers  (2026-08-10T08:41:59Z)
• COVENANT: Natural-Language Workflow Compilation for Aligned Agent Execution [12.755574564434788]
  We propose COVENANT, a compiler-and-interpreter architecture for workflow-aligned agent execution.<n>Our key insight is to treat workflow instructions as source programs rather than prompts.<n>COVENANT converts the instructions into a workflow abstract syntax tree (WAST) and lowers it to a workflow control-flow graph (WCFG)
  arXiv  Detail & Related papers  (2026-07-28T07:59:50Z)
• LLM-as-a-Verifier: A General-Purpose Verification Framework [74.40111651545979]
  We introduce LLM-as-a-Verifier, a general-purpose verification framework.<n>It provides fine-grained feedback for agentic tasks without requiring additional training.<n>It achieves state-of-the-art performance on several benchmarks.
  arXiv  Detail & Related papers  (2026-07-06T17:59:35Z)
• GuideMe: Multi-Domain Task Guidance and Intervention in Streaming Video [54.048366682378706]
  We construct GuideMe, the first multi-domain benchmark for streaming video that supports training and evaluation of MLLMs for closed-loop interactive task guidance.<n>It comprises 2,458 videos spanning 223.7 hours across diverse domains (eg, cooking, object manipulation, daily-life guidance, and fitness), with 47,775 interaction samples covering next-step instructions, completion feedback, error detection, and corrective guidance.
  arXiv  Detail & Related papers  (2026-07-03T06:00:04Z)
• EngiAI: A Multi-Agent Framework and Benchmark Suite for LLM-Driven Engineering Design [0.4499833362998488]
  We introduce a benchmark suite with three evaluation dimensions.<n>We present EngiAI, a Multi-Agent System (MAS) implementation built on LangGraph.
  arXiv  Detail & Related papers  (2026-05-19T12:12:09Z)
• PARM: Pipeline-Adapted Reward Model [60.769414637325326]
  Reward models (RMs) are central to aligning large language models (LLMs) with human preferences, powering advanced decoding strategies.<n>While most prior work focuses on single-step generation, real-world applications increasingly adopt multi-stage pipelines.<n>We investigate this through code generation for optimization, constructing a pipeline that integrates reward models into both formulation and solution stages.
  arXiv  Detail & Related papers  (2026-04-20T14:29:08Z)
• LLM-based Schema-Guided Extraction and Validation of Missing-Person Intelligence from Heterogeneous Data Sources [0.7734726150561088]
  Missing-person and child-safety investigations rely on heterogeneous case documents, including structured forms, bulletin-style posters, and narrative web profiles.<n>Variations in layout, terminology, and data quality impede rapid triage, large-scale analysis, and search-planning.<n>This paper introduces the Guardian Pack, an AI-driven parsing and normalization pipeline that transforms multi-source investigative documents into a unified, schema-compliant representation.
  arXiv  Detail & Related papers  (2026-04-08T01:35:56Z)
• Chimera: Latency- and Performance-Aware Multi-agent Serving for Heterogeneous LLMs [62.17306142810532]
  We present Chimera, a predictive scheduling system for multi-agent workflow serving on heterogeneous LLM clusters.<n>Chime traces the best latency, reducing end-to-end latency by 1.2--2.4$times$ and improving task performance by 8.0-9.5 percentage points on average over competitive baselines.
  arXiv  Detail & Related papers  (2026-03-23T17:01:42Z)
• Arrow-Guided VLM: Enhancing Flowchart Understanding via Arrow Direction Encoding [0.0]
  Flowcharts are indispensable tools in software design and business-process analysis.<n>Current vision-language models (VLMs) frequently misinterpret the directional arrows and graph topology.<n>We introduce a seven-stage pipeline grouped into three broader processes.
  arXiv  Detail & Related papers  (2025-05-09T04:27:36Z)
• ComfyGPT: A Self-Optimizing Multi-Agent System for Comprehensive ComfyUI Workflow Generation [96.44354750396019]
  ComfyGPT is a self-optimizing multi-agent system designed to generate ComfyUI based on task descriptions automatically.<n> FlowDataset is a large-scale dataset containing 13,571 workflow-description pairs.<n>FlowBench is a benchmark for evaluating workflow generation systems.
  arXiv  Detail & Related papers  (2025-03-22T06:48:50Z)
• Benchmarking Agentic Workflow Generation [80.74757493266057]
  We introduce WorfBench, a unified workflow generation benchmark with multi-faceted scenarios and intricate graph workflow structures.<n>We also present WorfEval, a systemic evaluation protocol utilizing subsequence and subgraph matching algorithms.<n>We observe that the generated can enhance downstream tasks, enabling them to achieve superior performance with less time during inference.
  arXiv  Detail & Related papers  (2024-10-10T12:41:19Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.