EDGE: Engine for Deterministic Graph Evaluation through Conversation Simulation from Graph Structured DSL Configuration

• URL: http://arxiv.org/abs/2608.29971v1
• Date: Sun, 30 Aug 2026 18:56:53 GMT
• Title: EDGE: Engine for Deterministic Graph Evaluation through Conversation Simulation from Graph Structured DSL Configuration
• Abstract summary: We introduce a formal evaluation methodology that is grounded in AgentGraph, a planner powered by a domain specific language.<n>We leverage graph algorithms that exhaustively enumerate conversational paths, forming a comprehensive evaluation set that captures the agent's complete behavioral space.<n>To quantify reliability, we define novel metrics that measure response and trajectory determinism, structural adherence and semantic consistency.
• Score: 0.518884791678612
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: As agentic systems evolve into complex multi agent orchestration workflows, there is a growing and critical need for systematic frameworks that measures an agent's behavioral consistency and determinism. In this paper, we introduce a formal evaluation methodology that is grounded in AgentGraph, a planner powered by a domain specific language that represents agent reasoning through a dynamically adjustable directed graph. We leverage this structural formalism and utilize graph traversal algorithms that exhaustively enumerate conversational paths, forming a comprehensive evaluation set that captures the agent's complete behavioral space. We then systematically replay these reproducible trajectories to compare observed outputs and state transitions against the intended DSL specification. To quantify reliability, we define novel metrics that measure response and trajectory determinism, structural adherence and semantic consistency across both exact replays and their linguistic variants. Our system's results demonstrate that agents configured using frameworks like AgentGraph and LangGraph with explicitly structured node transitions show superior determinism over agents that are not configured with controlled transitions.

Related papers

• Inference-Time Graph Engineering for Multi-Agent LLM Workflows [51.56038180639833]
  We introduce ReActNet, a training-free framework that compiles a query and a set of role-specialized agents into a sequence of directed communication graphs.<n>We show that ReActNet consistently improves over fixed-topology and learned-topology baselines while maintaining competitive inference cost.
  arXiv  Detail & Related papers  (2026-09-04T23:32:11Z)
• Terminal Agents: A Survey of AI Agents in Command-Line Environments [102.48222742145474]
  Large language model agents increasingly act through terminals, yet existing surveys disperse terminal-mediated behavior across software engineering, tool use, and computer-use research.<n>We regard terminal agents as systems whose dominant progress-bearing action--observation loop is mediated by terminal command execution, textual feedback, and stateful environment interaction.<n>Our synthesis shows that realized behavior is jointly shaped by the model, interface, harness, runtime, and environment.
  arXiv  Detail & Related papers  (2026-08-20T18:16:44Z)
• Agentic Configuration Management (ACM): A Reference Configuration Model for Governed Agentic Systems [0.0]
  Agentic Configuration Management (ACM) is a framework-independent governance and configuration reference model for heterogeneous agentic systems.<n>ACM combines typed and independently versioned Agentic Configuration Items, immutable revisions and baselines, explicit configuration-runtime separation, semantic assurance semantics, dependency-aware impact propagation, and runtime provenance.<n>We provide a Python reference implementation with adapters for LangGraph, CrewAI, and the OpenAI Agents SDK.
  arXiv  Detail & Related papers  (2026-08-11T17:28:39Z)
• Formal Verification of Agentic Systems over Operational Data [59.98246281888422]
  Agentic systems driven by large language models (LLMs) are increasingly deployed in real-world where they act on persistent operational data.<n>Existing approaches do not provide such system-level guarantees.<n>We study here the verification of agentic systems comprising a single LLM and a tool orchestration harness over relational operational data.
  arXiv  Detail & Related papers  (2026-08-04T13:01:30Z)
• How to Interpret Agent Behavior [56.59836196946289]
  We introduce ACT*ONOMY, a taxonomy for describing and analyzing agent behavior at runtime.<n>By providing a shared vocabulary, ACT*ONOMY helps researchers, agent designers, and end users interpret agent behavior more consistently.
  arXiv  Detail & Related papers  (2026-05-13T14:52:40Z)
• When Only the Final Text Survives: Implicit Execution Tracing for Multi-Agent Attribution [10.973058523304042]
  We introduce IET (Implicit Execution Tracing), a metadata-independent framework that enables token-level attribution directly from generated text.<n>During generation, agent-specific keyed signals are embedded into the token distribution, transforming the text into a self-describing execution trace detectable only with a secret key.<n>At detection time, a transition-aware scoring method identifies agent handover points and reconstructs the interaction graph.
  arXiv  Detail & Related papers  (2026-03-18T07:34:51Z)
• El Agente Gráfico: Structured Execution Graphs for Scientific Agents [7.47895130442454]
  We present El Agente Grfico, a single-agent framework that embeds large language models (LLMs)-driven decision-making within a type-safe execution environment.<n>Central to our approach is a structured abstraction of scientific concepts and an object-graph mapper that represents computational state as typed Python objects.<n>We evaluate the system by developing an automated benchmarking framework across a suite of university-level quantum chemistry tasks.
  arXiv  Detail & Related papers  (2026-02-19T23:47:05Z)
• Multi-Agent Procedural Graph Extraction with Structural and Logical Refinement [66.51979814832332]
  model formulates procedural graph extraction as a multi-round reasoning process with dedicated structural and logical refinement.<n>Experiments demonstrate that model achieves substantial improvements in both structural correctness and logical consistency over strong baselines.
  arXiv  Detail & Related papers  (2026-01-27T04:00:48Z)
• AI Agent for Reverse-Engineering Legacy Finite-Difference Code and Translating to Devito [0.0]
  This study develops an integrated AI framework to facilitate the transformation of legacy finite difference implementations into the Devito environment.<n>Retrieval-Augmented Generation (RAG) and open-source Large Language Models are combined through multi-stage iterative in the system's hybrid LangGraph architecture.
  arXiv  Detail & Related papers  (2026-01-26T11:31:00Z)
• AgentRouter: A Knowledge-Graph-Guided LLM Router for Collaborative Multi-Agent Question Answering [51.07491603393163]
  tAgent is a framework that formulates multi-agent QA as a knowledge-graph-guided routing problem supervised by empirical performance signals.<n>By leveraging soft supervision and weighted aggregation of agent outputs, Agent learns principled collaboration schemes that capture the complementary strengths of diverse agents.
  arXiv  Detail & Related papers  (2025-10-06T23:20:49Z)
• Learning to Model Graph Structural Information on MLPs via Graph Structure Self-Contrasting [50.181824673039436]
  We propose a Graph Structure Self-Contrasting (GSSC) framework that learns graph structural information without message passing. The proposed framework is based purely on Multi-Layer Perceptrons (MLPs), where the structural information is only implicitly incorporated as prior knowledge. It first applies structural sparsification to remove potentially uninformative or noisy edges in the neighborhood, and then performs structural self-contrasting in the sparsified neighborhood to learn robust node representations.
  arXiv  Detail & Related papers  (2024-09-09T12:56:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.