VAmoS Bench: Voice Agent Simulation Bench

• URL: http://arxiv.org/abs/2607.27453v1
• Date: Wed, 29 Jul 2026 20:42:38 GMT
• Title: VAmoS Bench: Voice Agent Simulation Bench
• Abstract summary: VAmoS Bench measures complete voice-agent systems end to end in a stateful customer-support task.<n>Each of 100 scenarios supplies a simulated caller with a private goal and a seeded backend.<n>The platform uses each scenario to populate and activate an isolated simulation in which the caller reaches Riley over audio.
• Score: 8.26112859764927
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Production voice agents span cascaded, speech-to-speech, and hybrid architectures. Voice-agent benchmarks typically measure component quality and conversational properties such as word error rate, latency, naturalness, and turn-taking. Fewer measure whether the agent handled a phone call correctly on its own. Contact centers refer to this as ``containment'': the share of phone calls the automated system resolves without handing off to a human. On some phone calls the right outcome is refusal or a redirect. To address this gap, we introduce VAmoS Bench, the Voice Agent Simulation Bench. It measures complete voice-agent systems end to end in a stateful customer-support task. The agent is Riley, a credit-card support representative for a fictional bank who can freeze, cancel, replace, or activate a card. Each of 100 scenarios supplies a simulated caller with a private goal and a seeded PostgreSQL backend. The platform uses each scenario to populate and activate an isolated simulation in which the caller reaches Riley over audio; roughly one-third apply adversarial pressure. The agent can use five tools that execute real SQL against the backend. Each scenario also defines binary assertions. A grader evaluates them against the complete trace of what the caller and agent said and what the agent did, including tool invocations, arguments, and returned rows. This catches an agent that claims to have changed a card without updating the database, as well as one that makes the right database change while disclosing protected information. This first benchmark version focuses on financial services. Its evaluation protocol supports an evolving leaderboard: additional voice agents can be evaluated on the same version, while later versions can expand the tasks and scenarios.

Related papers

• BekchiAI: Measuring, Observing, and Controlling LLM Agents in One Click [0.0]
  We present BekchiAI, a benchmark for measuring agentic skill and a platform for observing and controlling live agents.<n>The BekchiAI-Platform is a complementary web-based observability and control layer for deployed agents, providing full token and telemetry as well as remote run termination.
  arXiv  Detail & Related papers  (2026-08-27T09:30:08Z)
• SpeechGym: An Audio-Native Gym for Training Voice Agents via Reinforcement Learning [24.07999549415095]
  SpeechGym is an audio-native agentic environment in which two omni-modal models converse in native audio.<n>We present SpeechGym, an audio-native agentic environment in which two omni-modal models converse in native audio.
  arXiv  Detail & Related papers  (2026-08-26T22:16:05Z)
• Tencent WorkBuddy Bench: A Multi-Domain Coding-Agent Benchmark with Contamination-Resistant Task Construction [57.138342889101345]
  Tencent WorkBuddy Bench is a multi-domain evaluation suite for coding agents.<n>This report documents its construction methodology, scoring protocol, and a cross-model leaderboard.
  arXiv  Detail & Related papers  (2026-07-23T04:34:06Z)
• SWE-Together: Evaluating Coding Agents in Interactive User Sessions [19.069719245971786]
  We introduce SWE-Together, a multi-turn benchmark reconstructed from real user-agent coding sessions.<n>We curate 109 repository-level tasks from 11,260 recorded sessions, selecting sessions with recoverable repository states, clear user goals, and observable outcomes.<n>We build a reactive LLM-based user simulator that preserves the original users' intents and provides feedback when the coding agent's progress requires it.
  arXiv  Detail & Related papers  (2026-06-29T08:35:15Z)
• Capable but Careless: Do Computer-Use Agents Follow Contextual Integrity? [54.701199583505144]
  Cross-application access creates a privacy risk that has been largely overlooked.<n>We introduce AgentCIBench, an evaluation harness that turns this risk into executable, deterministically scored scenarios.<n>We evaluate 15 frontier agents and find a surprisingly high failure rate.
  arXiv  Detail & Related papers  (2026-06-22T11:36:58Z)
• PACMS: Submodular Context Selection as a Pluggable Engine for LLM Agents [0.9134244356393664]
  LLM agents operate over a context window that fills from several directions simultaneously.<n>The prevailing mechanism is recency truncation, sometimes paired with periodic summarization.<n>Retrieval augmented generation fetches external documents into the prompt but does not arbitrate the agent's emphalready-present pooled context.
  arXiv  Detail & Related papers  (2026-06-18T10:22:57Z)
• Good Agentic Friends Do Not Just Give Verbal Advice: They Can Update Your Weights [58.27172968382275]
  Multi-agent LLM systems usually collaborate by exchanging natural-language messages.<n>We study an alternative communication interface: instead of appending a sender's message to the receiver's context, compile the sender's hidden states into a transient, receiver-specific weight perturbation.
  arXiv  Detail & Related papers  (2026-05-13T17:58:32Z)
• ClawMark: A Living-World Benchmark for Multi-Turn, Multi-Day, Multimodal Coworker Agents [77.22389710754452]
  We introduce bench, a benchmark for coworker agents built around multi-turn multi-day tasks.<n>The current release contains 100 tasks across 13 professional scenarios, executed against five stateful sandboxed services.<n>The strongest model reaches 75.8 weighted score, but the best strict Task Success is only 20.0%, indicating that partial progress is common.
  arXiv  Detail & Related papers  (2026-04-26T16:05:02Z)
• AgentForge: Execution-Grounded Multi-Agent LLM Framework for Autonomous Software Engineering [3.2126925586839623]
  We introduce execution-grounded verification as a first-class principle.<n>We instantiate this principle in AGENTFORGE, a multi-agent framework.<n>AGENTFORGEtoken achieves 40.0% resolution on SWE-BENCH Lite.
  arXiv  Detail & Related papers  (2026-04-13T13:51:13Z)
• KnowU-Bench: Towards Interactive, Proactive, and Personalized Mobile Agent Evaluation [72.01173512175531]
  KnowU-Bench is an online benchmark for personalized mobile agents.<n>It covers 42 general GUI tasks, 86 personalized tasks, and 64 proactive tasks.<n>Our experiments reveal a striking degradation: agents that excel at explicit task execution fall below 50% under vague instructions.
  arXiv  Detail & Related papers  (2026-04-09T16:50:50Z)
• Cloning a Conversational Voice AI Agent from Call\,Recording Datasets for Telesales [0.0]
  We present a methodology for cloning a conversational voice AI agent from a corpus of call recordings.<n>Our system listens to customers over the telephone, responds with a synthetic voice, and follows a structured playbook learned from top performing human agents.<n>The cloned agent is evaluated against human agents on a rubric of 22 criteria covering introduction, product communication, sales drive, objection handling, and closing.
  arXiv  Detail & Related papers  (2025-09-05T07:36:12Z)
• Text-Based Detection of On-Hold Scripts in Contact Center Calls [0.6138671548064356]
  Average hold time is a concern for call centers because it affects customer satisfaction. This study presents a natural language processing model that detects on-hold phrases in customer service calls transcribed by automatic speech recognition technology.
  arXiv  Detail & Related papers  (2024-07-13T11:11:41Z)
• Towards Socially Intelligent Agents with Mental State Transition and Human Utility [97.01430011496576]
  We propose to incorporate a mental state and utility model into dialogue agents. The hybrid mental state extracts information from both the dialogue and event observations. The utility model is a ranking model that learns human preferences from a crowd-sourced social commonsense dataset.
  arXiv  Detail & Related papers  (2021-03-12T00:06:51Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.