CallScreenBench: Benchmarking On-Device Models as Phone Secretaries

• URL: http://arxiv.org/abs/2608.01033v1
• Date: Sun, 02 Aug 2026 06:31:25 GMT
• Title: CallScreenBench: Benchmarking On-Device Models as Phone Secretaries
• Abstract summary: A phone secretary takes an unknown inbound call on its owner's behalf.<n>Unlike the agents evaluated by most benchmarks, it has no task to complete and no cooperative user.<n>We present CallScreenBench, which scores this setting on five quality dimensions.
• Score: 3.917303377795418
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Language models small enough to run on a handset, quantized to a few bits, are increasingly capable of acting for their user, making on-device task automation newly plausible. One such task is answering the phone. A phone secretary takes an unknown inbound call on its owner's behalf. Unlike the agents evaluated by most benchmarks, it has no task to complete and no cooperative user: the caller holds the goal, may be an adversary, and must be judged from the opening turn with no oracle. What matters is not task success, but whether the owner would endorse how their proxy handled the call. We present CallScreenBench, which scores this setting on five quality dimensions. Each dimension is printed beside the counter-metric that bills it and is never averaged into a single number. We also report a guardedness profile for a toolless proxy that holds no credentials and calls no tools. Across six on-device models (0.6-4B parameters, 4-bit quantization), quality scales with capability, but triage does not. The appearance that it does is an artifact of measurement. Scripted degenerate agents supply the missing floors: after correcting for them, the number of model pairs whose triage performance separates falls from 11 of 15 to zero at the preregistered operating point. An agent that simply hangs up and echoes the caller also scores perfect message fidelity. We report which of our own metrics these floors defeat and declare no pass/fail threshold.

Related papers

• IBIB: A Protocol for Measuring Enterprise AI Systems by Serving Route, Not Model Identifier [0.0]
  All 18 audited benchmarks score advertised model identifiers.<n>A gold-blind capability-binding preflight verifies that a route can execute the evaluation contract before any task reaches it.<n>A reliability-inclusive first-pass scoring rule keeps failure in the score while keeping unsupported capability out.
  arXiv  Detail & Related papers  (2026-09-09T17:31:29Z)
• Nothing Breaks: No Single Peer Can Soundly Gate Post-Quantum Delivery [0.0]
  Post-quantum protection is delivered to a peer, not declared in a file.<n>We show that no single peer can soundly gate that relation.<n>We then show that agents make that state reachable at scale, driving a validated downgrade in 40 of 40 episodes from ordinary engineering prose.
  arXiv  Detail & Related papers  (2026-09-07T18:08:08Z)
• Fidelity Is Not Enough: Dispatch-Level Instrumentation for Agentic Datasheet Extraction [6.419341495897254]
  We log every tool call in an agentic benchmark of 25 hand-curated claims over three components, with 12 more on a fourth, 37 in all.<n>We build two instruments: a rule-based failure-attribution classifier, and a silent-failure detector whose two rules check only which tools were called, never the extracted value.
  arXiv  Detail & Related papers  (2026-08-28T15:25:12Z)
• Feedback That Backfires: Why Small Language Model Agents Repeat the Call They Just Watched Fail [0.0]
  Defining the corrective gain of a failure record as the change in log-probability of re-emitting the action that just failed, we find the gain is negative for every instruction-tuned model.<n>The problem is in the harness, not the model's grasp of error messages, and that predicts which remedies work.
  arXiv  Detail & Related papers  (2026-08-24T10:19:17Z)
• VAmoS Bench: Voice Agent Simulation Bench [8.26112859764927]
  VAmoS Bench measures complete voice-agent systems end to end in a stateful customer-support task.<n>Each of 100 scenarios supplies a simulated caller with a private goal and a seeded backend.<n>The platform uses each scenario to populate and activate an isolated simulation in which the caller reaches Riley over audio.
  arXiv  Detail & Related papers  (2026-07-29T20:42:38Z)
• PairCoder++: Pair Programming as a Universal Paradigm for Verified Code-Driven Multimodal and Structured-Artifact Generation [51.92442051257354]
  PairCoder improves essentially every benchmark whose artifact is verifiable, on full official metric suites rather than execution alone.<n>TikZ compile rate up 10 to 30 points on every model, at 2.9 to 9.2 times single model cost.
  arXiv  Detail & Related papers  (2026-07-02T08:36:02Z)
• PhoneHarness: Harnessing Phone-Use Agents through Mixed GUI, CLI, and Tool Actions [73.69976712292683]
  PhoneHarness is a mixed-action benchmark and execution harness for studying phone-use agents on verifiable mobile.<n>Its benchmark, PhoneHarness Bench, evaluates whether agents complete tasks with observable side effects, not only whether they produce plausible final answers.<n>Our findings suggest that reliable phone automation depends on action-surface routing and verifiable execution, not only visual GUI control.
  arXiv  Detail & Related papers  (2026-06-12T15:01:32Z)
• WildClawBench: A Benchmark for Real-World, Long-Horizon Agent Evaluation [88.10947115397971]
  This work presents WildClawBench, a native-runtime benchmark of 60 human-authored, bilingual, multimodal tasks spanning six thematic categories.<n>Each task averages roughly 8 minutes of wall-clock time and over 20 tool calls.<n> Grading is hybrid, combining deterministic rule-based checks, environment-state auditing of side effects, and an LLM/VLM judge for semantic verification.
  arXiv  Detail & Related papers  (2026-05-11T17:49:43Z)
• Can Agent Benchmarks Support Their Scores? Evidence-Supported Bounds for Interactive-Agent Evaluation [8.049844623361725]
  Interactive agent benchmarks map an agent run to a binary outcome through outcome checks.<n>These checks rely on surface level signals or fail to capture the agent's actual action path.<n>We introduce an outcome evidence reporting layer for existing benchmarks.
  arXiv  Detail & Related papers  (2026-05-11T12:20:15Z)
• Safe, or Simply Incapable? Rethinking Safety Evaluation for Phone-Use Agents [73.69976712292681]
  When a phone-use agent avoids harm, does that show safety, or simply inability to act?<n>A harmful outcome may be avoided because the agent recognized the risk and chose the safe action, or because it failed to understand the screen or execute any relevant action at all.<n>We address this problem with PhoneSafety, a benchmark of 700 safety-critical moments drawn from real phone interactions across more than 130 apps.
  arXiv  Detail & Related papers  (2026-05-08T11:58:57Z)
• ClawMark: A Living-World Benchmark for Multi-Turn, Multi-Day, Multimodal Coworker Agents [77.22389710754452]
  We introduce bench, a benchmark for coworker agents built around multi-turn multi-day tasks.<n>The current release contains 100 tasks across 13 professional scenarios, executed against five stateful sandboxed services.<n>The strongest model reaches 75.8 weighted score, but the best strict Task Success is only 20.0%, indicating that partial progress is common.
  arXiv  Detail & Related papers  (2026-04-26T16:05:02Z)
• Do Phone-Use Agents Respect Your Privacy? [97.81424230136075]
  We introduce MyPhoneBench, a verifiable evaluation framework for privacy behavior in mobile agents.<n>We operationalize privacy-respecting phone use as permissioned access, minimal disclosure, and user-controlled memory through a minimal privacy contract.<n>Across five frontier models on 10 mobile apps and 300 tasks, we find that task success, privacy-compliant task completion, and later-session use of saved preferences are distinct capabilities.
  arXiv  Detail & Related papers  (2026-04-01T14:50:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.