Full-Duplex Dialogue Evaluation

Over three years, evaluation could shift from measuring how quickly an agent responds to requiring evidence that it manages the shared opportunity to speak appropriately. The mechanism resembles intersection inspection: measure delays first, identify conflicting actions next, and then turn repeatable findings into operational checks rather than relying on speed alone. In the first year, researchers would define reusable participation labels and report separate scores for meaning, vocal delivery, and control of the conversational floor. Recorded interaction traces and replay tools would help teams examine interrupted exchanges, while multi-round tests would need to accommodate conversations that depart from reference scripts.

In the second year, shared labels could make results comparable enough to support further annotation, which would improve automatic scoring and lower the cost of repeated testing. Speech recognition and tools that identify who spoke when would connect audio events to judgments about whether an agent should continue speaking or yield. Human reviewers would concentrate on ambiguous participation and safety cases, while checking that automatic scores still agree with people on clearer examples of appropriate behavior. If this feedback loop works, teams could run a layered checklist each week instead of arranging a separate user study for every model revision.

By the third year, meeting agents, vehicle systems, and embodied pilots could use setting-specific versions of these checks to support decisions about release or restricted use. Research would assess participation policies through interaction outcomes, with gaze and vocal cues tested alongside response time and computational cost rather than treated as isolated predictions. Multilingual and multiparty calibration would become central because successful behavior in a two-person English exchange does not establish reliability across other languages or social settings. The clearest monitoring cue would be independently reused benchmarks whose separate scores appear in documented release decisions, showing that evaluation has become part of routine development. Poor agreement on yielding or weak alignment between automatic and human scores would undermine that transition, leaving difficult judgments dependent on specialized human studies.

Over three years, this scenario could move full-duplex evaluation toward reactive stress tests that judge acceptable behavior during disruption, rather than mainly counting interruptions or measuring speed. Its mechanism resembles power-grid disturbance testing: a response envelope defines acceptable actions and timing bounds while overlapping speech changes the conditions of an ongoing conversation. In the first year, simulated interlocutors would react to actual model output, introducing unfinished instructions or another speaker instead of forcing every system through a fixed script. Human studies would check whether those disturbances are realistic and whether the resulting failures matter to listeners, since a responsive simulator can still provoke misleading behavior.

A key early signal would be a reproducible ranking reversal, where systems that perform well on simple speed measures fail during sustained overlap or lose instructions across rounds. In the second year, a shared simulator with a versioned specification could let laboratories reproduce such failures and help developers rerun identical checks after model updates. Human studies would increasingly calibrate automated checks, while developers would record when systems choose to speak, wait, or yield so that failures can be traced. If several agents begin sharing conversational spaces, reports of collective disruption could create demand for coexistence tests even when each agent performs acceptably on its own.

By the third year, a layered testing regime could require systems to preserve instructions and safety constraints before awarding credit for useful contributions made without prompting. Automated scoring could handle repetitive checks in validated settings, while people assess unfamiliar situations and recalibrate acceptable behavior for different languages and conversational contexts. Repeated failures would expand the test catalog, giving developers clearer targets, but this same process could reward agents that avoid mistakes by becoming excessively quiet. Engagement checks and human preferences would therefore remain necessary, because acceptable timing alone does not establish that a contribution is useful or socially appropriate. Independent reuse and release decisions tied to named test versions would support this path; persistent simulator disagreement or unchanged rankings would weaken the case for a new regime.

Over three years, this scenario could make buyer acceptance requirements a major force in full-duplex evaluation, rather than assuming that research benchmarks will establish shared practice on their own. The mechanism resembles repeatable telecom tests for simultaneous speech: standardized disturbances make behavior measurable, while purchasing requirements give the resulting measurements practical authority beyond individual research teams. In the first year, researchers would define interruption timing and overlap scoring, keeping content, vocal delivery, and conversational control separate so that speed cannot hide inappropriate responses. Vehicle-assistant and public-contact-center developers would initially use these tools to check whether updates break working behavior, including cases involving multilingual or disfluent speech.

The decisive trigger would be a buyer requiring numeric acceptance criteria, potentially following a proposed interruption protocol from a driver-distraction or accessibility body. In the second year, that requirement could start a reinforcing cycle in which vendors publish results and testing groups reuse shared definitions, making evaluation easier to obtain. Developers could then automate release checks and expose inspectable control outputs that explain participation decisions, even when their speech models otherwise handle those decisions internally. Researchers would compare protocol scores with unfamiliar conversations, because improving on a fixed test does not necessarily mean that an agent has become a better conversational participant. Human-calibrated judgments of appropriateness would supplement scripted tests, while proactive contributions and multiparty participation would remain harder to evaluate through fixed scenarios alone.

By the third year, the deployment package could combine repeatable overlap tests with multi-round instruction and safety checks, with each evaluation layer reported separately rather than collapsed into one score. Rotating held-out conversations and recalibrating human judgments would help expose narrow test learning, while meeting-agent and robot trials could examine whether supplementary assessments transfer to new settings. A concrete monitoring cue is whether a buyer introduces explicit overlap acceptance criteria by September 2027; their absence would substantially weaken this proposed path. Even with adoption, standardized two-person tests would not establish general social competence, because appropriate yielding depends on language and context rather than a single physical reference.