ProMediConv: Benchmarking Proactive Conversational Agents in Legal Dispute Mediation
Abstract Overview
ProMediConv is a benchmark for legal dispute mediation that models mediation as a proactive, multi-stage, multi-party dialogue process rather than a simple reactive conversation. The framework defines three mediation stages, 11 mediation strategies, and four party behavior pattern (BP) states, and it is paired with a high-fidelity dataset reconstructed from 972 real-world mediation cases with utterance-level strategy and BP annotations. The paper also introduces MAD (Mean Attribute Difference), a metric designed to measure within-dialogue changes in parties' BP states, addressing limitations of outcome-only evaluation. Using this setup, the authors benchmark general LLMs, legal-specific LLMs, and a trained baseline agent to study the current capabilities and limitations of mediation-oriented conversational systems.
Novelty
The paper's main novelty is the combination of a realistic legal mediation benchmark, an annotated dataset built from authentic cases, and a process-sensitive evaluation metric for tracking party-state change during dialogue. It is distinctive in explicitly modeling proactive, strategy-driven, multi-party mediation and in using MAD to quantify whether an agent meaningfully shifts disputants' behavior patterns rather than only reaching a superficial final outcome.
Results
Across benchmark experiments, current general and legal-specific LLMs struggle with complex mediation, and prompt-based policy-planning methods provide unstable gains while increasing token cost. The authors' trained baseline, ProMediAgent, achieves the best reported SR@t, AT, and SSR values in the main evaluation (0.9071, 10.34, and 0.9428, respectively). The analyses further demonstrate that MAD captures quality differences missed by efficiency-focused metrics, exposes superficial short-cut resolutions, and correlates more strongly with human success judgments than SR@t.
Key Points
- ProMediConv formalizes legal mediation as a proactive three-stage, multi-party dialogue task with 11 mediation strategies and four BP states.
- The dataset contains 972 reconstructed real-world cases with utterance-level annotations, providing broader realism and scale than simplified or synthetic mediation setups.
- MAD offers a finer-grained view of mediation quality by measuring party-state shifts, exposing quality limitations that standard success and turn-based metrics miss.