Abstract: A dialogue is essentially a multi-turn interaction among interlocutors.
Effective evaluation metrics should reflect the dynamics of such interaction.
Existing automatic metrics are focused very much on the turn-level quality,
while ignoring such dynamics. To this end, we propose DynaEval, a unified
automatic evaluation framework which is not only capable of performing
turn-level evaluation, but also holistically considers the quality of the
entire dialogue. In DynaEval, the graph convolutional network (GCN) is adopted
to model a dialogue in totality, where the graph nodes denote each individual
utterance and the edges represent the dependency between pairs of utterances. A
contrastive loss is then applied to distinguish well-formed dialogues from
carefully constructed negative samples. Experiments show that DynaEval
significantly outperforms the state-of-the-art dialogue coherence model, and
correlates strongly with human judgements across multiple dialogue evaluation
aspects at both turn and dialogue level.