Abstract: The ability to quantify incivility online, in news and in congressional
debates, is of great interest to political scientists. Computational tools for
detecting online incivility for English are now fairly accessible and
potentially could be applied more broadly. We test the Jigsaw Perspective API
for its ability to detect the degree of incivility on a corpus that we
developed, consisting of manual annotations of civility in American news. We
demonstrate that toxicity models, as exemplified by Perspective, are inadequate
for the analysis of incivility in news. We carry out error analysis that points
to the need to develop methods to remove spurious correlations between words
often mentioned in the news, especially identity descriptors and incivility.
Without such improvements, applying Perspective or similar models on news is
likely to lead to wrong conclusions, that are not aligned with the human
perception of incivility.