Abstract: The main objective of eXplainable Artificial Intelligence (XAI) is to provide
effective explanations for black-box classifiers. The existing literature lists
many desirable properties for explanations to be useful, but there is no
consensus on how to quantitatively evaluate explanations in practice. Moreover,
explanations are typically used only to inspect black-box models, and the
proactive use of explanations as a decision support is generally overlooked.
Among the many approaches to XAI, a widely adopted paradigm is Local Linear
Explanations - with LIME and SHAP emerging as state-of-the-art methods. We show
that these methods are plagued by many defects including unstable explanations,
divergence of actual implementations from the promised theoretical properties,
and explanations for the wrong label. This highlights the need to have standard
and unbiased evaluation procedures for Local Linear Explanations in the XAI
field. In this paper we address the problem of identifying a clear and
unambiguous set of metrics for the evaluation of Local Linear Explanations.
This set includes both existing and novel metrics defined specifically for this
class of explanations. All metrics have been included in an open Python
framework, named LEAF. The purpose of LEAF is to provide a reference for end
users to evaluate explanations in a standardised and unbiased way, and to guide
researchers towards developing improved explainable techniques.