VECHR: A Dataset for Explainable and Robust Classification of Vulnerability Type in the European Court of Human Rights

• URL: http://arxiv.org/abs/2310.11368v4
• Date: Tue, 24 Oct 2023 12:30:25 GMT
• Title: VECHR: A Dataset for Explainable and Robust Classification of Vulnerability Type in the European Court of Human Rights
• Authors: Shanshan Xu, Leon Staufer, T.Y.S.S Santosh, Oana Ichim, Corina Heri, Matthias Grabmair
• Abstract summary: We present VECHR, a novel expert-annotated multi-label dataset of vulnerability type classification and explanation rationale. We benchmark the performance of state-of-the-art models on VECHR from both prediction and explainability perspectives. Our dataset poses unique challenges offering significant room for improvement regarding performance, explainability, and robustness.
• Score: 2.028075209232085
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Recognizing vulnerability is crucial for understanding and implementing targeted support to empower individuals in need. This is especially important at the European Court of Human Rights (ECtHR), where the court adapts Convention standards to meet actual individual needs and thus ensures effective human rights protection. However, the concept of vulnerability remains elusive at the ECtHR and no prior NLP research has dealt with it. To enable future research in this area, we present VECHR, a novel expert-annotated multi-label dataset comprising of vulnerability type classification and explanation rationale. We benchmark the performance of state-of-the-art models on VECHR from both prediction and explainability perspectives. Our results demonstrate the challenging nature of the task with lower prediction performance and limited agreement between models and experts. Further, we analyze the robustness of these models in dealing with out-of-domain (OOD) data and observe overall limited performance. Our dataset poses unique challenges offering significant room for improvement regarding performance, explainability, and robustness.

Related papers

• DePrompt: Desensitization and Evaluation of Personal Identifiable Information in Large Language Model Prompts [11.883785681042593]
  DePrompt is a desensitization protection and effectiveness evaluation framework for prompt. We integrate contextual attributes to define privacy types, achieving high-precision PII entity identification. Our framework is adaptable to prompts and can be extended to text usability-dependent scenarios.
  arXiv  Detail & Related papers  (2024-08-16T02:38:25Z)
• An evidence-based methodology for human rights impact assessment (HRIA) in the development of AI data-intensive systems [49.1574468325115]
  We show that human rights already underpin the decisions in the field of data use. This work presents a methodology and a model for a Human Rights Impact Assessment (HRIA) The proposed methodology is tested in concrete case-studies to prove its feasibility and effectiveness.
  arXiv  Detail & Related papers  (2024-07-30T16:27:52Z)
• Towards Explainability and Fairness in Swiss Judgement Prediction: Benchmarking on a Multilingual Dataset [2.7463268699570134]
  This study delves into the realm of explainability and fairness in Legal Judgement Prediction (LJP) models. We evaluate the explainability performance of state-of-the-art monolingual and multilingual BERT-based LJP models. We introduce a novel evaluation framework, Lower Court Insertion (LCI), which allows us to quantify the influence of lower court information on model predictions.
  arXiv  Detail & Related papers  (2024-02-26T20:42:40Z)
• A New Perspective on Evaluation Methods for Explainable Artificial Intelligence (XAI) [0.0]
  We argue that it is best approached in a nuanced way that incorporates resource availability, domain characteristics, and considerations of risk. This work aims to advance the field of Requirements Engineering for AI.
  arXiv  Detail & Related papers  (2023-07-26T15:15:44Z)
• Revisiting the Performance-Explainability Trade-Off in Explainable Artificial Intelligence (XAI) [0.0]
  We argue that it is best approached in a nuanced way that incorporates resource availability, domain characteristics, and considerations of risk. This work aims to advance the field of Requirements Engineering for AI.
  arXiv  Detail & Related papers  (2023-07-26T15:07:40Z)
• Deconfounding Legal Judgment Prediction for European Court of Human Rights Cases Towards Better Alignment with Experts [1.252149409594807]
  This work demonstrates that Legal Judgement Prediction systems without expert-informed adjustments can be vulnerable to shallow, distracting surface signals. To mitigate this, we use domain expertise to strategically identify statistically predictive but legally irrelevant information.
  arXiv  Detail & Related papers  (2022-10-25T08:37:25Z)
• SF-PATE: Scalable, Fair, and Private Aggregation of Teacher Ensembles [50.90773979394264]
  This paper studies a model that protects the privacy of individuals' sensitive information while also allowing it to learn non-discriminatory predictors. A key characteristic of the proposed model is to enable the adoption of off-the-selves and non-private fair models to create a privacy-preserving and fair model.
  arXiv  Detail & Related papers  (2022-04-11T14:42:54Z)
• Counterfactual Explanations as Interventions in Latent Space [62.997667081978825]
  Counterfactual explanations aim to provide to end users a set of features that need to be changed in order to achieve a desired outcome. Current approaches rarely take into account the feasibility of actions needed to achieve the proposed explanations. We present Counterfactual Explanations as Interventions in Latent Space (CEILS), a methodology to generate counterfactual explanations.
  arXiv  Detail & Related papers  (2021-06-14T20:48:48Z)
• Exploring Robustness of Unsupervised Domain Adaptation in Semantic Segmentation [74.05906222376608]
  We propose adversarial self-supervision UDA (or ASSUDA) that maximizes the agreement between clean images and their adversarial examples by a contrastive loss in the output space. This paper is rooted in two observations: (i) the robustness of UDA methods in semantic segmentation remains unexplored, which pose a security concern in this field; and (ii) although commonly used self-supervision (e.g., rotation and jigsaw) benefits image tasks such as classification and recognition, they fail to provide the critical supervision signals that could learn discriminative representation for segmentation tasks.
  arXiv  Detail & Related papers  (2021-05-23T01:50:44Z)
• Accurate and Robust Feature Importance Estimation under Distribution Shifts [49.58991359544005]
  PRoFILE is a novel feature importance estimation method. We show significant improvements over state-of-the-art approaches, both in terms of fidelity and robustness.
  arXiv  Detail & Related papers  (2020-09-30T05:29:01Z)
• Differentially Private and Fair Deep Learning: A Lagrangian Dual Approach [54.32266555843765]
  This paper studies a model that protects the privacy of the individuals sensitive information while also allowing it to learn non-discriminatory predictors. The method relies on the notion of differential privacy and the use of Lagrangian duality to design neural networks that can accommodate fairness constraints.
  arXiv  Detail & Related papers  (2020-09-26T10:50:33Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.