ANTONIO: Towards a Systematic Method of Generating NLP Benchmarks for Verification

• URL: http://arxiv.org/abs/2305.04003v3
• Date: Tue, 15 Aug 2023 15:09:02 GMT
• Title: ANTONIO: Towards a Systematic Method of Generating NLP Benchmarks for Verification
• Authors: Marco Casadio, Luca Arnaboldi, Matthew L. Daggitt, Omri Isac, Tanvi Dinkar, Daniel Kienitz, Verena Rieser, Ekaterina Komendantskaya
• Abstract summary: Many known neural network verification methods that work for computer vision and other numeric datasets do not work for NLP. Here, we study technical reasons that underlie this problem. We implement these methods as a Python library called ANTONIO that links to the neural network verifiers ERAN and Marabou.
• Score: 11.063566569882186
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Verification of machine learning models used in Natural Language Processing (NLP) is known to be a hard problem. In particular, many known neural network verification methods that work for computer vision and other numeric datasets do not work for NLP. Here, we study technical reasons that underlie this problem. Based on this analysis, we propose practical methods and heuristics for preparing NLP datasets and models in a way that renders them amenable to known verification methods based on abstract interpretation. We implement these methods as a Python library called ANTONIO that links to the neural network verifiers ERAN and Marabou. We perform evaluation of the tool using an NLP dataset R-U-A-Robot suggested as a benchmark for verifying legally critical NLP applications. We hope that, thanks to its general applicability, this work will open novel possibilities for including NLP verification problems into neural network verification competitions, and will popularise NLP problems within this community.

Related papers

• NLP-ADBench: NLP Anomaly Detection Benchmark [9.445800367013744]
  We introduce NLP-ADBench, the most comprehensive benchmark for NLP anomaly detection. No single model excels across all datasets, highlighting the need for automated model selection. Two-step methods leveraging transformer-based embeddings consistently outperform specialized end-to-end approaches.
  arXiv  Detail & Related papers  (2024-12-06T05:30:41Z)
• NLP4PBM: A Systematic Review on Process Extraction using Natural Language Processing with Rule-based, Machine and Deep Learning Methods [0.0]
  This literature review studies the field of automated process extraction, i.e., transforming textual descriptions into structured processes using Natural Language Processing (NLP) We found that Machine Learning (ML) / Deep Learning (DL) methods are being increasingly used for the NLP component. In some cases, they were chosen for their suitability towards process extraction, and results show that they can outperform classic rule-based methods.
  arXiv  Detail & Related papers  (2024-09-10T15:16:02Z)
• Targeted Visualization of the Backbone of Encoder LLMs [46.453758431767724]
  Attention based large language models (LLMs) are the state-of-the-art in natural language processing (NLP) Despite the success of encoder models, on which we focus in this work, they also bear several risks, including issues with bias or their susceptibility for adversarial attacks. We investigate the application of DeepView, a method for visualizing a part of the decision function together with a data set in two dimensions, to the NLP domain.
  arXiv  Detail & Related papers  (2024-03-26T12:51:02Z)
• Natural Language Processing for Dialects of a Language: A Survey [56.93337350526933]
  State-of-the-art natural language processing (NLP) models are trained on massive training corpora, and report a superlative performance on evaluation datasets. This survey delves into an important attribute of these datasets: the dialect of a language. Motivated by the performance degradation of NLP models for dialectal datasets and its implications for the equity of language technologies, we survey past research in NLP for dialects in terms of datasets, and approaches.
  arXiv  Detail & Related papers  (2024-01-11T03:04:38Z)
• Meta Learning for Natural Language Processing: A Survey [88.58260839196019]
  Deep learning has been the mainstream technique in natural language processing (NLP) area. Deep learning requires many labeled data and is less generalizable across domains. Meta-learning is an arising field in machine learning studying approaches to learn better algorithms.
  arXiv  Detail & Related papers  (2022-05-03T13:58:38Z)
• Interpreting Deep Learning Models in Natural Language Processing: A Review [33.80537635077772]
  A long-standing criticism against neural network models is the lack of interpretability. In this survey, we provide a comprehensive review of various interpretation methods for neural models in NLP.
  arXiv  Detail & Related papers  (2021-10-20T10:17:04Z)
• Combining Feature and Instance Attribution to Detect Artifacts [62.63504976810927]
  We propose methods to facilitate identification of training data artifacts. We show that this proposed training-feature attribution approach can be used to uncover artifacts in training data. We execute a small user study to evaluate whether these methods are useful to NLP researchers in practice.
  arXiv  Detail & Related papers  (2021-07-01T09:26:13Z)
• Model Explainability in Deep Learning Based Natural Language Processing [0.0]
  We reviewed and compared some popular machine learning model explainability methodologies. We applied one of the NLP explainability methods to a NLP classification model. We identified some common issues due to the special natures of NLP models.
  arXiv  Detail & Related papers  (2021-06-14T13:23:20Z)
• FedNLP: A Research Platform for Federated Learning in Natural Language Processing [55.01246123092445]
  We present the FedNLP, a research platform for federated learning in NLP. FedNLP supports various popular task formulations in NLP such as text classification, sequence tagging, question answering, seq2seq generation, and language modeling. Preliminary experiments with FedNLP reveal that there exists a large performance gap between learning on decentralized and centralized datasets.
  arXiv  Detail & Related papers  (2021-04-18T11:04:49Z)
• NSL: Hybrid Interpretable Learning From Noisy Raw Data [66.15862011405882]
  This paper introduces a hybrid neural-symbolic learning framework, called NSL, that learns interpretable rules from labelled unstructured data. NSL combines pre-trained neural networks for feature extraction with FastLAS, a state-of-the-art ILP system for rule learning under the answer set semantics. We demonstrate that NSL is able to learn robust rules from MNIST data and achieve comparable or superior accuracy when compared to neural network and random forest baselines.
  arXiv  Detail & Related papers  (2020-12-09T13:02:44Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.