"Model Cards for Model Reporting" in 2024: Reclassifying Category of Ethical Considerations in Terms of Trustworthiness and Risk Management
- URL: http://arxiv.org/abs/2403.15394v1
- Date: Thu, 15 Feb 2024 14:56:00 GMT
- Title: "Model Cards for Model Reporting" in 2024: Reclassifying Category of Ethical Considerations in Terms of Trustworthiness and Risk Management
- Authors: DeBrae Kennedy-Mayo, Jake Gord,
- Abstract summary: In 2019, the paper entitled "Model Cards for Model Reporting" introduced a new tool for documenting model performance.
One of the categories detailed in that paper is ethical considerations, which includes the subcategories of data, human life, mitigations, risks and harms, and use cases.
We propose to reclassify this category in the original model card due to the recent maturing of the field known as trustworthy AI.
- Score: 0.0
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: In 2019, the paper entitled "Model Cards for Model Reporting" introduced a new tool for documenting model performance and encouraged the practice of transparent reporting for a defined list of categories. One of the categories detailed in that paper is ethical considerations, which includes the subcategories of data, human life, mitigations, risks and harms, and use cases. We propose to reclassify this category in the original model card due to the recent maturing of the field known as trustworthy AI, a term which analyzes whether the algorithmic properties of the model indicate that the AI system is deserving of trust from its stakeholders. In our examination of trustworthy AI, we highlight three respected organizations - the European Commission's High-Level Expert Group on AI, the OECD, and the U.S.-based NIST - that have written guidelines on various aspects of trustworthy AI. These recent publications converge on numerous characteristics of the term, including accountability, explainability, fairness, privacy, reliability, robustness, safety, security, and transparency, while recognizing that the implementation of trustworthy AI varies by context. Our reclassification of the original model-card category known as ethical considerations involves a two-step process: 1) adding a new category known as trustworthiness, where the subcategories will be derived from the discussion of trustworthy AI in our paper, and 2) maintaining the subcategories of ethical considerations under a renamed category known as risk environment and risk management, a title which we believe better captures today's understanding of the essence of these topics. We hope that this reclassification will further the goals of the original paper and continue to prompt those releasing trained models to accompany these models with documentation that will assist in the evaluation of their algorithmic properties.
Related papers
- Documenting Ethical Considerations in Open Source AI Models [8.517777178514242]
This study investigates how developers document ethical aspects of open source AI models in practice.
After filtering an initial set of 2,347 documents, we identified 265 relevant ones.
Six themes emerge, with the three largest ones being model behavioural risks, model use cases, and model risk mitigation.
arXiv Detail & Related papers (2024-06-26T05:02:44Z) - Multi-Modal Prompt Learning on Blind Image Quality Assessment [65.0676908930946]
Image Quality Assessment (IQA) models benefit significantly from semantic information, which allows them to treat different types of objects distinctly.
Traditional methods, hindered by a lack of sufficiently annotated data, have employed the CLIP image-text pretraining model as their backbone to gain semantic awareness.
Recent approaches have attempted to address this mismatch using prompt technology, but these solutions have shortcomings.
This paper introduces an innovative multi-modal prompt-based methodology for IQA.
arXiv Detail & Related papers (2024-04-23T11:45:32Z) - Unpacking the Ethical Value Alignment in Big Models [46.560886177083084]
This paper provides an overview of the risks and challenges associated with big models, surveys existing AI ethics guidelines, and examines the ethical implications arising from the limitations of these models.
We introduce a novel conceptual paradigm for aligning the ethical values of big models and discuss promising research directions for alignment criteria, evaluation, and method.
arXiv Detail & Related papers (2023-10-26T16:45:40Z) - Evaluating the Fairness of Discriminative Foundation Models in Computer
Vision [51.176061115977774]
We propose a novel taxonomy for bias evaluation of discriminative foundation models, such as Contrastive Language-Pretraining (CLIP)
We then systematically evaluate existing methods for mitigating bias in these models with respect to our taxonomy.
Specifically, we evaluate OpenAI's CLIP and OpenCLIP models for key applications, such as zero-shot classification, image retrieval and image captioning.
arXiv Detail & Related papers (2023-10-18T10:32:39Z) - AI Risk Profiles: A Standards Proposal for Pre-Deployment AI Risk
Disclosures [0.8702432681310399]
We propose a risk profiling standard which can guide downstream decision-making.
The standard is built on our proposed taxonomy of AI risks, which reflects a high-level categorization of the wide variety of risks proposed in the literature.
We apply this methodology to a number of prominent AI systems using publicly available information.
arXiv Detail & Related papers (2023-09-22T20:45:15Z) - An Audit Framework for Technical Assessment of Binary Classifiers [0.0]
Multilevel models using logistic regression (MLogRM) and random forest models (RFM) are increasingly deployed in industry for the purpose of binary classification.
The European Commission's proposed Artificial Intelligence Act (AIA) necessitates, under certain conditions, that application of such models is fair, transparent, and ethical.
This paper proposes and demonstrates an audit framework for technical assessment of RFMs and MLogRMs by focussing on model-, discrimination, and transparency & explainability-related aspects.
arXiv Detail & Related papers (2022-11-17T12:48:11Z) - Novel Class Discovery without Forgetting [72.52222295216062]
We identify and formulate a new, pragmatic problem setting of NCDwF: Novel Class Discovery without Forgetting.
We propose a machine learning model to incrementally discover novel categories of instances from unlabeled data.
We introduce experimental protocols based on CIFAR-10, CIFAR-100 and ImageNet-1000 to measure the trade-off between knowledge retention and novel class discovery.
arXiv Detail & Related papers (2022-07-21T17:54:36Z) - Multisource AI Scorecard Table for System Evaluation [3.74397577716445]
The paper describes a Multisource AI Scorecard Table (MAST) that provides the developer and user of an artificial intelligence (AI)/machine learning (ML) system with a standard checklist.
The paper explores how the analytic tradecraft standards outlined in Intelligence Community Directive (ICD) 203 can provide a framework for assessing the performance of an AI system.
arXiv Detail & Related papers (2021-02-08T03:37:40Z) - Trustworthy AI [75.99046162669997]
Brittleness to minor adversarial changes in the input data, ability to explain the decisions, address the bias in their training data, are some of the most prominent limitations.
We propose the tutorial on Trustworthy AI to address six critical issues in enhancing user and public trust in AI systems.
arXiv Detail & Related papers (2020-11-02T20:04:18Z) - A Methodology for Creating AI FactSheets [67.65802440158753]
This paper describes a methodology for creating the form of AI documentation we call FactSheets.
Within each step of the methodology, we describe the issues to consider and the questions to explore.
This methodology will accelerate the broader adoption of transparent AI documentation.
arXiv Detail & Related papers (2020-06-24T15:08:59Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.