Do-AIQ: A Design-of-Experiment Approach to Quality Evaluation of AI Mislabel Detection Algorithm

• URL: http://arxiv.org/abs/2208.09953v1
• Date: Sun, 21 Aug 2022 19:47:41 GMT
• Title: Do-AIQ: A Design-of-Experiment Approach to Quality Evaluation of AI Mislabel Detection Algorithm
• Authors: J. Lian, K. Choi, B. Veeramani, A. Hu, L. Freeman, E. Bowen, X. Deng
• Abstract summary: The quality of Artificial Intelligence (AI) algorithms is of significant importance for confidently adopting algorithms in various applications such as cybersecurity, healthcare, and autonomous driving. This work presents a principled framework of using a design-of-experimental approach to systematically evaluate the quality of AI algorithms, named as Do-AIQ.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The quality of Artificial Intelligence (AI) algorithms is of significant importance for confidently adopting algorithms in various applications such as cybersecurity, healthcare, and autonomous driving. This work presents a principled framework of using a design-of-experimental approach to systematically evaluate the quality of AI algorithms, named as Do-AIQ. Specifically, we focus on investigating the quality of the AI mislabel data algorithm against data poisoning. The performance of AI algorithms is affected by hyperparameters in the algorithm and data quality, particularly, data mislabeling, class imbalance, and data types. To evaluate the quality of the AI algorithms and obtain a trustworthy assessment on the quality of the algorithms, we establish a design-of-experiment framework to construct an efficient space-filling design in a high-dimensional constraint space and develop an effective surrogate model using additive Gaussian process to enable the emulation of the quality of AI algorithms. Both theoretical and numerical studies are conducted to justify the merits of the proposed framework. The proposed framework can set an exemplar for AI algorithm to enhance the AI assurance of robustness, reproducibility, and transparency.

Related papers

• AI-Aided Kalman Filters [65.35350122917914]
  The Kalman filter (KF) and its variants are among the most celebrated algorithms in signal processing. Recent developments illustrate the possibility of fusing deep neural networks (DNNs) with classic Kalman-type filtering. This article provides a tutorial-style overview of design approaches for incorporating AI in aiding KF-type algorithms.
  arXiv  Detail & Related papers  (2024-10-16T06:47:53Z)
• Beyond Algorithmic Fairness: A Guide to Develop and Deploy Ethical AI-Enabled Decision-Support Tools [0.0]
  The integration of artificial intelligence (AI) and optimization hold substantial promise for improving the efficiency, reliability, and resilience of engineered systems. This paper identifies ethical considerations required when deploying algorithms at the intersection of AI and optimization. Rather than providing a prescriptive set of rules, this paper aims to foster reflection and awareness among researchers.
  arXiv  Detail & Related papers  (2024-09-17T18:37:53Z)
• Computer Vision Intelligence Test Modeling and Generation: A Case Study on Smart OCR [3.0561992956541606]
  We first present a comprehensive literature review of previous work, covering key facets of AI software testing processes. We then introduce a 3D classification model to systematically evaluate the image-based text extraction AI function. To evaluate the performance of our proposed AI software quality test, we propose four evaluation metrics to cover different aspects.
  arXiv  Detail & Related papers  (2024-09-14T23:33:28Z)
• Towards Efficient and Trustworthy AI Through Hardware-Algorithm-Communication Co-Design [32.815326729969904]
  State-of-the-art AI models are largely incapable of providing trustworthy measures of their uncertainty. This paper highlights research directions at the intersection of hardware and software design.
  arXiv  Detail & Related papers  (2023-09-27T18:39:46Z)
• Human-Centric Multimodal Machine Learning: Recent Advances and Testbed on AI-based Recruitment [66.91538273487379]
  There is a certain consensus about the need to develop AI applications with a Human-Centric approach. Human-Centric Machine Learning needs to be developed based on four main requirements: (i) utility and social good; (ii) privacy and data ownership; (iii) transparency and accountability; and (iv) fairness in AI-driven decision-making processes. We study how current multimodal algorithms based on heterogeneous sources of information are affected by sensitive elements and inner biases in the data.
  arXiv  Detail & Related papers  (2023-02-13T16:44:44Z)
• On Robust Numerical Solver for ODE via Self-Attention Mechanism [82.95493796476767]
  We explore training efficient and robust AI-enhanced numerical solvers with a small data size by mitigating intrinsic noise disturbances. We first analyze the ability of the self-attention mechanism to regulate noise in supervised learning and then propose a simple-yet-effective numerical solver, Attr, which introduces an additive self-attention mechanism to the numerical solution of differential equations.
  arXiv  Detail & Related papers  (2023-02-05T01:39:21Z)
• Enabling AI-Generated Content (AIGC) Services in Wireless Edge Networks [68.00382171900975]
  In wireless edge networks, the transmission of incorrectly generated content may unnecessarily consume network resources. We present the AIGC-as-a-service concept and discuss the challenges in deploying A at the edge networks. We propose a deep reinforcement learning-enabled algorithm for optimal ASP selection.
  arXiv  Detail & Related papers  (2023-01-09T09:30:23Z)
• RL-EA: A Reinforcement Learning-Based Evolutionary Algorithm Framework for Electromagnetic Detection Satellite Scheduling Problem [6.438148195340613]
  This paper proposes a mixed-integer programming model for the EDSSP problem and an evolutionary algorithm framework based on reinforcement learning (RL-EA) Various scales experiments are used to examine the planning effect of the proposed algorithm.
  arXiv  Detail & Related papers  (2022-06-12T08:53:56Z)
• Stochastic Hard Thresholding Algorithms for AUC Maximization [49.00683387735522]
  We develop a hard thresholding algorithm for AUC in distributiond classification. We conduct experiments to show the efficiency and effectiveness of the proposed algorithms.
  arXiv  Detail & Related papers  (2020-11-04T16:49:29Z)
• Investigating the Robustness of Artificial Intelligent Algorithms with Mixture Experiments [1.877815076482061]
  robustness of AI algorithms is of great interest as inaccurate prediction could result in safety concerns and limit the adoption of AI systems. A robust classification algorithm is expected to have high accuracy and low variability under different application scenarios. We conduct a comprehensive set of mixture experiments to collect prediction performance results. Then statistical analyses are conducted to understand how various factors affect the robustness of AI classification algorithms.
  arXiv  Detail & Related papers  (2020-10-10T15:38:53Z)
• Bias in Multimodal AI: Testbed for Fair Automatic Recruitment [73.85525896663371]
  We study how current multimodal algorithms based on heterogeneous sources of information are affected by sensitive elements and inner biases in the data. We train automatic recruitment algorithms using a set of multimodal synthetic profiles consciously scored with gender and racial biases. Our methodology and results show how to generate fairer AI-based tools in general, and in particular fairer automated recruitment systems.
  arXiv  Detail & Related papers  (2020-04-15T15:58:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.