Concurrent Classifier Error Detection (CCED) in Large Scale Machine Learning Systems

• URL: http://arxiv.org/abs/2306.01820v1
• Date: Fri, 2 Jun 2023 12:36:05 GMT
• Title: Concurrent Classifier Error Detection (CCED) in Large Scale Machine Learning Systems
• Authors: Pedro Reviriego, Ziheng Wang, Alvaro Alonso, Zhen Gao, Farzad Niknia, Shanshan Liu and Fabrizio Lombardi
• Abstract summary: We introduce Concurrent Error Detection (CCED), a scheme to implement CED in Machine Learning systems. CCED identifies a set of check signals in the main ML system and feeds them to the concurrent ML that is trained to detect errors. Results show that more than 95 percent of the errors are detected when using a simple Random Forest classifier.
• Score: 10.839595991409828
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The complexity of Machine Learning (ML) systems increases each year, with current implementations of large language models or text-to-image generators having billions of parameters and requiring billions of arithmetic operations. As these systems are widely utilized, ensuring their reliable operation is becoming a design requirement. Traditional error detection mechanisms introduce circuit or time redundancy that significantly impacts system performance. An alternative is the use of Concurrent Error Detection (CED) schemes that operate in parallel with the system and exploit their properties to detect errors. CED is attractive for large ML systems because it can potentially reduce the cost of error detection. In this paper, we introduce Concurrent Classifier Error Detection (CCED), a scheme to implement CED in ML systems using a concurrent ML classifier to detect errors. CCED identifies a set of check signals in the main ML system and feeds them to the concurrent ML classifier that is trained to detect errors. The proposed CCED scheme has been implemented and evaluated on two widely used large-scale ML models: Contrastive Language Image Pretraining (CLIP) used for image classification and Bidirectional Encoder Representations from Transformers (BERT) used for natural language applications. The results show that more than 95 percent of the errors are detected when using a simple Random Forest classifier that is order of magnitude simpler than CLIP or BERT. These results illustrate the potential of CCED to implement error detection in large-scale ML models.

Related papers

• LM-Combiner: A Contextual Rewriting Model for Chinese Grammatical Error Correction [49.0746090186582]
  Over-correction is a critical problem in Chinese grammatical error correction (CGEC) task. Recent work using model ensemble methods can effectively mitigate over-correction and improve the precision of the GEC system. We propose the LM-Combiner, a rewriting model that can directly modify the over-correction of GEC system outputs without a model ensemble.
  arXiv  Detail & Related papers  (2024-03-26T06:12:21Z)
• Concurrent Linguistic Error Detection (CLED) for Large Language Models [13.01669288397494]
  We propose Concurrent Linguistic Error Detection (CLED) for large language models (LLMs) CLED extracts some linguistic features of the text generated by the LLM and feeds them to a concurrent classifier that detects errors. The proposed CLED scheme has been evaluated on the T5 model when used for news summarization and on the OPUS-MT model when used for translation.
  arXiv  Detail & Related papers  (2024-03-25T03:17:27Z)
• MLAD: A Unified Model for Multi-system Log Anomaly Detection [35.68387377240593]
  We propose MLAD, a novel anomaly detection model that incorporates semantic relational reasoning across multiple systems. Specifically, we employ Sentence-bert to capture the similarities between log sequences and convert them into highly-dimensional learnable semantic vectors. We revamp the formulas of the Attention layer to discern the significance of each keyword in the sequence and model the overall distribution of the multi-system dataset.
  arXiv  Detail & Related papers  (2024-01-15T12:51:13Z)
• Prompt Optimization via Adversarial In-Context Learning [51.18075178593142]
  adv-ICL is implemented as a two-player game between a generator and a discriminator. The generator tries to generate realistic enough output to fool the discriminator. We show that adv-ICL results in significant improvements over state-of-the-art prompt optimization techniques.
  arXiv  Detail & Related papers  (2023-12-05T09:44:45Z)
• Generative error correction for code-switching speech recognition using large language models [49.06203730433107]
  Code-switching (CS) speech refers to the phenomenon of mixing two or more languages within the same sentence. We propose to leverage large language models (LLMs) and lists of hypotheses generated by an ASR to address the CS problem.
  arXiv  Detail & Related papers  (2023-10-17T14:49:48Z)
• Efficient Simulation of Leakage Errors in Quantum Error Correcting Codes Using Tensor Network Methods [0.2209921757303168]
  We present a computationally efficient simulation methodology for studying leakage errors in quantum error correcting codes (QECCs) Our approach enables the simulation of various leakage processes, including thermal noise and coherent errors, without approximations. By leveraging the small amount of entanglement generated during the error correction process, we are able to study large systems, up to a few hundred qudits, over many code cycles.
  arXiv  Detail & Related papers  (2023-08-16T07:36:33Z)
• Witnessing entanglement in trapped-ion quantum error correction under realistic noise [41.94295877935867]
  Quantum Error Correction (QEC) exploits redundancy by encoding logical information into multiple physical qubits. We present a detailed microscopic error model to estimate the average gate infidelity of two-qubit light-shift gates used in trapped-ion platforms. We then apply this realistic error model to quantify the multipartite entanglement generated by circuits that act as QEC building blocks.
  arXiv  Detail & Related papers  (2022-12-14T20:00:36Z)
• Improving Distinction between ASR Errors and Speech Disfluencies with Feature Space Interpolation [0.0]
  Fine-tuning pretrained language models (LMs) is a popular approach to automatic speech recognition (ASR) error detection during post-processing. This paper proposes a scheme to improve existing LM-based ASR error detection systems.
  arXiv  Detail & Related papers  (2021-08-04T02:11:37Z)
• TELESTO: A Graph Neural Network Model for Anomaly Classification in Cloud Services [77.454688257702]
  Machine learning (ML) and artificial intelligence (AI) are applied on IT system operation and maintenance. One direction aims at the recognition of re-occurring anomaly types to enable remediation automation. We propose a method that is invariant to dimensionality changes of given data.
  arXiv  Detail & Related papers  (2021-02-25T14:24:49Z)
• A Novel Anomaly Detection Algorithm for Hybrid Production Systems based on Deep Learning and Timed Automata [73.38551379469533]
  DAD:DeepAnomalyDetection is a new approach for automatic model learning and anomaly detection in hybrid production systems. It combines deep learning and timed automata for creating behavioral model from observations. The algorithm has been applied to few data sets including two from real systems and has shown promising results.
  arXiv  Detail & Related papers  (2020-10-29T08:27:43Z)
• Unsupervised Anomaly Detection with Adversarial Mirrored AutoEncoders [51.691585766702744]
  We propose a variant of Adversarial Autoencoder which uses a mirrored Wasserstein loss in the discriminator to enforce better semantic-level reconstruction. We put forward an alternative measure of anomaly score to replace the reconstruction-based metric. Our method outperforms the current state-of-the-art methods for anomaly detection on several OOD detection benchmarks.
  arXiv  Detail & Related papers  (2020-03-24T08:26:58Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.