Enterprise Disk Drive Scrubbing Based on Mondrian Conformal Predictors

• URL: http://arxiv.org/abs/2306.17169v1
• Date: Thu, 1 Jun 2023 04:11:22 GMT
• Title: Enterprise Disk Drive Scrubbing Based on Mondrian Conformal Predictors
• Authors: Rahul Vishwakarma, Jinha Hwang, Soundouss Messoudi, Ava Hedayatipour
• Abstract summary: scrubbing the entire storage array at once can adversely impact system performance. We propose a selective disk scrubbing method that enhances the overall reliability and power efficiency in data centers. By scrubbing just 22.7% of the total storage disks, we can achieve optimized energy consumption and reduce the carbon footprint of the data center.
• Score: 1.290382979353427
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Disk scrubbing is a process aimed at resolving read errors on disks by reading data from the disk. However, scrubbing the entire storage array at once can adversely impact system performance, particularly during periods of high input/output operations. Additionally, the continuous reading of data from disks when scrubbing can result in wear and tear, especially on larger capacity disks, due to the significant time and energy consumption involved. To address these issues, we propose a selective disk scrubbing method that enhances the overall reliability and power efficiency in data centers. Our method employs a Machine Learning model based on Mondrian Conformal prediction to identify specific disks for scrubbing, by proactively predicting the health status of each disk in the storage pool, forecasting n-days in advance, and using an open-source dataset. For disks predicted as non-healthy, we mark them for replacement without further action. For healthy drives, we create a set and quantify their relative health across the entire storage pool based on the predictor's confidence. This enables us to prioritize selective scrubbing for drives with established scrubbing frequency based on the scrub cycle. The method we propose provides an efficient and dependable solution for managing enterprise disk drives. By scrubbing just 22.7% of the total storage disks, we can achieve optimized energy consumption and reduce the carbon footprint of the data center.

Related papers

• Optimizing SSD Caches for Cloud Block Storage Systems Using Machine Learning Approaches [40.13303683102544]
  This paper proposes a novel approach to dynamically optimize the write policy in cloud-based storage systems. The proposed method identifies write-only data and selectively filters it out in real-time, thereby minimizing the number of unnecessary write operations.
  arXiv  Detail & Related papers  (2024-12-29T17:37:18Z)
• Dataset Growth [59.68869191071907]
  InfoGrowth is an efficient online algorithm for data cleaning and selection. It can improve data quality/efficiency on both single-modal and multi-modal tasks.
  arXiv  Detail & Related papers  (2024-05-28T16:43:57Z)
• Disk failure prediction based on multi-layer domain adaptive learning [6.757777833155211]
  This paper presents a novel method for predicting disk failures by leveraging multi-layer domain adaptive learning techniques. It has been demonstrated that the proposed technique can generate a reliable prediction model and improve the ability to predict failures on disk data with few failure samples.
  arXiv  Detail & Related papers  (2023-10-10T11:28:40Z)
• TFBEST: Dual-Aspect Transformer with Learnable Positional Encoding for Failure Prediction [1.223779595809275]
  We propose a Temporal-fusion Bi-encoder Self-attention Transformer (TFBEST) for predicting failures in hard-drives. It is an encoder-decoder based deep learning technique that enhances the context gained from understanding health statistics. Experiments on HDD data show that our method significantly outperforms the state-of-the-art RUL prediction methods.
  arXiv  Detail & Related papers  (2023-09-06T01:03:14Z)
• Fast Machine Unlearning Without Retraining Through Selective Synaptic Dampening [51.34904967046097]
  Selective Synaptic Dampening (SSD) is a fast, performant, and does not require long-term storage of the training data. We present a novel two-step, post hoc, retrain-free approach to machine unlearning which is fast, performant, and does not require long-term storage of the training data.
  arXiv  Detail & Related papers  (2023-08-15T11:30:45Z)
• Meta-prediction Model for Distillation-Aware NAS on Unseen Datasets [55.2118691522524]
  Distillation-aware Neural Architecture Search (DaNAS) aims to search for an optimal student architecture. We propose a distillation-aware meta accuracy prediction model, DaSS (Distillation-aware Student Search), which can predict a given architecture's final performances on a dataset.
  arXiv  Detail & Related papers  (2023-05-26T14:00:35Z)
• Large-scale End-of-Life Prediction of Hard Disks in Distributed Datacenters [0.0]
  Large-scale predictive analyses are performed using severely skewed health statistics data. We present an encoder-decoder LSTM model where the context gained from understanding health statistics sequences aid in predicting an output sequence of the number of days remaining before a disk potentially fails.
  arXiv  Detail & Related papers  (2023-03-15T21:55:07Z)
• UNFUSED: UNsupervised Finetuning Using SElf supervised Distillation [53.06337011259031]
  We introduce UnFuSeD, a novel approach to leverage self-supervised learning for audio classification. We use the encoder to generate pseudo-labels for unsupervised fine-tuning before the actual fine-tuning step. UnFuSeD achieves state-of-the-art results on the LAPE Benchmark, significantly outperforming all our baselines.
  arXiv  Detail & Related papers  (2023-03-10T02:43:36Z)
• AutoShard: Automated Embedding Table Sharding for Recommender Systems [54.82606459574231]
  We introduce our novel practice in Meta, namely AutoShard, which uses a neural cost model to directly predict the multi-table costs. AutoShard can efficiently shard hundreds of tables in seconds. Our algorithms have been deployed in Meta production environment.
  arXiv  Detail & Related papers  (2022-08-12T17:48:01Z)
• Remaining Useful Life Estimation of Hard Disk Drives using Bidirectional LSTM Networks [0.0]
  We introduce methods of extracting meaningful attributes associated with operational failure and of pre-processing health statistics data. We use a Bidirectional LSTM with a multi-day look back period to learn the temporal progression of health indicators and baseline them against vanilla LSTM and Random Forest models. Our approach can predict the occurrence of disk failure with an accuracy of 96.4% considering test data 60 days before failure.
  arXiv  Detail & Related papers  (2021-09-11T19:26:07Z)
• Decision-making Oriented Clustering: Application to Pricing and Power Consumption Scheduling [61.062312682535755]
  We formulate the framework of decision-making oriented clustering and propose an algorithm providing a decision-based partition of the data space and good representative decisions. By applying this novel framework and algorithm to a typical problem of real-time pricing and that of power consumption scheduling, we obtain several insightful analytical results.
  arXiv  Detail & Related papers  (2021-06-02T08:41:04Z)
• STANNIS: Low-Power Acceleration of Deep Neural Network Training Using Computational Storage [1.4680035572775534]
  This paper proposes a framework for distributed, in-storage training of neural networks on clusters of computational storage devices. Such devices not only contain hardware accelerators but also eliminate data movement between the host and storage, resulting in both improved performance and power savings.
  arXiv  Detail & Related papers  (2020-02-17T19:22:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.