Evaluating Fault Tolerance and Scalability in Distributed File Systems: A Case Study of GFS, HDFS, and MinIO

• URL: http://arxiv.org/abs/2502.01981v2
• Date: Fri, 28 Feb 2025 20:52:39 GMT
• Title: Evaluating Fault Tolerance and Scalability in Distributed File Systems: A Case Study of GFS, HDFS, and MinIO
• Authors: Shubham Malhotra, Fnu Yashu, Muhammad Saqib, Dipkumar Mehta, Jagdish Jangid, Sachin Dixit,
• Abstract summary: Distributed File Systems (DFS) are essential for managing vast datasets across multiple servers, offering benefits in scalability, fault tolerance, and data accessibility.<n>This paper presents a comprehensive evaluation of three prominent DFSs - Google File System (GFS), Hadoop Distributed File System (HDFS), and MinIO.<n>Through detailed analysis, how these systems handle data redundancy, server failures, and client access protocols, ensuring reliability in dynamic, large-scale environments is assessed.
• Score: 0.9307293959047378
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Distributed File Systems (DFS) are essential for managing vast datasets across multiple servers, offering benefits in scalability, fault tolerance, and data accessibility. This paper presents a comprehensive evaluation of three prominent DFSs - Google File System (GFS), Hadoop Distributed File System (HDFS), and MinIO - focusing on their fault tolerance mechanisms and scalability under varying data loads and client demands. Through detailed analysis, how these systems handle data redundancy, server failures, and client access protocols, ensuring reliability in dynamic, large-scale environments is assessed. In addition, the impact of system design on performance, particularly in distributed cloud and computing architectures is assessed. By comparing the strengths and limitations of each DFS, the paper provides practical insights for selecting the most appropriate system for different enterprise needs, from high availability storage to big data analytics.

Related papers

• C-FedRAG: A Confidential Federated Retrieval-Augmented Generation System [7.385458207094507]
  We introduce Confidential Computing (CC) techniques as a solution for secure Federated Retrieval Augmented Generation (FedRAG)<n>Our proposed Confidential FedRAG system (C-FedRAG) enables secure connection and scaling of a RAG across a decentralized network of data providers by ensuring context confidentiality.
  arXiv  Detail & Related papers  (2024-12-17T18:42:21Z)
• Reproduction Research of FSA-Benchmark [0.0]
  Failure-slow disks experience a gradual decline in performance before ultimately failing.<n>Unlike outright disk failures, fail-slow conditions can go undetected for prolonged periods, leading to considerable impacts on system performance and user experience.
  arXiv  Detail & Related papers  (2024-12-12T01:31:11Z)
• G-NAS: Generalizable Neural Architecture Search for Single Domain Generalization Object Detection [55.86838901572496]
  Differentiable Neural Architecture Search (NAS) is known for its high capacity for complex data fitting. Generalizable loss (G-loss) is an OoD-aware objective, preventing NAS from over-fitting. Experimental results on the S-DGOD urban-scene datasets demonstrate that the proposed G-NAS achieves SOTA performance compared to baseline methods.
  arXiv  Detail & Related papers  (2024-02-07T08:57:59Z)
• Towards Learned Predictability of Storage Systems [0.0]
  Storage systems have become a fundamental building block of datacenters. Despite the growing popularity and interests in storage, designing and implementing reliable storage systems remains challenging. To move towards predictability of storage systems, various mechanisms and field studies have been proposed in the past few years. Based on three representative research works, we discuss where and how machine learning should be applied in this field.
  arXiv  Detail & Related papers  (2023-07-30T17:53:08Z)
• Distributed Traffic Synthesis and Classification in Edge Networks: A Federated Self-supervised Learning Approach [83.2160310392168]
  This paper proposes FS-GAN to support automatic traffic analysis and synthesis over a large number of heterogeneous datasets. FS-GAN is composed of multiple distributed Generative Adversarial Networks (GANs) FS-GAN can classify data of unknown types of service and create synthetic samples that capture the traffic distribution of the unknown types.
  arXiv  Detail & Related papers  (2023-02-01T03:23:11Z)
• Block size estimation for data partitioning in HPC applications using machine learning techniques [38.063905789566746]
  This paper describes a methodology, namely BLEST-ML (BLock size ESTimation through Machine Learning), for block size estimation. The proposed methodology was evaluated by designing an implementation tailored to dislib, a distributed computing library. The results we obtained show the ability of BLEST-ML to efficiently determine a suitable way to split a given dataset.
  arXiv  Detail & Related papers  (2022-11-19T23:04:14Z)
• MIX-RS: A Multi-indexing System based on HDFS for Remote Sensing Data Storage [21.033380514644616]
  A large volume of remote sensing (RS) data has been generated with the deployment of satellite technologies. The characteristics of RS data (e.g., enormous volume, large single-file size and demanding requirement of fault tolerance) make the Hadoop Distributed File System (HDFS) an ideal choice for RS data storage. To use RS data, one of the most important techniques is geospatial indexing. We propose a framework called Multi-IndeXing-RS (MIX-RS) that unifies the multi-indexing mechanism on top of the HDFS.
  arXiv  Detail & Related papers  (2022-08-05T05:11:12Z)
• FedHiSyn: A Hierarchical Synchronous Federated Learning Framework for Resource and Data Heterogeneity [56.82825745165945]
  Federated Learning (FL) enables training a global model without sharing the decentralized raw data stored on multiple devices to protect data privacy. We propose a hierarchical synchronous FL framework, i.e., FedHiSyn, to tackle the problems of straggler effects and outdated models. We evaluate the proposed framework based on MNIST, EMNIST, CIFAR10 and CIFAR100 datasets and diverse heterogeneous settings of devices.
  arXiv  Detail & Related papers  (2022-06-21T17:23:06Z)
• An Analysis of Distributed Systems Syllabi With a Focus on Performance-Related Topics [65.86247008403002]
  We analyze a dataset of 51 current ( 2019-2020) Distributed Systems syllabi from top Computer Science programs. We study the scale of the infrastructure mentioned in DS courses, from small client-server systems to cloud-scale, peer-to-peer, global-scale systems.
  arXiv  Detail & Related papers  (2021-03-02T16:49:09Z)
• Data Mining with Big Data in Intrusion Detection Systems: A Systematic Literature Review [68.15472610671748]
  Cloud computing has become a powerful and indispensable technology for complex, high performance and scalable computation. The rapid rate and volume of data creation has begun to pose significant challenges for data management and security. The design and deployment of intrusion detection systems (IDS) in the big data setting has, therefore, become a topic of importance.
  arXiv  Detail & Related papers  (2020-05-23T20:57:12Z)
• DataFed: Towards Reproducible Research via Federated Data Management [0.0]
  DataFed is a lightweight, distributed scientific data management system. It spans a federation of storage systems within a loosely-coupled network of scientific facilities.
  arXiv  Detail & Related papers  (2020-04-07T21:05:22Z)
• A Privacy-Preserving Distributed Architecture for Deep-Learning-as-a-Service [68.84245063902908]
  This paper introduces a novel distributed architecture for deep-learning-as-a-service. It is able to preserve the user sensitive data while providing Cloud-based machine and deep learning services.
  arXiv  Detail & Related papers  (2020-03-30T15:12:03Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.