BigBird: Big Data Storage and Analytics at Scale in Hybrid Cloud

• URL: http://arxiv.org/abs/2203.11472v1
• Date: Tue, 22 Mar 2022 05:42:46 GMT
• Title: BigBird: Big Data Storage and Analytics at Scale in Hybrid Cloud
• Authors: Saurabh Deochake, Vrushali Channapattan, Gary Steelman
• Abstract summary: This paper showcases our approach in designing a scalable big data storage and analytics management framework using BigQuery in Google Cloud Platform. Although the paper discusses the framework implementation in Google Cloud Platform, it can easily be applied to all major cloud providers.
• Score: 0.0
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Implementing big data storage at scale is a complex and arduous task that requires an advanced infrastructure. With the rise of public cloud computing, various big data management services can be readily leveraged. As a critical part of Twitter's "Project Partly Cloudy", the cold storage data and analytics systems are being moved to the public cloud. This paper showcases our approach in designing a scalable big data storage and analytics management framework using BigQuery in Google Cloud Platform while ensuring security, privacy, and data protection. The paper also discusses the limitations on the public cloud resources and how they can be effectively overcome when designing a big data storage and analytics solution at scale. Although the paper discusses the framework implementation in Google Cloud Platform, it can easily be applied to all major cloud providers.

Related papers

• PVContext: Hybrid Context Model for Point Cloud Compression [61.24130634750288]
  We propose PVContext, a hybrid context model for effective octree-based point cloud compression. PVContext comprises two components with distinct modalities: the Voxel Context, which accurately represents local geometric information using voxels, and the Point Context, which efficiently preserves global shape information from point clouds.
  arXiv  Detail & Related papers  (2024-09-19T12:47:35Z)
• Mitigating Data Sharing in Public Cloud using Blockchain [0.0]
  We propose a secure data ecosystem in the cloud with the key aspects being Data Rights, Data Sharing, and Data Validation. This will ensure that existing public cloud-based systems can easily deploy blockchain enhancing trustworthiness and non-repudiation of cloud data.
  arXiv  Detail & Related papers  (2024-04-21T13:12:44Z)
• Protecting Sensitive Tabular Data in Hybrid Clouds [0.0]
  Regulated industries, such as Healthcare and Finance, are starting to move parts of their data and workloads to the public cloud. We address the security and performance challenges of big data analytics using a hybrid cloud in a real-life use case from a hospital.
  arXiv  Detail & Related papers  (2023-12-03T11:20:24Z)
• Exploring Security Practices in Infrastructure as Code: An Empirical Study [54.669404064111795]
  Cloud computing has become popular thanks to the widespread use of Infrastructure as Code (IaC) tools. scripting process does not automatically prevent practitioners from introducing misconfigurations, vulnerabilities, or privacy risks. Ensuring security relies on practitioners understanding and the adoption of explicit policies, guidelines, or best practices.
  arXiv  Detail & Related papers  (2023-08-07T23:43:32Z)
• Towards Confidential Computing: A Secure Cloud Architecture for Big Data Analytics and AI [0.0]
  Cloud computing has become a viable solution for big data analytics and artificial intelligence. Data security in certain fields such as biomedical research remains a major concern when moving to cloud.
  arXiv  Detail & Related papers  (2023-05-28T16:08:44Z)
• Remote Data Auditing and How it May Affect the Chain of Custody in a Cloud Environment [0.0]
  More and more organizations are relying on outsourcing their data to cloud-based environments. Law enforcement agencies from the national level down to large city police departments are also using the cloud environment to store data. This data solution presents in own set of problems in that the outsourced data can become untrustworthy due to the lack of control of the data owners.
  arXiv  Detail & Related papers  (2022-08-26T16:10:34Z)
• Measuring the Carbon Intensity of AI in Cloud Instances [91.28501520271972]
  We provide a framework for measuring software carbon intensity, and propose to measure operational carbon emissions. We evaluate a suite of approaches for reducing emissions on the Microsoft Azure cloud compute platform.
  arXiv  Detail & Related papers  (2022-06-10T17:04:04Z)
• Privacy-Preserving Cloud Computing: Ecosystem, Life Cycle, Layered Architecture and Future Roadmap [0.0]
  This survey paper on privacy-preserving cloud computing can help pave the way for future research in related areas. This paper helps to identify existing trends by establishing a layered architecture along with a life cycle and an ecosystem for privacy-preserving cloud systems.
  arXiv  Detail & Related papers  (2022-04-23T18:47:26Z)
• Unsupervised Point Cloud Representation Learning with Deep Neural Networks: A Survey [104.71816962689296]
  Unsupervised point cloud representation learning has attracted increasing attention due to the constraint in large-scale point cloud labelling. This paper provides a comprehensive review of unsupervised point cloud representation learning using deep neural networks.
  arXiv  Detail & Related papers  (2022-02-28T07:46:05Z)
• Auto-Split: A General Framework of Collaborative Edge-Cloud AI [49.750972428032355]
  This paper describes the techniques and engineering practice behind Auto-Split, an edge-cloud collaborative prototype of Huawei Cloud. To the best of our knowledge, there is no existing industry product that provides the capability of Deep Neural Network (DNN) splitting.
  arXiv  Detail & Related papers  (2021-08-30T08:03:29Z)
• A Privacy-Preserving Distributed Architecture for Deep-Learning-as-a-Service [68.84245063902908]
  This paper introduces a novel distributed architecture for deep-learning-as-a-service. It is able to preserve the user sensitive data while providing Cloud-based machine and deep learning services.
  arXiv  Detail & Related papers  (2020-03-30T15:12:03Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.