Empowering Data Mesh with Federated Learning

• URL: http://arxiv.org/abs/2403.17878v2
• Date: Wed, 27 Mar 2024 16:01:00 GMT
• Title: Empowering Data Mesh with Federated Learning
• Authors: Haoyuan Li, Salman Toor,
• Abstract summary: New paradigm, Data Mesh, treats domains as a first-class concern by distributing the data ownership from the central team to each data domain. Many multi-million dollar organizations like Paypal, Netflix, and Zalando have already transformed their data analysis pipelines based on this new architecture. We introduce a pioneering approach that incorporates Federated Learning into Data Mesh.
• Score: 5.087058648342379
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The evolution of data architecture has seen the rise of data lakes, aiming to solve the bottlenecks of data management and promote intelligent decision-making. However, this centralized architecture is limited by the proliferation of data sources and the growing demand for timely analysis and processing. A new data paradigm, Data Mesh, is proposed to overcome these challenges. Data Mesh treats domains as a first-class concern by distributing the data ownership from the central team to each data domain, while keeping the federated governance to monitor domains and their data products. Many multi-million dollar organizations like Paypal, Netflix, and Zalando have already transformed their data analysis pipelines based on this new architecture. In this decentralized architecture where data is locally preserved by each domain team, traditional centralized machine learning is incapable of conducting effective analysis across multiple domains, especially for security-sensitive organizations. To this end, we introduce a pioneering approach that incorporates Federated Learning into Data Mesh. To the best of our knowledge, this is the first open-source applied work that represents a critical advancement toward the integration of federated learning methods into the Data Mesh paradigm, underscoring the promising prospects for privacy-preserving and decentralized data analysis strategies within Data Mesh architecture.

Related papers

• An advanced data fabric architecture leveraging homomorphic encryption and federated learning [10.779491433438144]
  This paper introduces a secure approach for medical image analysis using federated learning and partially homomorphic encryption within a distributed data fabric architecture. The study demonstrates the method's effectiveness through a case study on pituitary tumor classification, achieving a significant level of accuracy.
  arXiv  Detail & Related papers  (2024-02-15T08:50:36Z)
• Lightweight Knowledge Representations for Automating Data Analysis [33.094930396228676]
  We take the first steps towards automating a key aspect of the data science pipeline: data analysis. We present an taxonomy of data analytic operations that scopes analytics across domains and data, as well as a method for codifying domain-specific knowledge that links this taxonomy to actual data. In this way, we produce information spaces over data that enable complex analyses and search over this data scopes and pave the way for fully automated data analysis.
  arXiv  Detail & Related papers  (2023-10-15T06:44:45Z)
• Tackling Computational Heterogeneity in FL: A Few Theoretical Insights [68.8204255655161]
  We introduce and analyse a novel aggregation framework that allows for formalizing and tackling computational heterogeneous data. Proposed aggregation algorithms are extensively analyzed from a theoretical, and an experimental prospective.
  arXiv  Detail & Related papers  (2023-07-12T16:28:21Z)
• Towards Avoiding the Data Mess: Industry Insights from Data Mesh Implementations [1.5029560229270191]
  Data mesh is a socio-technical, decentralized, distributed concept for enterprise data management. We conduct 15 semi-structured interviews with industry experts. Our findings synthesize insights from industry experts and provide researchers and professionals with preliminary guidelines for the successful adoption of data mesh.
  arXiv  Detail & Related papers  (2023-02-03T13:09:57Z)
• DataPerf: Benchmarks for Data-Centric AI Development [81.03754002516862]
  DataPerf is a community-led benchmark suite for evaluating ML datasets and data-centric algorithms. We provide an open, online platform with multiple rounds of challenges to support this iterative development. The benchmarks, online evaluation platform, and baseline implementations are open source.
  arXiv  Detail & Related papers  (2022-07-20T17:47:54Z)
• FedILC: Weighted Geometric Mean and Invariant Gradient Covariance for Federated Learning on Non-IID Data [69.0785021613868]
  Federated learning is a distributed machine learning approach which enables a shared server model to learn by aggregating the locally-computed parameter updates with the training data from spatially-distributed client silos. We propose the Federated Invariant Learning Consistency (FedILC) approach, which leverages the gradient covariance and the geometric mean of Hessians to capture both inter-silo and intra-silo consistencies. This is relevant to various fields such as medical healthcare, computer vision, and the Internet of Things (IoT)
  arXiv  Detail & Related papers  (2022-05-19T03:32:03Z)
• Towards Data-Free Domain Generalization [12.269045654957765]
  How can knowledge contained in models trained on different source data domains be merged into a single model that generalizes well to unseen target domains? Prior domain generalization methods typically rely on using source domain data, making them unsuitable for private decentralized data. We propose DEKAN, an approach that extracts and fuses domain-specific knowledge from the available teacher models into a student model robust to domain shift.
  arXiv  Detail & Related papers  (2021-10-09T11:44:05Z)
• A communication efficient distributed learning framework for smart environments [0.4898659895355355]
  This paper proposes a distributed learning framework to move data analytics closer to where data is generated. Using distributed machine learning techniques, it is possible to drastically reduce the network overhead, while obtaining performance comparable to the cloud solution. The analysis also shows when each distributed learning approach is preferable, based on the specific distribution of the data on the nodes.
  arXiv  Detail & Related papers  (2021-09-27T13:44:34Z)
• Unsupervised Domain Adaptive Learning via Synthetic Data for Person Re-identification [101.1886788396803]
  Person re-identification (re-ID) has gained more and more attention due to its widespread applications in video surveillance. Unfortunately, the mainstream deep learning methods still need a large quantity of labeled data to train models. In this paper, we develop a data collector to automatically generate synthetic re-ID samples in a computer game, and construct a data labeler to simultaneously annotate them.
  arXiv  Detail & Related papers  (2021-09-12T15:51:41Z)
• Rethinking Architecture Design for Tackling Data Heterogeneity in Federated Learning [53.73083199055093]
  We show that attention-based architectures (e.g., Transformers) are fairly robust to distribution shifts. Our experiments show that replacing convolutional networks with Transformers can greatly reduce catastrophic forgetting of previous devices.
  arXiv  Detail & Related papers  (2021-06-10T21:04:18Z)
• Inferring Latent Domains for Unsupervised Deep Domain Adaptation [54.963823285456925]
  Unsupervised Domain Adaptation (UDA) refers to the problem of learning a model in a target domain where labeled data are not available. This paper introduces a novel deep architecture which addresses the problem of UDA by automatically discovering latent domains in visual datasets. We evaluate our approach on publicly available benchmarks, showing that it outperforms state-of-the-art domain adaptation methods.
  arXiv  Detail & Related papers  (2021-03-25T14:33:33Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.