Digital Asset Data Lakehouse. The concept based on a blockchain research center

• URL: http://arxiv.org/abs/2503.15968v1
• Date: Thu, 20 Mar 2025 09:12:39 GMT
• Title: Digital Asset Data Lakehouse. The concept based on a blockchain research center
• Authors: Raul Cristian Bag,
• Abstract summary: This paper introduces a novel software architecture designed to meet the demand for robust, scalable, and secure data management platforms.<n>We detail the architectural design, including its components and interactions, and discuss how it addresses common challenges in managing blockchain data and digital assets.<n>Our results indicate that the proposed architecture not only enhances the efficiency and scalability of distributed data management but also opens new avenues for innovation in the research area.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: In the rapidly evolving landscape of digital assets and blockchain technologies, the necessity for robust, scalable, and secure data management platforms has never been more critical. This paper introduces a novel software architecture designed to meet these demands by leveraging the inherent strengths of cloud-native technologies and modular micro-service based architectures, to facilitate efficient data management, storage and access, across different stakeholders. We detail the architectural design, including its components and interactions, and discuss how it addresses common challenges in managing blockchain data and digital assets, such as scalability, data siloing, and security vulnerabilities. We demonstrate the capabilities of the platform by employing it into multiple real-life scenarios, namely providing data in near real-time to scientists in help with their research. Our results indicate that the proposed architecture not only enhances the efficiency and scalability of distributed data management but also opens new avenues for innovation in the research reproducibility area. This work lays the groundwork for future research and development in machine learning operations systems, offering a scalable and secure framework for the burgeoning digital economy.

Related papers

• FinML-Chain: A Blockchain-Integrated Dataset for Enhanced Financial Machine Learning [2.0695662173473206]
  We present a framework for integrating high-frequency on-chain data with low-frequency off-chain data. This framework generates modular datasets for analyzing economic mechanisms such as the Transaction Fee Mechanism. We demonstrate the framework's ability to produce datasets that advance financial research and improve understanding of blockchain-driven systems.
  arXiv  Detail & Related papers  (2024-11-25T10:55:11Z)
• Swarm Learning: A Survey of Concepts, Applications, and Trends [3.55026004901472]
  Deep learning models have raised privacy and security concerns due to their reliance on large datasets on central servers.<n>Federated learning (FL) has introduced a novel approach to building a versatile, large-scale machine learning framework.<n>Swarm learning (SL) has been proposed in collaboration with Hewlett Packard Enterprise (HPE)<n>SL represents a decentralized machine learning framework that leverages blockchain technology for secure, scalable, and private data management.
  arXiv  Detail & Related papers  (2024-05-01T14:59:24Z)
• Multi-Tier Computing-Enabled Digital Twin in 6G Networks [50.236861239246835]
  In Industry 4.0, industries such as manufacturing, automotive, and healthcare are rapidly adopting DT-based development. The main challenges to date have been the high demands on communication and computing resources, as well as privacy and security concerns. To achieve low latency and high security services in the emerging DT, multi-tier computing has been proposed by combining edge/fog computing and cloud computing.
  arXiv  Detail & Related papers  (2023-12-28T13:02:53Z)
• A Blockchain Solution for Collaborative Machine Learning over IoT [0.31410859223862103]
  Federated learning (FL) and blockchain technologies have emerged as promising approaches to address these challenges. We present a novel IoT solution that combines the incremental learning vector quantization algorithm (XuILVQ) with blockchain technology. Our proposed architecture addresses the shortcomings of existing blockchain-based FL solutions by reducing computational and communication overheads while maintaining data privacy and security.
  arXiv  Detail & Related papers  (2023-11-23T18:06:05Z)
• Serving Deep Learning Model in Relational Databases [70.53282490832189]
  Serving deep learning (DL) models on relational data has become a critical requirement across diverse commercial and scientific domains. We highlight three pivotal paradigms: The state-of-the-art DL-centric architecture offloads DL computations to dedicated DL frameworks. The potential UDF-centric architecture encapsulates one or more tensor computations into User Defined Functions (UDFs) within the relational database management system (RDBMS)
  arXiv  Detail & Related papers  (2023-10-07T06:01:35Z)
• Computation-efficient Deep Learning for Computer Vision: A Survey [121.84121397440337]
  Deep learning models have reached or even exceeded human-level performance in a range of visual perception tasks. Deep learning models usually demand significant computational resources, leading to impractical power consumption, latency, or carbon emissions in real-world scenarios. New research focus is computationally efficient deep learning, which strives to achieve satisfactory performance while minimizing the computational cost during inference.
  arXiv  Detail & Related papers  (2023-08-27T03:55:28Z)
• APPFLChain: A Privacy Protection Distributed Artificial-Intelligence Architecture Based on Federated Learning and Consortium Blockchain [6.054775780656853]
  We propose a new system architecture called APPFLChain. It is an integrated architecture of a Hyperledger Fabric-based blockchain and a federated-learning paradigm. Our new system can maintain a high degree of security and privacy as users do not need to share sensitive personal information to the server.
  arXiv  Detail & Related papers  (2022-06-26T05:30:07Z)
• Neural Architecture Search for Dense Prediction Tasks in Computer Vision [74.9839082859151]
  Deep learning has led to a rising demand for neural network architecture engineering. neural architecture search (NAS) aims at automatically designing neural network architectures in a data-driven manner rather than manually. NAS has become applicable to a much wider range of problems in computer vision.
  arXiv  Detail & Related papers  (2022-02-15T08:06:50Z)
• YMIR: A Rapid Data-centric Development Platform for Vision Applications [82.67319997259622]
  This paper introduces an open source platform for rapid development of computer vision applications. The platform puts the efficient data development at the center of the machine learning development process.
  arXiv  Detail & Related papers  (2021-11-19T05:02:55Z)
• A Privacy-Preserving Distributed Architecture for Deep-Learning-as-a-Service [68.84245063902908]
  This paper introduces a novel distributed architecture for deep-learning-as-a-service. It is able to preserve the user sensitive data while providing Cloud-based machine and deep learning services.
  arXiv  Detail & Related papers  (2020-03-30T15:12:03Z)
• Knowledge Integration of Collaborative Product Design Using Cloud Computing Infrastructure [65.2157099438235]
  The main focus of this paper is the concept of ongoing research in providing the knowledge integration service for collaborative product design and development using cloud computing infrastructure. Proposed knowledge integration services support users by giving real-time access to knowledge resources.
  arXiv  Detail & Related papers  (2020-01-16T18:44:27Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.