Related papers: Lessons from a Big-Bang Integration: Challenges in Edge Computing and Machine Learning

Lessons from a Big-Bang Integration: Challenges in Edge Computing and Machine Learning

URL: http://arxiv.org/abs/2507.17270v1
Date: Wed, 23 Jul 2025 07:16:45 GMT
Title: Lessons from a Big-Bang Integration: Challenges in Edge Computing and Machine Learning
Authors: Alessandro Aneggi, Andrea Janes,
Abstract summary: The project faced critical setbacks due to a big-bang integration approach.<n>The study identifies technical and organisational barriers, including poor communication.<n>It also considers psychological factors such as a bias toward fully developed components over mockups.
Score: 52.86213078016168
License: http://creativecommons.org/licenses/by-nc-nd/4.0/
Abstract: This experience report analyses a one year project focused on building a distributed real-time analytics system using edge computing and machine learning. The project faced critical setbacks due to a big-bang integration approach, where all components developed by multiple geographically dispersed partners were merged at the final stage. The integration effort resulted in only six minutes of system functionality, far below the expected 40 minutes. Through root cause analysis, the study identifies technical and organisational barriers, including poor communication, lack of early integration testing, and resistance to topdown planning. It also considers psychological factors such as a bias toward fully developed components over mockups. The paper advocates for early mock based deployment, robust communication infrastructures, and the adoption of topdown thinking to manage complexity and reduce risk in reactive, distributed projects. These findings underscore the limitations of traditional Agile methods in such contexts and propose simulation-driven engineering and structured integration cycles as key enablers for future success.

Related papers

The Lazy Student's Dream: ChatGPT Passing an Engineering Course on Its Own [3.2334676138718677]
This paper presents a comprehensive investigation into the capability of Large Language Models (LLMs) to successfully complete a control systems course.<n>We assess LLM performance using ChatGPT under a "minimal effort" protocol that simulates realistic student usage patterns.<n>Our analysis provides quantitative insights into AI's strengths and limitations in handling mathematical formulations, coding challenges, and theoretical concepts in control systems engineering.
arXiv Detail & Related papers (2025-02-23T18:47:14Z)
It Is Time To Steer: A Scalable Framework for Analysis-driven Attack Graph Generation [50.06412862964449]
Attack Graph (AG) represents the best-suited solution to support cyber risk assessment for multi-step attacks on computer networks. Current solutions propose to address the generation problem from the algorithmic perspective and postulate the analysis only after the generation is complete. This paper rethinks the classic AG analysis through a novel workflow in which the analyst can query the system anytime.
arXiv Detail & Related papers (2023-12-27T10:44:58Z)
Decentralised and collaborative machine learning framework for IoT [0.32985979395737774]
We propose a decentralised and collaborative machine learning framework specially oriented to resource-constrained devices. First, an incremental learning algorithm based on prototypes that was specifically implemented to work in low-performance computing elements. Second, two random-based protocols to exchange the local models among the computing elements in the network.
arXiv Detail & Related papers (2023-12-19T14:25:41Z)
GowFed -- A novel Federated Network Intrusion Detection System [0.15469452301122172]
This work presents GowFed, a novel network threat detection system that combines the usage of Gower Dissimilarity matrices and Federated averaging. Different approaches of GowFed have been developed based on state-of the-art knowledge: (1) a vanilla version; and (2) a version instrumented with an attention mechanism. Overall, GowFed intends to be the first stepping stone towards the combined usage of Federated Learning and Gower Dissimilarity matrices to detect network threats in industrial-level networks.
arXiv Detail & Related papers (2022-10-28T23:53:37Z)
Distributed intelligence on the Edge-to-Cloud Continuum: A systematic literature review [62.997667081978825]
This review aims at providing a comprehensive vision of the main state-of-the-art libraries and frameworks for machine learning and data analytics available today. The main simulation, emulation, deployment systems, and testbeds for experimental research on the Edge-to-Cloud Continuum available today are also surveyed.
arXiv Detail & Related papers (2022-04-29T08:06:05Z)
A review of Federated Learning in Intrusion Detection Systems for IoT [0.15469452301122172]
Intrusion detection systems are evolving into intelligent systems that perform data analysis searching for anomalies in their environment. Deep learning technologies opened the door to build more complex and effective threat detection models. Current approaches rely on powerful centralized servers that receive data from all their parties. This paper focuses on the application of Federated Learning approaches in the field of Intrusion Detection.
arXiv Detail & Related papers (2022-04-26T17:00:07Z)
Contextual Model Aggregation for Fast and Robust Federated Learning in Edge Computing [88.76112371510999]
Federated learning is a prime candidate for distributed machine learning at the network edge. Existing algorithms face issues with slow convergence and/or robustness of performance. We propose a contextual aggregation scheme that achieves the optimal context-dependent bound on loss reduction.
arXiv Detail & Related papers (2022-03-23T21:42:31Z)
End-to-End Constrained Optimization Learning: A Survey [69.22203885491534]
It focuses on surveying the work on integrating solvers and optimization methods with machine learning architectures. These approaches hold the promise to develop new hybrid machine learning and optimization methods to predict fast, approximate, structural, solutions to problems and to enable logical inference.
arXiv Detail & Related papers (2021-03-30T14:19:30Z)
Performance Analysis of Deep Learning Workloads on a Composable System [0.08388591755871731]
Composable infrastructure is defined as resources, such as compute, storage, accelerators and networking, that are shared in a pool. This paper details the design of an enterprise composable infrastructure that we have implemented and made available to our partners in the IBM Research AI Hardware Center.
arXiv Detail & Related papers (2021-03-19T17:15:42Z)
Investigating Bi-Level Optimization for Learning and Vision from a Unified Perspective: A Survey and Beyond [114.39616146985001]
In machine learning and computer vision fields, despite the different motivations and mechanisms, a lot of complex problems contain a series of closely related subproblms. In this paper, we first uniformly express these complex learning and vision problems from the perspective of Bi-Level Optimization (BLO) Then we construct a value-function-based single-level reformulation and establish a unified algorithmic framework to understand and formulate mainstream gradient-based BLO methodologies.
arXiv Detail & Related papers (2021-01-27T16:20:23Z)

This list is automatically generated from the titles and abstracts of the papers in this site.