Lessons from a Big-Bang Integration: Challenges in Edge Computing and Machine Learning

• URL: http://arxiv.org/abs/2507.17270v1
• Date: Wed, 23 Jul 2025 07:16:45 GMT
• Title: Lessons from a Big-Bang Integration: Challenges in Edge Computing and Machine Learning
• Abstract summary: The project faced critical setbacks due to a big-bang integration approach.<n>The study identifies technical and organisational barriers, including poor communication.<n>It also considers psychological factors such as a bias toward fully developed components over mockups.
• Score: 52.86213078016168
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: This experience report analyses a one year project focused on building a distributed real-time analytics system using edge computing and machine learning. The project faced critical setbacks due to a big-bang integration approach, where all components developed by multiple geographically dispersed partners were merged at the final stage. The integration effort resulted in only six minutes of system functionality, far below the expected 40 minutes. Through root cause analysis, the study identifies technical and organisational barriers, including poor communication, lack of early integration testing, and resistance to topdown planning. It also considers psychological factors such as a bias toward fully developed components over mockups. The paper advocates for early mock based deployment, robust communication infrastructures, and the adoption of topdown thinking to manage complexity and reduce risk in reactive, distributed projects. These findings underscore the limitations of traditional Agile methods in such contexts and propose simulation-driven engineering and structured integration cycles as key enablers for future success.

Related papers

• GenAI for Systems: Recurring Challenges and Design Principles from Software to Silicon [62.2138479061386]
  Generative AI is reshaping how computing systems are designed, optimized, and built, yet research remains fragmented across software, architecture, and chip design communities.<n>This paper takes a cross-stack perspective, examining how generative models are being applied from code generation and distributed runtimes through hardware design space exploration to RTL synthesis, physical layout, and verification.
  arXiv  Detail & Related papers  (2026-02-16T22:45:33Z)
• Advances and Frontiers of LLM-based Issue Resolution in Software Engineering: A Comprehensive Survey [59.3507264893654]
  Issue resolution is a complex Software Engineering task integral to real-world development.<n> benchmarks like SWE-bench revealed this task as profoundly difficult for large language models.<n>This paper presents a systematic survey of this emerging domain.
  arXiv  Detail & Related papers  (2026-01-15T18:55:03Z)
• A Comprehensive Survey on Benchmarks and Solutions in Software Engineering of LLM-Empowered Agentic System [56.40989626804489]
  This survey provides the first holistic analysis of Large Language Models-powered software engineering.<n>We review over 150 recent papers and propose a taxonomy along two key dimensions: (1) Solutions, categorized into prompt-based, fine-tuning-based, and agent-based paradigms, and (2) Benchmarks, including tasks such as code generation, translation, and repair.
  arXiv  Detail & Related papers  (2025-10-10T06:56:50Z)
• Requirements for Game-Based Learning Design Framework for Information System Integration in the Context of Post-Merger Integration [0.0]
  Post-merger integration states unique challenges for professionals responsible for information system integration.<n>There is a significant gap in training for information system integration in this context.<n>This paper explores how game-based learning design can address these limitations by transforming static method training into engaging learning experience.
  arXiv  Detail & Related papers  (2025-10-07T17:25:44Z)
• The Lazy Student's Dream: ChatGPT Passing an Engineering Course on Its Own [3.2334676138718677]
  This paper presents a comprehensive investigation into the capability of Large Language Models (LLMs) to successfully complete a control systems course.<n>We assess LLM performance using ChatGPT under a "minimal effort" protocol that simulates realistic student usage patterns.<n>Our analysis provides quantitative insights into AI's strengths and limitations in handling mathematical formulations, coding challenges, and theoretical concepts in control systems engineering.
  arXiv  Detail & Related papers  (2025-02-23T18:47:14Z)
• It Is Time To Steer: A Scalable Framework for Analysis-driven Attack Graph Generation [50.06412862964449]
  Attack Graph (AG) represents the best-suited solution to support cyber risk assessment for multi-step attacks on computer networks. Current solutions propose to address the generation problem from the algorithmic perspective and postulate the analysis only after the generation is complete. This paper rethinks the classic AG analysis through a novel workflow in which the analyst can query the system anytime.
  arXiv  Detail & Related papers  (2023-12-27T10:44:58Z)
• Decentralised and collaborative machine learning framework for IoT [0.32985979395737774]
  We propose a decentralised and collaborative machine learning framework specially oriented to resource-constrained devices. First, an incremental learning algorithm based on prototypes that was specifically implemented to work in low-performance computing elements. Second, two random-based protocols to exchange the local models among the computing elements in the network.
  arXiv  Detail & Related papers  (2023-12-19T14:25:41Z)
• GowFed -- A novel Federated Network Intrusion Detection System [0.15469452301122172]
  This work presents GowFed, a novel network threat detection system that combines the usage of Gower Dissimilarity matrices and Federated averaging. Different approaches of GowFed have been developed based on state-of the-art knowledge: (1) a vanilla version; and (2) a version instrumented with an attention mechanism. Overall, GowFed intends to be the first stepping stone towards the combined usage of Federated Learning and Gower Dissimilarity matrices to detect network threats in industrial-level networks.
  arXiv  Detail & Related papers  (2022-10-28T23:53:37Z)
• Distributed intelligence on the Edge-to-Cloud Continuum: A systematic literature review [62.997667081978825]
  This review aims at providing a comprehensive vision of the main state-of-the-art libraries and frameworks for machine learning and data analytics available today. The main simulation, emulation, deployment systems, and testbeds for experimental research on the Edge-to-Cloud Continuum available today are also surveyed.
  arXiv  Detail & Related papers  (2022-04-29T08:06:05Z)
• A review of Federated Learning in Intrusion Detection Systems for IoT [0.15469452301122172]
  Intrusion detection systems are evolving into intelligent systems that perform data analysis searching for anomalies in their environment. Deep learning technologies opened the door to build more complex and effective threat detection models. Current approaches rely on powerful centralized servers that receive data from all their parties. This paper focuses on the application of Federated Learning approaches in the field of Intrusion Detection.
  arXiv  Detail & Related papers  (2022-04-26T17:00:07Z)
• Contextual Model Aggregation for Fast and Robust Federated Learning in Edge Computing [88.76112371510999]
  Federated learning is a prime candidate for distributed machine learning at the network edge. Existing algorithms face issues with slow convergence and/or robustness of performance. We propose a contextual aggregation scheme that achieves the optimal context-dependent bound on loss reduction.
  arXiv  Detail & Related papers  (2022-03-23T21:42:31Z)
• End-to-End Constrained Optimization Learning: A Survey [69.22203885491534]
  It focuses on surveying the work on integrating solvers and optimization methods with machine learning architectures. These approaches hold the promise to develop new hybrid machine learning and optimization methods to predict fast, approximate, structural, solutions to problems and to enable logical inference.
  arXiv  Detail & Related papers  (2021-03-30T14:19:30Z)
• Performance Analysis of Deep Learning Workloads on a Composable System [0.08388591755871731]
  Composable infrastructure is defined as resources, such as compute, storage, accelerators and networking, that are shared in a pool. This paper details the design of an enterprise composable infrastructure that we have implemented and made available to our partners in the IBM Research AI Hardware Center.
  arXiv  Detail & Related papers  (2021-03-19T17:15:42Z)
• Investigating Bi-Level Optimization for Learning and Vision from a Unified Perspective: A Survey and Beyond [114.39616146985001]
  In machine learning and computer vision fields, despite the different motivations and mechanisms, a lot of complex problems contain a series of closely related subproblms. In this paper, we first uniformly express these complex learning and vision problems from the perspective of Bi-Level Optimization (BLO) Then we construct a value-function-based single-level reformulation and establish a unified algorithmic framework to understand and formulate mainstream gradient-based BLO methodologies.
  arXiv  Detail & Related papers  (2021-01-27T16:20:23Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.