Learning From Lessons Learned: Preliminary Findings From a Study of Learning From Failure

• URL: http://arxiv.org/abs/2402.09538v1
• Date: Wed, 14 Feb 2024 19:29:04 GMT
• Title: Learning From Lessons Learned: Preliminary Findings From a Study of Learning From Failure
• Authors: Jonathan Sillito, Matt Pope
• Abstract summary: Organizations analyze and learn from system failures. Co-evolve both the technical and human parts of their systems based on what they learn. Despite established processes and tool support, it is not straightforward to take what was learned from a failure and successfully improve the reliability of the socio-technical system.
• Score: 3.045851438458641
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Due to various sources of uncertainty, emergent behavior, and ongoing changes, the reliability of many socio-technical systems depends on an iterative and collaborative process in which organizations (1) analyze and learn from system failures, and then (2) co-evolve both the technical and human parts of their systems based on what they learn. Many organizations have defined processes for learning from failure, often involving postmortem analyses conducted after any system failures that are judged to be sufficiently severe. Despite established processes and tool support, our preliminary research, and professional experience, suggest that it is not straightforward to take what was learned from a failure and successfully improve the reliability of the socio-technical system. To better understand this collaborative process and the associated challenges, we are conducting a study of how teams learn from failure. We are gathering incident reports from multiple organizations and conducting interviews with engineers and managers with relevant experience. Our analytic interest is in what is learned by teams as they reflect on failures, the learning processes involved, and how they use what is learned. Our data collection and analysis are not yet complete, but we have so far analyzed 13 incident reports and seven interviews. In this short paper we (1) present our preliminary findings, and (2) outline our broader research plans.

Related papers

• What Could Possibly Go Wrong: Undesirable Patterns in Collective Development [4.2330023661329355]
  Various studies have attempted to capture the social dynamics within software engineering. Certain teamwork issues remain unstudied. This paper introduces the concept of undesirable patterns in collective development.
  arXiv  Detail & Related papers  (2024-09-02T15:13:18Z)
• Leveraging Skill-to-Skill Supervision for Knowledge Tracing [13.753990664747265]
  Knowledge tracing plays a pivotal role in intelligent tutoring systems. Recent advances in knowledge tracing models have enabled better exploitation of problem solving history. Knowledge tracing algorithms that incorporate knowledge directly are important to settings with limited data or cold starts.
  arXiv  Detail & Related papers  (2023-06-12T03:23:22Z)
• A Domain-Agnostic Approach for Characterization of Lifelong Learning Systems [128.63953314853327]
  "Lifelong Learning" systems are capable of 1) Continuous Learning, 2) Transfer and Adaptation, and 3) Scalability. We show that this suite of metrics can inform the development of varied and complex Lifelong Learning systems.
  arXiv  Detail & Related papers  (2023-01-18T21:58:54Z)
• When Do Curricula Work in Federated Learning? [56.88941905240137]
  We find that curriculum learning largely alleviates non-IIDness. The more disparate the data distributions across clients the more they benefit from learning. We propose a novel client selection technique that benefits from the real-world disparity in the clients.
  arXiv  Detail & Related papers  (2022-12-24T11:02:35Z)
• Critical Learning Periods for Multisensory Integration in Deep Networks [112.40005682521638]
  We show that the ability of a neural network to integrate information from diverse sources hinges critically on being exposed to properly correlated signals during the early phases of training. We show that critical periods arise from the complex and unstable early transient dynamics, which are decisive of final performance of the trained system and their learned representations.
  arXiv  Detail & Related papers  (2022-10-06T23:50:38Z)
• Anti-Retroactive Interference for Lifelong Learning [65.50683752919089]
  We design a paradigm for lifelong learning based on meta-learning and associative mechanism of the brain. It tackles the problem from two aspects: extracting knowledge and memorizing knowledge. It is theoretically analyzed that the proposed learning paradigm can make the models of different tasks converge to the same optimum.
  arXiv  Detail & Related papers  (2022-08-27T09:27:36Z)
• Learning Physical Concepts in Cyber-Physical Systems: A Case Study [72.74318982275052]
  We provide an overview of the current state of research regarding methods for learning physical concepts in time series data. We also analyze the most important methods from the current state of the art using the example of a three-tank system.
  arXiv  Detail & Related papers  (2021-11-28T14:24:52Z)
• Findings from Experiments of On-line Joint Reinforcement Learning of Semantic Parser and Dialogue Manager with real Users [3.9686445409447617]
  On-line learning is pursued in this paper as a convenient way to alleviate these difficulties. A new challenge is to control the cost of the on-line learning borne by the user.
  arXiv  Detail & Related papers  (2021-10-25T18:51:41Z)
• Discovering and Validating AI Errors With Crowdsourced Failure Reports [10.4818618376202]
  We introduce crowdsourced failure reports, end-user descriptions of how or why a model failed, and show how developers can use them to detect AI errors. We also design and implement Deblinder, a visual analytics system for synthesizing failure reports. In semi-structured interviews and think-aloud studies with 10 AI practitioners, we explore the affordances of the Deblinder system and the applicability of failure reports in real-world settings.
  arXiv  Detail & Related papers  (2021-09-23T23:26:59Z)
• Curriculum Learning for Reinforcement Learning Domains: A Framework and Survey [53.73359052511171]
  Reinforcement learning (RL) is a popular paradigm for addressing sequential decision tasks in which the agent has only limited environmental feedback. We present a framework for curriculum learning (CL) in RL, and use it to survey and classify existing CL methods in terms of their assumptions, capabilities, and goals.
  arXiv  Detail & Related papers  (2020-03-10T20:41:24Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.