Debiasing Architectural Decision-Making: An Experiment With Students and Practitioners

• URL: http://arxiv.org/abs/2502.04011v1
• Date: Thu, 06 Feb 2025 12:12:53 GMT
• Title: Debiasing Architectural Decision-Making: An Experiment With Students and Practitioners
• Authors: Klara Borowa, Rodrigo Rebouças de Almeida, Marion Wiese,
• Abstract summary: This study was to design and evaluate a debiasing workshop with individuals at various stages of their professional careers. We found that the workshop had a more substantial impact on practitioners. We assume that the practitioners' attachment to their systems may be the cause of their susceptibility to biases.
• Score: 2.9767565026354195
• License:
• Abstract: Cognitive biases are predictable, systematic errors in human reasoning. They influence decision-making in various areas, including architectural decision-making, where architects face many choices. For example, anchoring can cause architects to unconsciously prefer the first architectural solution that they came up with, without considering any solution alternatives. Prior research suggests that training individuals in debiasing techniques during a practical workshop can help reduce the impact of biases. The goal of this study was to design and evaluate a debiasing workshop with individuals at various stages of their professional careers. To test the workshop's effectiveness, we performed an experiment with 16 students and 20 practitioners, split into control and workshop group pairs. We recorded and analyzed their think-aloud discussions about improving the architectures of systems they collaborated on. The workshop improved the participants' argumentation when discussing architectural decisions and increased the use of debiasing techniques taught during the workshop. This led to the successful reduction of the researched biases' occurrences. In particular, anchoring and optimism bias occurrences decreased significantly. We also found that practitioners were more susceptible to cognitive biases than students, so the workshop had a more substantial impact on practitioners. We assume that the practitioners' attachment to their systems may be the cause of their susceptibility to biases. Finally, we identified factors that may reduce the effectiveness of the debiasing workshop. On that basis, we prepared a set of teaching suggestions for educators. Overall, we recommend using this workshop to educate both students and experienced practitioners about the typical harmful influences of cognitive bias on architectural decisions and how to avoid them.

Related papers

• Learning to Assist Humans without Inferring Rewards [65.28156318196397]
  We build upon prior work that studies assistance through the lens of empowerment. An assistive agent aims to maximize the influence of the human's actions. We prove that these representations estimate a similar notion of empowerment to that studied by prior work.
  arXiv  Detail & Related papers  (2024-11-04T21:31:04Z)
• Be Aware of the Neighborhood Effect: Modeling Selection Bias under Interference [50.95521705711802]
  Previous studies have focused on addressing selection bias to achieve unbiased learning of the prediction model. This paper formally formulates the neighborhood effect as an interference problem from the perspective of causal inference. We propose a novel ideal loss that can be used to deal with selection bias in the presence of neighborhood effect.
  arXiv  Detail & Related papers  (2024-04-30T15:20:41Z)
• Evaluating and Optimizing Educational Content with Large Language Model Judgments [52.33701672559594]
  We use Language Models (LMs) as educational experts to assess the impact of various instructions on learning outcomes. We introduce an instruction optimization approach in which one LM generates instructional materials using the judgments of another LM as a reward function. Human teachers' evaluations of these LM-generated worksheets show a significant alignment between the LM judgments and human teacher preferences.
  arXiv  Detail & Related papers  (2024-03-05T09:09:15Z)
• Cognitive Bias in Decision-Making with LLMs [19.87475562475802]
  Large language models (LLMs) offer significant potential as tools to support an expanding range of decision-making tasks. LLMs have been shown to inherit societal biases against protected groups, as well as be subject to bias functionally resembling cognitive bias. Our work introduces BiasBuster, a framework designed to uncover, evaluate, and mitigate cognitive bias in LLMs.
  arXiv  Detail & Related papers  (2024-02-25T02:35:56Z)
• On (Mis)perceptions of testing effectiveness: an empirical study [1.8026347864255505]
  This research aims to discover how well the perceptions of the defect detection effectiveness of different techniques match their real effectiveness in the absence of prior experience. In the original study, we conduct a controlled experiment with students applying two testing techniques and a code review technique. At the end of the experiment, they take a survey to find out which technique they perceive to be most effective. The results of the replicated study confirm the findings of the original study and suggest that participants' perceptions might be based not on their opinions about complexity or preferences for techniques but on how well they think that they have applied the techniques.
  arXiv  Detail & Related papers  (2024-02-11T14:50:01Z)
• The Influence of Cognitive Biases on Architectural Technical Debt [0.9208007322096533]
  The results show which classes of architectural technical debt originate from cognitive biases. We identify a set of debiasing techniques that can be used in order to prevent the negative influence of cognitive biases.
  arXiv  Detail & Related papers  (2023-09-25T14:37:38Z)
• What rationales drive architectural decisions? An empirical inquiry [0.7499722271664147]
  There is a rationale behind every architectural decision that motivates an architect to choose one architectural solution out of a set of options. This study aims to identify which categories of rationale most frequently impact architectural decisions and investigates why these are important to practitioners.
  arXiv  Detail & Related papers  (2023-09-25T14:18:51Z)
• Rehearsal: Simulating Conflict to Teach Conflict Resolution [54.32934135393982]
  Rehearsal is a system that allows users to rehearse conflicts with a believable simulated interlocutor. Users can utilize Rehearsal to practice handling a variety of predefined conflict scenarios. Rehearsal uses IRP to generate utterances grounded in conflict resolution theory.
  arXiv  Detail & Related papers  (2023-09-21T17:59:20Z)
• Overcoming the Mental Set Effect in Programming Problem Solving [5.625796693054094]
  The Einstellung effect is the tendency to approach problem-solving with a preconceived mindset. This effect can significantly impact creative thinking, as the development of patterns of thought can hinder the emergence of novel and creative ideas. The study contributes to the existing literature by providing insights into creativity support during problem-solving in software development.
  arXiv  Detail & Related papers  (2023-07-13T10:49:02Z)
• Optimising Equal Opportunity Fairness in Model Training [60.0947291284978]
  Existing debiasing methods, such as adversarial training and removing protected information from representations, have been shown to reduce bias. We propose two novel training objectives which directly optimise for the widely-used criterion of it equal opportunity, and show that they are effective in reducing bias while maintaining high performance over two classification tasks.
  arXiv  Detail & Related papers  (2022-05-05T01:57:58Z)
• Learning "What-if" Explanations for Sequential Decision-Making [92.8311073739295]
  Building interpretable parameterizations of real-world decision-making on the basis of demonstrated behavior is essential. We propose learning explanations of expert decisions by modeling their reward function in terms of preferences with respect to "what if" outcomes. We highlight the effectiveness of our batch, counterfactual inverse reinforcement learning approach in recovering accurate and interpretable descriptions of behavior.
  arXiv  Detail & Related papers  (2020-07-02T14:24:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.