Debiasing Architectural Decision-Making: An Experiment With Students and Practitioners

• URL: http://arxiv.org/abs/2502.04011v1
• Date: Thu, 06 Feb 2025 12:12:53 GMT
• Title: Debiasing Architectural Decision-Making: An Experiment With Students and Practitioners
• Authors: Klara Borowa, Rodrigo Rebouças de Almeida, Marion Wiese,
• Abstract summary: This study was to design and evaluate a debiasing workshop with individuals at various stages of their professional careers.<n>We found that the workshop had a more substantial impact on practitioners.<n>We assume that the practitioners' attachment to their systems may be the cause of their susceptibility to biases.
• Score: 2.9767565026354195
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Cognitive biases are predictable, systematic errors in human reasoning. They influence decision-making in various areas, including architectural decision-making, where architects face many choices. For example, anchoring can cause architects to unconsciously prefer the first architectural solution that they came up with, without considering any solution alternatives. Prior research suggests that training individuals in debiasing techniques during a practical workshop can help reduce the impact of biases. The goal of this study was to design and evaluate a debiasing workshop with individuals at various stages of their professional careers. To test the workshop's effectiveness, we performed an experiment with 16 students and 20 practitioners, split into control and workshop group pairs. We recorded and analyzed their think-aloud discussions about improving the architectures of systems they collaborated on. The workshop improved the participants' argumentation when discussing architectural decisions and increased the use of debiasing techniques taught during the workshop. This led to the successful reduction of the researched biases' occurrences. In particular, anchoring and optimism bias occurrences decreased significantly. We also found that practitioners were more susceptible to cognitive biases than students, so the workshop had a more substantial impact on practitioners. We assume that the practitioners' attachment to their systems may be the cause of their susceptibility to biases. Finally, we identified factors that may reduce the effectiveness of the debiasing workshop. On that basis, we prepared a set of teaching suggestions for educators. Overall, we recommend using this workshop to educate both students and experienced practitioners about the typical harmful influences of cognitive bias on architectural decisions and how to avoid them.

Related papers

• Further Evidence on a Controversial Topic about Human-Based Experiments: Professionals vs. Students [3.358019319437577]
  We compare 62 students and 42 software professionals on a bug-fixing task on the same Java program.<n>Considering the differences between the two groups of participants, the gathered data show that the students outperformed the professionals in fixing bugs.
  arXiv  Detail & Related papers  (2025-06-13T09:05:36Z)
• Improving Question Embeddings with Cognitive Representation Optimization for Knowledge Tracing [77.14348157016518]
  Research on KT modeling focuses on predicting future student performance based on existing, unupdated records of student learning interactions.<n>We propose a knowledge-tracking cognitive representation optimization (CRO-KT) model that uses dynamic programming algorithms to optimize the structure of cognitive representation.
  arXiv  Detail & Related papers  (2025-04-05T09:32:03Z)
• Learning to Assist Humans without Inferring Rewards [65.28156318196397]
  We build upon prior work that studies assistance through the lens of empowerment.<n>An assistive agent aims to maximize the influence of the human's actions.<n>We prove that these representations estimate a similar notion of empowerment to that studied by prior work.
  arXiv  Detail & Related papers  (2024-11-04T21:31:04Z)
• Biased AI can Influence Political Decision-Making [64.9461133083473]
  This paper presents two experiments investigating the effects of partisan bias in large language models (LLMs) on political opinions and decision-making.<n>We found that participants exposed to partisan biased models were significantly more likely to adopt opinions and make decisions which matched the LLM's bias.
  arXiv  Detail & Related papers  (2024-10-08T22:56:00Z)
• Be Aware of the Neighborhood Effect: Modeling Selection Bias under Interference [50.95521705711802]
  Previous studies have focused on addressing selection bias to achieve unbiased learning of the prediction model. This paper formally formulates the neighborhood effect as an interference problem from the perspective of causal inference. We propose a novel ideal loss that can be used to deal with selection bias in the presence of neighborhood effect.
  arXiv  Detail & Related papers  (2024-04-30T15:20:41Z)
• Evaluating and Optimizing Educational Content with Large Language Model Judgments [52.33701672559594]
  We use Language Models (LMs) as educational experts to assess the impact of various instructions on learning outcomes. We introduce an instruction optimization approach in which one LM generates instructional materials using the judgments of another LM as a reward function. Human teachers' evaluations of these LM-generated worksheets show a significant alignment between the LM judgments and human teacher preferences.
  arXiv  Detail & Related papers  (2024-03-05T09:09:15Z)
• Cognitive Bias in Decision-Making with LLMs [19.87475562475802]
  Large language models (LLMs) offer significant potential as tools to support an expanding range of decision-making tasks. LLMs have been shown to inherit societal biases against protected groups, as well as be subject to bias functionally resembling cognitive bias. Our work introduces BiasBuster, a framework designed to uncover, evaluate, and mitigate cognitive bias in LLMs.
  arXiv  Detail & Related papers  (2024-02-25T02:35:56Z)
• On (Mis)perceptions of testing effectiveness: an empirical study [1.8026347864255505]
  This research aims to discover how well the perceptions of the defect detection effectiveness of different techniques match their real effectiveness in the absence of prior experience. In the original study, we conduct a controlled experiment with students applying two testing techniques and a code review technique. At the end of the experiment, they take a survey to find out which technique they perceive to be most effective. The results of the replicated study confirm the findings of the original study and suggest that participants' perceptions might be based not on their opinions about complexity or preferences for techniques but on how well they think that they have applied the techniques.
  arXiv  Detail & Related papers  (2024-02-11T14:50:01Z)
• The Influence of Cognitive Biases on Architectural Technical Debt [0.9208007322096533]
  The results show which classes of architectural technical debt originate from cognitive biases. We identify a set of debiasing techniques that can be used in order to prevent the negative influence of cognitive biases.
  arXiv  Detail & Related papers  (2023-09-25T14:37:38Z)
• What rationales drive architectural decisions? An empirical inquiry [0.7499722271664147]
  There is a rationale behind every architectural decision that motivates an architect to choose one architectural solution out of a set of options. This study aims to identify which categories of rationale most frequently impact architectural decisions and investigates why these are important to practitioners.
  arXiv  Detail & Related papers  (2023-09-25T14:18:51Z)
• Rehearsal: Simulating Conflict to Teach Conflict Resolution [54.32934135393982]
  Rehearsal is a system that allows users to rehearse conflicts with a believable simulated interlocutor. Users can utilize Rehearsal to practice handling a variety of predefined conflict scenarios. Rehearsal uses IRP to generate utterances grounded in conflict resolution theory.
  arXiv  Detail & Related papers  (2023-09-21T17:59:20Z)
• Overcoming the Mental Set Effect in Programming Problem Solving [5.625796693054094]
  The Einstellung effect is the tendency to approach problem-solving with a preconceived mindset. This effect can significantly impact creative thinking, as the development of patterns of thought can hinder the emergence of novel and creative ideas. The study contributes to the existing literature by providing insights into creativity support during problem-solving in software development.
  arXiv  Detail & Related papers  (2023-07-13T10:49:02Z)
• On student-teacher deviations in distillation: does it pay to disobey? [54.908344098305804]
  Knowledge distillation has been widely used to improve the test accuracy of a "student" network. Despite being trained to fit the teacher's probabilities, the student may not only significantly deviate from the teacher probabilities, but may also outdo the teacher in performance.
  arXiv  Detail & Related papers  (2023-01-30T14:25:02Z)
• Optimising Equal Opportunity Fairness in Model Training [60.0947291284978]
  Existing debiasing methods, such as adversarial training and removing protected information from representations, have been shown to reduce bias. We propose two novel training objectives which directly optimise for the widely-used criterion of it equal opportunity, and show that they are effective in reducing bias while maintaining high performance over two classification tasks.
  arXiv  Detail & Related papers  (2022-05-05T01:57:58Z)
• Learning "What-if" Explanations for Sequential Decision-Making [92.8311073739295]
  Building interpretable parameterizations of real-world decision-making on the basis of demonstrated behavior is essential. We propose learning explanations of expert decisions by modeling their reward function in terms of preferences with respect to "what if" outcomes. We highlight the effectiveness of our batch, counterfactual inverse reinforcement learning approach in recovering accurate and interpretable descriptions of behavior.
  arXiv  Detail & Related papers  (2020-07-02T14:24:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.