Impact of Usability Mechanisms: A Family of Experiments on Efficiency, Effectiveness and User Satisfaction

• URL: http://arxiv.org/abs/2408.12736v1
• Date: Thu, 22 Aug 2024 21:23:18 GMT
• Title: Impact of Usability Mechanisms: A Family of Experiments on Efficiency, Effectiveness and User Satisfaction
• Authors: Juan M. Ferreira, Francy Rodríguez, Adrián Santos, Silvia T. Acuña, Natalia Juristo,
• Abstract summary: We use a family of three experiments to increase the precision and generalization of the results in the baseline experiment. We find that the Abort Operation and Preferences usability mechanisms appear to improve system usability a great deal with respect to efficiency, effectiveness and user satisfaction.
• Score: 0.5419296578793327
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Context: The usability software quality attribute aims to improve system user performance. In a previous study, we found evidence of the impact of a set of usability characteristics from the viewpoint of users in terms of efficiency, effectiveness and satisfaction. However, the impact level appears to depend on the usability feature and suggest priorities with respect to their implementation depending on how they promote user performance. Objectives: We use a family of three experiments to increase the precision and generalization of the results in the baseline experiment and provide findings on the impact on user performance of the Abort Operation, Progress Feedback and Preferences usability mechanisms. Method: We conduct two replications of the baseline experiment in academic settings. We analyse the data of 367 experimental subjects and apply aggregation (meta-analysis) procedures. Results: We find that the Abort Operation and Preferences usability mechanisms appear to improve system usability a great deal with respect to efficiency, effectiveness and user satisfaction. Conclusions: We find that the family of experiments further corroborates the results of the baseline experiment. Most of the results are statistically significant, and, because of the large number of experimental subjects, the evidence that we gathered in the replications is sufficient to outweigh other experiments.

Related papers

• Reduced-Rank Multi-objective Policy Learning and Optimization [57.978477569678844]
  In practice, causal researchers do not have a single outcome in mind a priori. In government-assisted social benefit programs, policymakers collect many outcomes to understand the multidimensional nature of poverty. We present a data-driven dimensionality-reduction methodology for multiple outcomes in the context of optimal policy learning.
  arXiv  Detail & Related papers  (2024-04-29T08:16:30Z)
• Effect Size Estimation for Duration Recommendation in Online Experiments: Leveraging Hierarchical Models and Objective Utility Approaches [13.504353263032359]
  The selection of the assumed effect size (AES) critically determines the duration of an experiment, and hence its accuracy and efficiency. Traditionally, experimenters determine AES based on domain knowledge, but this method becomes impractical for online experimentation services managing numerous experiments. We propose two solutions for data-driven AES selection in for online experimentation services.
  arXiv  Detail & Related papers  (2023-12-20T09:34:28Z)
• Adaptive Instrument Design for Indirect Experiments [48.815194906471405]
  Unlike RCTs, indirect experiments estimate treatment effects by leveragingconditional instrumental variables. In this paper we take the initial steps towards enhancing sample efficiency for indirect experiments by adaptively designing a data collection policy. Our main contribution is a practical computational procedure that utilizes influence functions to search for an optimal data collection policy.
  arXiv  Detail & Related papers  (2023-12-05T02:38:04Z)
• A Double Machine Learning Approach to Combining Experimental and Observational Data [59.29868677652324]
  We propose a double machine learning approach to combine experimental and observational studies. Our framework tests for violations of external validity and ignorability under milder assumptions.
  arXiv  Detail & Related papers  (2023-07-04T02:53:11Z)
• The Impact of Expertise in the Loop for Exploring Machine Rationality [35.26871426747907]
  We analyzed the impact of the levels of expertise on the outcome quality and corresponding subjective satisfaction. We found that novices can achieve an expert level of quality performance, but participants with higher expertise led to more optimization with more explicit preference.
  arXiv  Detail & Related papers  (2023-02-11T11:53:55Z)
• Fair Effect Attribution in Parallel Online Experiments [57.13281584606437]
  A/B tests serve the purpose of reliably identifying the effect of changes introduced in online services. It is common for online platforms to run a large number of simultaneous experiments by splitting incoming user traffic randomly. Despite a perfect randomization between different groups, simultaneous experiments can interact with each other and create a negative impact on average population outcomes.
  arXiv  Detail & Related papers  (2022-10-15T17:15:51Z)
• Understanding reinforcement learned crowds [9.358303424584902]
  Reinforcement Learning methods are used to animate virtual agents. It is not obvious what is their real impact, and how they affect the results. We analyze some of these arbitrary choices in terms of their impact on the learning performance.
  arXiv  Detail & Related papers  (2022-09-19T20:47:49Z)
• On Inductive Biases for Heterogeneous Treatment Effect Estimation [91.3755431537592]
  We investigate how to exploit structural similarities of an individual's potential outcomes (POs) under different treatments. We compare three end-to-end learning strategies to overcome this problem.
  arXiv  Detail & Related papers  (2021-06-07T16:30:46Z)
• Influence Functions in Deep Learning Are Fragile [52.31375893260445]
  influence functions approximate the effect of samples in test-time predictions. influence estimates are fairly accurate for shallow networks. Hessian regularization is important to get highquality influence estimates.
  arXiv  Detail & Related papers  (2020-06-25T18:25:59Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.