The Impact of Expertise in the Loop for Exploring Machine Rationality

• URL: http://arxiv.org/abs/2302.05665v1
• Date: Sat, 11 Feb 2023 11:53:55 GMT
• Title: The Impact of Expertise in the Loop for Exploring Machine Rationality
• Authors: Changkun Ou, Sven Mayer, Andreas Butz
• Abstract summary: We analyzed the impact of the levels of expertise on the outcome quality and corresponding subjective satisfaction. We found that novices can achieve an expert level of quality performance, but participants with higher expertise led to more optimization with more explicit preference.
• Score: 35.26871426747907
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Human-in-the-loop optimization utilizes human expertise to guide machine optimizers iteratively and search for an optimal solution in a solution space. While prior empirical studies mainly investigated novices, we analyzed the impact of the levels of expertise on the outcome quality and corresponding subjective satisfaction. We conducted a study (N=60) in text, photo, and 3D mesh optimization contexts. We found that novices can achieve an expert level of quality performance, but participants with higher expertise led to more optimization iteration with more explicit preference while keeping satisfaction low. In contrast, novices were more easily satisfied and terminated faster. Therefore, we identified that experts seek more diverse outcomes while the machine reaches optimal results, and the observed behavior can be used as a performance indicator for human-in-the-loop system designers to improve underlying models. We inform future research to be cautious about the impact of user expertise when designing human-in-the-loop systems.

Related papers

• Discovering Preference Optimization Algorithms with and for Large Language Models [50.843710797024805]
  offline preference optimization is a key method for enhancing and controlling the quality of Large Language Model (LLM) outputs. We perform objective discovery to automatically discover new state-of-the-art preference optimization algorithms without (expert) human intervention. Experiments demonstrate the state-of-the-art performance of DiscoPOP, a novel algorithm that adaptively blends logistic and exponential losses.
  arXiv  Detail & Related papers  (2024-06-12T16:58:41Z)
• Human-Algorithm Collaborative Bayesian Optimization for Engineering Systems [0.0]
  We re-introduce the human back into the data-driven decision making loop by outlining an approach for collaborative Bayesian optimization. Our methodology exploits the hypothesis that humans are more efficient at making discrete choices rather than continuous ones. We demonstrate our approach across a number of applied and numerical case studies including bioprocess optimization and reactor geometry design.
  arXiv  Detail & Related papers  (2024-04-16T23:17:04Z)
• Enhanced Bayesian Optimization via Preferential Modeling of Abstract Properties [49.351577714596544]
  We propose a human-AI collaborative Bayesian framework to incorporate expert preferences about unmeasured abstract properties into surrogate modeling. We provide an efficient strategy that can also handle any incorrect/misleading expert bias in preferential judgments.
  arXiv  Detail & Related papers  (2024-02-27T09:23:13Z)
• Inverse Reinforcement Learning with Sub-optimal Experts [56.553106680769474]
  We study the theoretical properties of the class of reward functions that are compatible with a given set of experts. Our results show that the presence of multiple sub-optimal experts can significantly shrink the set of compatible rewards. We analyze a uniform sampling algorithm that results in being minimax optimal whenever the sub-optimal experts' performance level is sufficiently close to the one of the optimal agent.
  arXiv  Detail & Related papers  (2024-01-08T12:39:25Z)
• Experience-Based Evolutionary Algorithms for Expensive Optimization [8.466374531816427]
  We argue that hard optimization problems could be tackled efficiently by making better use of experiences gained in related problems. We propose an experience-based surrogate-assisted evolutionary algorithm (SAEA) framework to enhance the optimization efficiency of expensive problems.
  arXiv  Detail & Related papers  (2023-04-09T05:47:14Z)
• BO-Muse: A human expert and AI teaming framework for accelerated experimental design [58.61002520273518]
  Our algorithm lets the human expert take the lead in the experimental process. We show that our algorithm converges sub-linearly, at a rate faster than the AI or human alone.
  arXiv  Detail & Related papers  (2023-03-03T02:56:05Z)
• Incorporating Expert Prior Knowledge into Experimental Design via Posterior Sampling [58.56638141701966]
  Experimenters can often acquire the knowledge about the location of the global optimum. It is unknown how to incorporate the expert prior knowledge about the global optimum into Bayesian optimization. An efficient Bayesian optimization approach has been proposed via posterior sampling on the posterior distribution of the global optimum.
  arXiv  Detail & Related papers  (2020-02-26T01:57:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.