Black Magic in Deep Learning: How Human Skill Impacts Network Training

• URL: http://arxiv.org/abs/2008.05981v1
• Date: Thu, 13 Aug 2020 15:56:14 GMT
• Title: Black Magic in Deep Learning: How Human Skill Impacts Network Training
• Authors: Kanav Anand, Ziqi Wang, Marco Loog, Jan van Gemert
• Abstract summary: We present an initial study based on 31 participants with different levels of experience. The results show a strong positive correlation between the participant's experience and the final performance.
• Score: 24.802914836352738
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: How does a user's prior experience with deep learning impact accuracy? We present an initial study based on 31 participants with different levels of experience. Their task is to perform hyperparameter optimization for a given deep learning architecture. The results show a strong positive correlation between the participant's experience and the final performance. They additionally indicate that an experienced participant finds better solutions using fewer resources on average. The data suggests furthermore that participants with no prior experience follow random strategies in their pursuit of optimal hyperparameters. Our study investigates the subjective human factor in comparisons of state of the art results and scientific reproducibility in deep learning.

Related papers

• "Give Me an Example Like This": Episodic Active Reinforcement Learning from Demonstrations [3.637365301757111]
  Methods like Reinforcement Learning from Expert Demonstrations (RLED) introduce external expert demonstrations to facilitate agent exploration during the learning process. How to select the best set of human demonstrations that is most beneficial for learning becomes a major concern. This paper presents EARLY, an algorithm that enables a learning agent to generate optimized queries of expert demonstrations in a trajectory-based feature space.
  arXiv  Detail & Related papers  (2024-06-05T08:52:21Z)
• On (Mis)perceptions of testing effectiveness: an empirical study [1.8026347864255505]
  This research aims to discover how well the perceptions of the defect detection effectiveness of different techniques match their real effectiveness in the absence of prior experience. In the original study, we conduct a controlled experiment with students applying two testing techniques and a code review technique. At the end of the experiment, they take a survey to find out which technique they perceive to be most effective. The results of the replicated study confirm the findings of the original study and suggest that participants' perceptions might be based not on their opinions about complexity or preferences for techniques but on how well they think that they have applied the techniques.
  arXiv  Detail & Related papers  (2024-02-11T14:50:01Z)
• ASPEST: Bridging the Gap Between Active Learning and Selective Prediction [56.001808843574395]
  Selective prediction aims to learn a reliable model that abstains from making predictions when uncertain. Active learning aims to lower the overall labeling effort, and hence human dependence, by querying the most informative examples. In this work, we introduce a new learning paradigm, active selective prediction, which aims to query more informative samples from the shifted target domain.
  arXiv  Detail & Related papers  (2023-04-07T23:51:07Z)
• The Impact of Expertise in the Loop for Exploring Machine Rationality [35.26871426747907]
  We analyzed the impact of the levels of expertise on the outcome quality and corresponding subjective satisfaction. We found that novices can achieve an expert level of quality performance, but participants with higher expertise led to more optimization with more explicit preference.
  arXiv  Detail & Related papers  (2023-02-11T11:53:55Z)
• Personalizing Intervened Network for Long-tailed Sequential User Behavior Modeling [66.02953670238647]
  Tail users suffer from significantly lower-quality recommendation than the head users after joint training. A model trained on tail users separately still achieve inferior results due to limited data. We propose a novel approach that significantly improves the recommendation performance of the tail users.
  arXiv  Detail & Related papers  (2022-08-19T02:50:19Z)
• Basis for Intentions: Efficient Inverse Reinforcement Learning using Past Experience [89.30876995059168]
  inverse reinforcement learning (IRL) -- inferring the reward function of an agent from observing its behavior. This paper addresses the problem of IRL -- inferring the reward function of an agent from observing its behavior.
  arXiv  Detail & Related papers  (2022-08-09T17:29:49Z)
• Improving Human Sequential Decision-Making with Reinforcement Learning [29.334511328067777]
  We design a novel machine learning algorithm that is capable of extracting "best practices" from trace data. Our algorithm selects the tip that best bridges the gap between the actions taken by human workers and those taken by the optimal policy. Experiments show that the tips generated by our algorithm can significantly improve human performance.
  arXiv  Detail & Related papers  (2021-08-19T02:57:58Z)
• Bayesian Active Learning for Wearable Stress and Affect Detection [0.7106986689736827]
  Stress detection using on-device deep learning algorithms has been on the rise owing to advancements in pervasive computing. In this paper, we propose a framework with capabilities to represent model uncertainties through approximations in Bayesian Neural Networks. Our proposed framework achieves a considerable efficiency boost during inference, with a substantially low number of acquired pool points.
  arXiv  Detail & Related papers  (2020-12-04T16:19:37Z)
• Revisiting Fundamentals of Experience Replay [91.24213515992595]
  We present a systematic and extensive analysis of experience replay in Q-learning methods. We focus on two fundamental properties: the replay capacity and the ratio of learning updates to experience collected.
  arXiv  Detail & Related papers  (2020-07-13T21:22:17Z)
• Omni-supervised Facial Expression Recognition via Distilled Data [120.11782405714234]
  We propose omni-supervised learning to exploit reliable samples in a large amount of unlabeled data for network training. We experimentally verify that the new dataset can significantly improve the ability of the learned FER model. To tackle this, we propose to apply a dataset distillation strategy to compress the created dataset into several informative class-wise images.
  arXiv  Detail & Related papers  (2020-05-18T09:36:51Z)
• Fine-Tuning Pretrained Language Models: Weight Initializations, Data Orders, and Early Stopping [62.78338049381917]
  Fine-tuning pretrained contextual word embedding models to supervised downstream tasks has become commonplace in natural language processing. We experiment with four datasets from the GLUE benchmark, fine-tuning BERT hundreds of times on each while varying only the random seeds. We find substantial performance increases compared to previously reported results, and we quantify how the performance of the best-found model varies as a function of the number of fine-tuning trials.
  arXiv  Detail & Related papers  (2020-02-15T02:40:10Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.