Robustness of Algorithms for Causal Structure Learning to Hyperparameter Choice

• URL: http://arxiv.org/abs/2310.18212v2
• Date: Tue, 20 Feb 2024 11:09:19 GMT
• Title: Robustness of Algorithms for Causal Structure Learning to Hyperparameter Choice
• Authors: Damian Machlanski, Spyridon Samothrakis, Paul Clarke
• Abstract summary: Hyper parameter tuning can make the difference between state-of-the-art and poor prediction performance for any algorithm. We investigate the influence of hyper parameter selection on causal structure learning tasks.
• Score: 2.3020018305241337
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Hyperparameters play a critical role in machine learning. Hyperparameter tuning can make the difference between state-of-the-art and poor prediction performance for any algorithm, but it is particularly challenging for structure learning due to its unsupervised nature. As a result, hyperparameter tuning is often neglected in favour of using the default values provided by a particular implementation of an algorithm. While there have been numerous studies on performance evaluation of causal discovery algorithms, how hyperparameters affect individual algorithms, as well as the choice of the best algorithm for a specific problem, has not been studied in depth before. This work addresses this gap by investigating the influence of hyperparameters on causal structure learning tasks. Specifically, we perform an empirical evaluation of hyperparameter selection for some seminal learning algorithms on datasets of varying levels of complexity. We find that, while the choice of algorithm remains crucial to obtaining state-of-the-art performance, hyperparameter selection in ensemble settings strongly influences the choice of algorithm, in that a poor choice of hyperparameters can lead to analysts using algorithms which do not give state-of-the-art performance for their data.

Related papers

• Tuning structure learning algorithms with out-of-sample and resampling strategies [6.85316573653194]
  Out-of-sample tuning for structure learning employs out-of-sample and resampling strategies to estimate the optimal hyperparameter configuration for structure learning. We show that employing OTSL leads to improvements in graphical accuracy compared to the state-of-the-art.
  arXiv  Detail & Related papers  (2023-06-24T10:39:44Z)
• Enhancing Machine Learning Model Performance with Hyper Parameter Optimization: A Comparative Study [0.0]
  One of the most critical issues in machine learning is the selection of appropriate hyper parameters for training models. Hyper parameter optimization (HPO) is a popular topic that artificial intelligence studies have focused on recently. In this study, classical methods, such as grid, random search and Bayesian optimization, and population-based algorithms, such as genetic algorithms and particle swarm optimization, are discussed.
  arXiv  Detail & Related papers  (2023-02-14T10:12:10Z)
• Improved Algorithms for Neural Active Learning [74.89097665112621]
  We improve the theoretical and empirical performance of neural-network(NN)-based active learning algorithms for the non-parametric streaming setting. We introduce two regret metrics by minimizing the population loss that are more suitable in active learning than the one used in state-of-the-art (SOTA) related work.
  arXiv  Detail & Related papers  (2022-10-02T05:03:38Z)
• Optimizing Large-Scale Hyperparameters via Automated Learning Algorithm [97.66038345864095]
  We propose a new hyperparameter optimization method with zeroth-order hyper-gradients (HOZOG) Specifically, we first formulate hyperparameter optimization as an A-based constrained optimization problem. Then, we use the average zeroth-order hyper-gradients to update hyper parameters.
  arXiv  Detail & Related papers  (2021-02-17T21:03:05Z)
• Online hyperparameter optimization by real-time recurrent learning [57.01871583756586]
  Our framework takes advantage of the analogy between hyperparameter optimization and parameter learning in neural networks (RNNs) It adapts a well-studied family of online learning algorithms for RNNs to tune hyperparameters and network parameters simultaneously. This procedure yields systematically better generalization performance compared to standard methods, at a fraction of wallclock time.
  arXiv  Detail & Related papers  (2021-02-15T19:36:18Z)
• Leveraging Benchmarking Data for Informed One-Shot Dynamic Algorithm Selection [0.9281671380673306]
  A key challenge in the application of evolutionary algorithms is the selection of an algorithm instance that best suits the problem at hand. We analyze in this work how such prior performance data can be used to infer informed dynamic algorithm selection schemes for the solution of pseudo-Boolean optimization problems.
  arXiv  Detail & Related papers  (2021-02-12T12:27:02Z)
• Towards Optimally Efficient Tree Search with Deep Learning [76.64632985696237]
  This paper investigates the classical integer least-squares problem which estimates signals integer from linear models. The problem is NP-hard and often arises in diverse applications such as signal processing, bioinformatics, communications and machine learning. We propose a general hyper-accelerated tree search (HATS) algorithm by employing a deep neural network to estimate the optimal estimation for the underlying simplified memory-bounded A* algorithm.
  arXiv  Detail & Related papers  (2021-01-07T08:00:02Z)
• Generalization in portfolio-based algorithm selection [97.74604695303285]
  We provide the first provable guarantees for portfolio-based algorithm selection. We show that if the portfolio is large, overfitting is inevitable, even with an extremely simple algorithm selector.
  arXiv  Detail & Related papers  (2020-12-24T16:33:17Z)
• On tuning deep learning models: a data mining perspective [0.0]
  Four types of deep learning algorithms are investigated in terms of tuning and data mining perspective. The number of features has not contributed to the decline in the accuracy of deep learning algorithms. A uniform distribution is much more crucial to reach reliable results in terms of data mining.
  arXiv  Detail & Related papers  (2020-11-19T14:40:42Z)
• Data-driven Algorithm Design [21.39493074700162]
  Data driven algorithm design is an important aspect of modern data science and algorithm design. A small tweak to the parameters can cause a cascade of changes in the algorithm's behavior. We provide strong computational and statistical performance guarantees for batch and online scenarios.
  arXiv  Detail & Related papers  (2020-11-14T00:51:57Z)
• Extreme Algorithm Selection With Dyadic Feature Representation [78.13985819417974]
  We propose the setting of extreme algorithm selection (XAS) where we consider fixed sets of thousands of candidate algorithms. We assess the applicability of state-of-the-art AS techniques to the XAS setting and propose approaches leveraging a dyadic feature representation.
  arXiv  Detail & Related papers  (2020-01-29T09:40:58Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.