Structure Learning with Continuous Optimization: A Sober Look and Beyond

• URL: http://arxiv.org/abs/2304.02146v2
• Date: Mon, 19 Aug 2024 17:13:58 GMT
• Title: Structure Learning with Continuous Optimization: A Sober Look and Beyond
• Authors: Ignavier Ng, Biwei Huang, Kun Zhang,
• Abstract summary: This paper investigates in which cases continuous optimization for acyclic graph (DAG) structure learning can and cannot perform well. We provide insights into several aspects of the search procedure, including thresholding and sparsity, and show that they play an important role in the final solutions.
• Score: 21.163991683650526
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: This paper investigates in which cases continuous optimization for directed acyclic graph (DAG) structure learning can and cannot perform well and why this happens, and suggests possible directions to make the search procedure more reliable. Reisach et al. (2021) suggested that the remarkable performance of several continuous structure learning approaches is primarily driven by a high agreement between the order of increasing marginal variances and the topological order, and demonstrated that these approaches do not perform well after data standardization. We analyze this phenomenon for continuous approaches assuming equal and non-equal noise variances, and show that the statement may not hold in either case by providing counterexamples, justifications, and possible alternative explanations. We further demonstrate that nonconvexity may be a main concern especially for the non-equal noise variances formulation, while recent advances in continuous structure learning fail to achieve improvement in this case. Our findings suggest that future works should take into account the non-equal noise variances formulation to handle more general settings and for a more comprehensive empirical evaluation. Lastly, we provide insights into other aspects of the search procedure, including thresholding and sparsity, and show that they play an important role in the final solutions.

Related papers

• Revisiting Differentiable Structure Learning: Inconsistency of $\ell_1$ Penalty and Beyond [19.373348700715578]
  Recent advances in differentiable structure learning have framed the problem of learning directed acyclic graphs as a continuous optimization problem. In this work, we investigate critical limitations in differentiable structure learning methods.
  arXiv  Detail & Related papers  (2024-10-24T03:17:14Z)
• On the Generalization Ability of Unsupervised Pretraining [53.06175754026037]
  Recent advances in unsupervised learning have shown that unsupervised pre-training, followed by fine-tuning, can improve model generalization. This paper introduces a novel theoretical framework that illuminates the critical factor influencing the transferability of knowledge acquired during unsupervised pre-training to the subsequent fine-tuning phase. Our results contribute to a better understanding of unsupervised pre-training and fine-tuning paradigm, and can shed light on the design of more effective pre-training algorithms.
  arXiv  Detail & Related papers  (2024-03-11T16:23:42Z)
• Hierarchical Decomposition of Prompt-Based Continual Learning: Rethinking Obscured Sub-optimality [55.88910947643436]
  Self-supervised pre-training is essential for handling vast quantities of unlabeled data in practice. HiDe-Prompt is an innovative approach that explicitly optimize the hierarchical components with an ensemble of task-specific prompts and statistics. Our experiments demonstrate the superior performance of HiDe-Prompt and its robustness to pre-training paradigms in continual learning.
  arXiv  Detail & Related papers  (2023-10-11T06:51:46Z)
• Differentiable Bayesian Structure Learning with Acyclicity Assurance [7.568978862189266]
  We propose an alternative approach for strictly constraining the acyclicty of the graphs with an integration of the knowledge from the topological orderings. Our approach can reduce inference complexity while ensuring the structures of the generated graphs to be acyclic.
  arXiv  Detail & Related papers  (2023-09-04T06:44:46Z)
• Advancing Counterfactual Inference through Nonlinear Quantile Regression [77.28323341329461]
  We propose a framework for efficient and effective counterfactual inference implemented with neural networks. The proposed approach enhances the capacity to generalize estimated counterfactual outcomes to unseen data. Empirical results conducted on multiple datasets offer compelling support for our theoretical assertions.
  arXiv  Detail & Related papers  (2023-06-09T08:30:51Z)
• Causal Structural Learning from Time Series: A Convex Optimization Approach [12.4517307615083]
  Structural learning aims to learn directed acyclic graphs (DAGs) from observational data. Recent DAG learning remains a highly non-adaptive structural learning problem. We propose a data approach for causal learning using a recently developed monotone variational (VI) formulation.
  arXiv  Detail & Related papers  (2023-01-26T16:39:58Z)
• Tractable Uncertainty for Structure Learning [21.46601360284884]
  We present Tractable Uncertainty for STructure, a framework for approximate posterior inference. Probability circuits can be used as an augmented representation for structure learning methods.
  arXiv  Detail & Related papers  (2022-04-29T15:54:39Z)
• FineDiving: A Fine-grained Dataset for Procedure-aware Action Quality Assessment [93.09267863425492]
  We argue that understanding both high-level semantics and internal temporal structures of actions in competitive sports videos is the key to making predictions accurate and interpretable. We construct a new fine-grained dataset, called FineDiving, developed on diverse diving events with detailed annotations on action procedures.
  arXiv  Detail & Related papers  (2022-04-07T17:59:32Z)
• Interpolation can hurt robust generalization even when there is no noise [76.3492338989419]
  We show that avoiding generalization through ridge regularization can significantly improve generalization even in the absence of noise. We prove this phenomenon for the robust risk of both linear regression and classification and hence provide the first theoretical result on robust overfitting.
  arXiv  Detail & Related papers  (2021-08-05T23:04:15Z)
• Convergence rates and approximation results for SGD and its continuous-time counterpart [16.70533901524849]
  This paper proposes a thorough theoretical analysis of convex Gradient Descent (SGD) with non-increasing step sizes. First, we show that the SGD can be provably approximated by solutions of inhomogeneous Differential Equation (SDE) using coupling. Recent analyses of deterministic and optimization methods by their continuous counterpart, we study the long-time behavior of the continuous processes at hand and non-asymptotic bounds.
  arXiv  Detail & Related papers  (2020-04-08T18:31:34Z)
• Learning Overlapping Representations for the Estimation of Individualized Treatment Effects [97.42686600929211]
  Estimating the likely outcome of alternatives from observational data is a challenging problem. We show that algorithms that learn domain-invariant representations of inputs are often inappropriate. We develop a deep kernel regression algorithm and posterior regularization framework that substantially outperforms the state-of-the-art on a variety of benchmarks data sets.
  arXiv  Detail & Related papers  (2020-01-14T12:56:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.