Topological Regularization via Persistence-Sensitive Optimization

• URL: http://arxiv.org/abs/2011.05290v1
• Date: Tue, 10 Nov 2020 18:19:43 GMT
• Title: Topological Regularization via Persistence-Sensitive Optimization
• Authors: Arnur Nigmetov, Aditi S. Krishnapriyan, Nicole Sanderson, Dmitriy Morozov
• Abstract summary: A key tool in machine learning and statistics, regularization relies on regularization to reduce overfitting. We propose a method that builds on persistence-sensitive simplification and translates required changes to the persistence diagram into changes on large subsets of the domain. This approach enables a faster and more precise topological regularization, the benefits of which we illustrate.
• Score: 10.29838087001588
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Optimization, a key tool in machine learning and statistics, relies on regularization to reduce overfitting. Traditional regularization methods control a norm of the solution to ensure its smoothness. Recently, topological methods have emerged as a way to provide a more precise and expressive control over the solution, relying on persistent homology to quantify and reduce its roughness. All such existing techniques back-propagate gradients through the persistence diagram, which is a summary of the topological features of a function. Their downside is that they provide information only at the critical points of the function. We propose a method that instead builds on persistence-sensitive simplification and translates the required changes to the persistence diagram into changes on large subsets of the domain, including both critical and regular points. This approach enables a faster and more precise topological regularization, the benefits of which we illustrate with experimental evidence.

Related papers

• Beyond Progress Measures: Theoretical Insights into the Mechanism of Grokking [50.465604300990904]
  Grokking refers to the abrupt improvement in test accuracy after extended overfitting. In this work, we investigate the grokking mechanism underlying the Transformer in the task of prime number operations.
  arXiv  Detail & Related papers  (2025-04-04T04:42:38Z)
• Enhanced Denoising and Convergent Regularisation Using Tweedie Scaling [10.35768862376698]
  This work introduces a novel scaling method that explicitly integrates and adjusts the strength of regularisation. The scaling parameter enhances interpretability by reflecting the quality of the denoiser's learning process. The proposed approach ensures that the resulting family of regularisations is provably stable and convergent.
  arXiv  Detail & Related papers  (2025-03-07T21:51:59Z)
• Hard constraint learning approaches with trainable influence functions for evolutionary equations [8.812375888020398]
  This paper develops a novel deep learning approach for solving evolutionary equations. Sequential learning strategies divide a large temporal domain into multiple subintervals and solve them one by one in a chronological order. The improved hard constraint strategy strictly ensures the continuity and smoothness of the PINN solution at time interval nodes.
  arXiv  Detail & Related papers  (2025-02-21T07:54:01Z)
• Function-Space Regularization in Neural Networks: A Probabilistic Perspective [51.133793272222874]
  We show that we can derive a well-motivated regularization technique that allows explicitly encoding information about desired predictive functions into neural network training. We evaluate the utility of this regularization technique empirically and demonstrate that the proposed method leads to near-perfect semantic shift detection and highly-calibrated predictive uncertainty estimates.
  arXiv  Detail & Related papers  (2023-12-28T17:50:56Z)
• FastPart: Over-Parameterized Stochastic Gradient Descent for Sparse optimisation on Measures [1.9950682531209156]
  This paper presents a novel algorithm that leverages Gradient Descent strategies in conjunction with Random Features to augment the scalability of Conic Particle Gradient Descent (CPGD) We provide rigorous proofs demonstrating the following key findings: (i) The total variation norms of the solution measures along the descent trajectory remain bounded, ensuring stability and preventing undesirable divergence; (ii) We establish a global convergence guarantee with a convergence rate of $mathcalO(log(K)/sqrtK)$ over $K$, showcasing the efficiency and effectiveness of our algorithm; (iii) Additionally, we analyze and establish
  arXiv  Detail & Related papers  (2023-12-10T20:41:43Z)
• Smoothing the Edges: Smooth Optimization for Sparse Regularization using Hadamard Overparametrization [10.009748368458409]
  We present a framework for smooth optimization of explicitly regularized objectives for (structured) sparsity. Our method enables fully differentiable approximation-free optimization and is thus compatible with the ubiquitous gradient descent paradigm in deep learning.
  arXiv  Detail & Related papers  (2023-07-07T13:06:12Z)
• Scalable Bayesian Meta-Learning through Generalized Implicit Gradients [64.21628447579772]
  Implicit Bayesian meta-learning (iBaML) method broadens the scope of learnable priors, but also quantifies the associated uncertainty. Analytical error bounds are established to demonstrate the precision and efficiency of the generalized implicit gradient over the explicit one.
  arXiv  Detail & Related papers  (2023-03-31T02:10:30Z)
• Incremental Correction in Dynamic Systems Modelled with Neural Networks for Constraint Satisfaction [6.729108277517129]
  The proposed approach is to linearise the dynamics around the baseline values of its arguments. The online update approach can be useful for enhancing overall targeting accuracy of finite-horizon control.
  arXiv  Detail & Related papers  (2022-09-08T10:33:30Z)
• Domain-Adjusted Regression or: ERM May Already Learn Features Sufficient for Out-of-Distribution Generalization [52.7137956951533]
  We argue that devising simpler methods for learning predictors on existing features is a promising direction for future research. We introduce Domain-Adjusted Regression (DARE), a convex objective for learning a linear predictor that is provably robust under a new model of distribution shift. Under a natural model, we prove that the DARE solution is the minimax-optimal predictor for a constrained set of test distributions.
  arXiv  Detail & Related papers  (2022-02-14T16:42:16Z)
• Breaking the Convergence Barrier: Optimization via Fixed-Time Convergent Flows [4.817429789586127]
  We introduce a Poly-based optimization framework for achieving acceleration, based on the notion of fixed-time stability dynamical systems. We validate the accelerated convergence properties of the proposed schemes on a range of numerical examples against the state-of-the-art optimization algorithms.
  arXiv  Detail & Related papers  (2021-12-02T16:04:40Z)
• Automating Control of Overestimation Bias for Continuous Reinforcement Learning [65.63607016094305]
  We present a data-driven approach for guiding bias correction. We demonstrate its effectiveness on the Truncated Quantile Critics -- a state-of-the-art continuous control algorithm.
  arXiv  Detail & Related papers  (2021-10-26T09:27:12Z)
• Deep learning: a statistical viewpoint [120.94133818355645]
  Deep learning has revealed some major surprises from a theoretical perspective. In particular, simple gradient methods easily find near-perfect solutions to non-optimal training problems. We conjecture that specific principles underlie these phenomena.
  arXiv  Detail & Related papers  (2021-03-16T16:26:36Z)
• A Fast and Robust Method for Global Topological Functional Optimization [70.11080854486953]
  We introduce a novel backpropagation scheme that is significantly faster, more stable, and produces more robust optima. This scheme can also be used to produce a stable visualization of dots in a persistence diagram as a distribution over critical, and near-critical, simplices in the data structure.
  arXiv  Detail & Related papers  (2020-09-17T18:46:16Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.