Markov-Lipschitz Deep Learning
- URL: http://arxiv.org/abs/2006.08256v5
- Date: Wed, 30 Sep 2020 09:17:19 GMT
- Title: Markov-Lipschitz Deep Learning
- Authors: Stan Z. Li, Zelin Zang, Lirong Wu
- Abstract summary: A prior constraint, called locally smoothness (LIS), is imposed across-layers and encoded into a Markov random field (MRF)-Gibbs distribution.
This leads to the best possible solutions for local geometry preservation and robustness.
Experiments, comparisons, and ablation study demonstrate significant advantages of MLDL for manifold learning and manifold data generation.
- Score: 37.7499958388076
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: We propose a novel framework, called Markov-Lipschitz deep learning (MLDL),
to tackle geometric deterioration caused by collapse, twisting, or crossing in
vector-based neural network transformations for manifold-based representation
learning and manifold data generation. A prior constraint, called locally
isometric smoothness (LIS), is imposed across-layers and encoded into a Markov
random field (MRF)-Gibbs distribution. This leads to the best possible
solutions for local geometry preservation and robustness as measured by locally
geometric distortion and locally bi-Lipschitz continuity. Consequently, the
layer-wise vector transformations are enhanced into well-behaved,
LIS-constrained metric homeomorphisms. Extensive experiments, comparisons, and
ablation study demonstrate significant advantages of MLDL for manifold learning
and manifold data generation. MLDL is general enough to enhance any vector
transformation-based networks. The code is available at
https://github.com/westlake-cairi/Markov-Lipschitz-Deep-Learning.
Related papers
- Deep Learning as Ricci Flow [38.27936710747996]
Deep neural networks (DNNs) are powerful tools for approximating the distribution of complex data.
We show that the transformations performed by DNNs during classification tasks have parallels to those expected under Hamilton's Ricci flow.
Our findings motivate the use of tools from differential and discrete geometry to the problem of explainability in deep learning.
arXiv Detail & Related papers (2024-04-22T15:12:47Z) - Scalable manifold learning by uniform landmark sampling and constrained
locally linear embedding [0.6144680854063939]
We propose a scalable manifold learning (scML) method that can manipulate large-scale and high-dimensional data in an efficient manner.
We empirically validated the effectiveness of scML on synthetic datasets and real-world benchmarks of different types.
scML scales well with increasing data sizes and embedding dimensions, and exhibits promising performance in preserving the global structure.
arXiv Detail & Related papers (2024-01-02T08:43:06Z) - ParsNets: A Parsimonious Orthogonal and Low-Rank Linear Networks for
Zero-Shot Learning [22.823915322924304]
This paper provides a novel parsimonious yet efficient design for zero-shot learning (ZSL), dubbed ParsNets, to achieve equivalent or even better performance against existing deep models.
To facilitate the generalization of local linearities, we construct a maximal margin geometry on the learned features by enforcing low-rank constraints on intra-class samples and high-rank constraints on inter-class samples.
To enhance the model's adaptability and counterbalance over/under-fittings in ZSL, a set of sample-wise indicators is employed to select a sparse subset from these base linear networks to form a composite
arXiv Detail & Related papers (2023-12-15T11:32:11Z) - A Unified Algebraic Perspective on Lipschitz Neural Networks [88.14073994459586]
This paper introduces a novel perspective unifying various types of 1-Lipschitz neural networks.
We show that many existing techniques can be derived and generalized via finding analytical solutions of a common semidefinite programming (SDP) condition.
Our approach, called SDP-based Lipschitz Layers (SLL), allows us to design non-trivial yet efficient generalization of convex potential layers.
arXiv Detail & Related papers (2023-03-06T14:31:09Z) - Scaling Forward Gradient With Local Losses [117.22685584919756]
Forward learning is a biologically plausible alternative to backprop for learning deep neural networks.
We show that it is possible to substantially reduce the variance of the forward gradient by applying perturbations to activations rather than weights.
Our approach matches backprop on MNIST and CIFAR-10 and significantly outperforms previously proposed backprop-free algorithms on ImageNet.
arXiv Detail & Related papers (2022-10-07T03:52:27Z) - GSMFlow: Generation Shifts Mitigating Flow for Generalized Zero-Shot
Learning [55.79997930181418]
Generalized Zero-Shot Learning aims to recognize images from both the seen and unseen classes by transferring semantic knowledge from seen to unseen classes.
It is a promising solution to take the advantage of generative models to hallucinate realistic unseen samples based on the knowledge learned from the seen classes.
We propose a novel flow-based generative framework that consists of multiple conditional affine coupling layers for learning unseen data generation.
arXiv Detail & Related papers (2022-07-05T04:04:37Z) - Circular-Symmetric Correlation Layer based on FFT [11.634729459989996]
We propose a Circular-symmetric Correlation Layer (CCL) based on the formalism of roto-translation equivariant correlation on the continuous group $S1 times mathbbR$.
We showcase the performance analysis of a general network equipped with CCL on various recognition and classification tasks and datasets.
arXiv Detail & Related papers (2021-07-26T21:06:20Z) - ResNet-LDDMM: Advancing the LDDMM Framework Using Deep Residual Networks [86.37110868126548]
In this work, we make use of deep residual neural networks to solve the non-stationary ODE (flow equation) based on a Euler's discretization scheme.
We illustrate these ideas on diverse registration problems of 3D shapes under complex topology-preserving transformations.
arXiv Detail & Related papers (2021-02-16T04:07:13Z) - Belief Propagation Reloaded: Learning BP-Layers for Labeling Problems [83.98774574197613]
We take one of the simplest inference methods, a truncated max-product Belief propagation, and add what is necessary to make it a proper component of a deep learning model.
This BP-Layer can be used as the final or an intermediate block in convolutional neural networks (CNNs)
The model is applicable to a range of dense prediction problems, is well-trainable and provides parameter-efficient and robust solutions in stereo, optical flow and semantic segmentation.
arXiv Detail & Related papers (2020-03-13T13:11:35Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.