Manifoldron: Direct Space Partition via Manifold Discovery
- URL: http://arxiv.org/abs/2201.05279v1
- Date: Fri, 14 Jan 2022 02:28:17 GMT
- Title: Manifoldron: Direct Space Partition via Manifold Discovery
- Authors: Dayang Wang, Feng-Lei Fan, Bo-Jian Hou, Hao Zhang, Rongjie Lai,
Hengyong Yu, Fei Wang
- Abstract summary: We propose a new type of machine learning models referred to as Manifoldron.
Manifoldron directly derives decision boundaries from data and partitions the space via manifold structure discovery.
We show that the proposed Manifoldron performs competitively compared to the mainstream machine learning models.
- Score: 20.598782641025284
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: A neural network with the widely-used ReLU activation has been shown to
partition the sample space into many convex polytopes for prediction. However,
the parameterized way a neural network and other machine learning models use to
partition the space has imperfections, e.g., the compromised interpretability
for complex models, the inflexibility in decision boundary construction due to
the generic character of the model, and the risk of being trapped into shortcut
solutions. In contrast, although the non-parameterized models can adorably
avoid or downplay these issues, they are usually insufficiently powerful either
due to over-simplification or the failure to accommodate the manifold
structures of data. In this context, we first propose a new type of machine
learning models referred to as Manifoldron that directly derives decision
boundaries from data and partitions the space via manifold structure discovery.
Then, we systematically analyze the key characteristics of the Manifoldron
including interpretability, manifold characterization capability, and its link
to neural networks. The experimental results on 9 small and 11 large datasets
demonstrate that the proposed Manifoldron performs competitively compared to
the mainstream machine learning models. We have shared our code
https://github.com/wdayang/Manifoldron for free download and evaluation.
Related papers
- SMILE: Zero-Shot Sparse Mixture of Low-Rank Experts Construction From Pre-Trained Foundation Models [85.67096251281191]
We present an innovative approach to model fusion called zero-shot Sparse MIxture of Low-rank Experts (SMILE) construction.
SMILE allows for the upscaling of source models into an MoE model without extra data or further training.
We conduct extensive experiments across diverse scenarios, such as image classification and text generation tasks, using full fine-tuning and LoRA fine-tuning.
arXiv Detail & Related papers (2024-08-19T17:32:15Z) - VTAE: Variational Transformer Autoencoder with Manifolds Learning [144.0546653941249]
Deep generative models have demonstrated successful applications in learning non-linear data distributions through a number of latent variables.
The nonlinearity of the generator implies that the latent space shows an unsatisfactory projection of the data space, which results in poor representation learning.
We show that geodesics and accurate computation can substantially improve the performance of deep generative models.
arXiv Detail & Related papers (2023-04-03T13:13:19Z) - The Contextual Lasso: Sparse Linear Models via Deep Neural Networks [5.607237982617641]
We develop a new statistical estimator that fits a sparse linear model to the explanatory features such that the sparsity pattern and coefficients vary as a function of the contextual features.
An extensive suite of experiments on real and synthetic data suggests that the learned models, which remain highly transparent, can be sparser than the regular lasso.
arXiv Detail & Related papers (2023-02-02T05:00:29Z) - Interpretations Steered Network Pruning via Amortized Inferred Saliency
Maps [85.49020931411825]
Convolutional Neural Networks (CNNs) compression is crucial to deploying these models in edge devices with limited resources.
We propose to address the channel pruning problem from a novel perspective by leveraging the interpretations of a model to steer the pruning process.
We tackle this challenge by introducing a selector model that predicts real-time smooth saliency masks for pruned models.
arXiv Detail & Related papers (2022-09-07T01:12:11Z) - Semi-Supervised Manifold Learning with Complexity Decoupled Chart Autoencoders [45.29194877564103]
This work introduces a chart autoencoder with an asymmetric encoding-decoding process that can incorporate additional semi-supervised information such as class labels.
We discuss the approximation power of such networks and derive a bound that essentially depends on the intrinsic dimension of the data manifold rather than the dimension of ambient space.
arXiv Detail & Related papers (2022-08-22T19:58:03Z) - Low-Rank Constraints for Fast Inference in Structured Models [110.38427965904266]
This work demonstrates a simple approach to reduce the computational and memory complexity of a large class of structured models.
Experiments with neural parameterized structured models for language modeling, polyphonic music modeling, unsupervised grammar induction, and video modeling show that our approach matches the accuracy of standard models at large state spaces.
arXiv Detail & Related papers (2022-01-08T00:47:50Z) - The emergence of a concept in shallow neural networks [0.0]
We consider restricted Boltzmann machine (RBMs) trained over an unstructured dataset made of blurred copies of definite but unavailable archetypes''
We show that there exists a critical sample size beyond which the RBM can learn archetypes.
arXiv Detail & Related papers (2021-09-01T15:56:38Z) - Closed-form Continuous-Depth Models [99.40335716948101]
Continuous-depth neural models rely on advanced numerical differential equation solvers.
We present a new family of models, termed Closed-form Continuous-depth (CfC) networks, that are simple to describe and at least one order of magnitude faster.
arXiv Detail & Related papers (2021-06-25T22:08:51Z) - Bridge the Gap Between Model-based and Model-free Human Reconstruction [10.818838437018682]
We present an end-to-end neural network that simultaneously predicts the pixel-aligned implicit surface and the explicit mesh model built by graph convolutional neural network.
Experiments on DeepHuman dataset showed that our approach is effective.
arXiv Detail & Related papers (2021-06-11T11:13:42Z) - Model Fusion via Optimal Transport [64.13185244219353]
We present a layer-wise model fusion algorithm for neural networks.
We show that this can successfully yield "one-shot" knowledge transfer between neural networks trained on heterogeneous non-i.i.d. data.
arXiv Detail & Related papers (2019-10-12T22:07:15Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.