Subspace Learning Machine (SLM): Methodology and Performance
- URL: http://arxiv.org/abs/2205.05296v1
- Date: Wed, 11 May 2022 06:44:51 GMT
- Title: Subspace Learning Machine (SLM): Methodology and Performance
- Authors: Hongyu Fu, Yijing Yang, Vinod K. Mishra, C.-C. Jay Kuo
- Abstract summary: Subspace learning machine (SLM) is a new classification model inspired by feedforward multilayer perceptron (FF-MLP), decision tree (DT) and extreme learning machine (ELM)
SLM first identifies a discriminant subspace, $S0$, by examining the discriminant power of each input feature.
It uses probabilistic projections of features in $S0$ to yield 1D subspaces and finds the optimal partition for each of them.
- Score: 28.98486923400986
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: Inspired by the feedforward multilayer perceptron (FF-MLP), decision tree
(DT) and extreme learning machine (ELM), a new classification model, called the
subspace learning machine (SLM), is proposed in this work. SLM first identifies
a discriminant subspace, $S^0$, by examining the discriminant power of each
input feature. Then, it uses probabilistic projections of features in $S^0$ to
yield 1D subspaces and finds the optimal partition for each of them. This is
equivalent to partitioning $S^0$ with hyperplanes. A criterion is developed to
choose the best $q$ partitions that yield $2q$ partitioned subspaces among
them. We assign $S^0$ to the root node of a decision tree and the intersections
of $2q$ subspaces to its child nodes of depth one. The partitioning process is
recursively applied at each child node to build an SLM tree. When the samples
at a child node are sufficiently pure, the partitioning process stops and each
leaf node makes a prediction. The idea can be generalized to regression,
leading to the subspace learning regressor (SLR). Furthermore, ensembles of
SLM/SLR trees can yield a stronger predictor. Extensive experiments are
conducted for performance benchmarking among SLM/SLR trees, ensembles and
classical classifiers/regressors.
Related papers
- Nested subspace learning with flags [0.03295497357381917]
We propose a simple trick to enforce nestedness in subspace learning methods.
We apply the flag trick to several classical machine learning methods and show that it successfully addresses the nestedness issue.
arXiv Detail & Related papers (2025-02-09T20:29:56Z) - ODTE -- An ensemble of multi-class SVM-based oblique decision trees [0.7182449176083623]
ODTE is a new ensemble that uses oblique decision trees as base classifiers.
We introduce STree, the base algorithm for growing oblique decision trees.
ODTE ranks consistently above its competitors.
arXiv Detail & Related papers (2024-11-20T14:58:32Z) - Projection by Convolution: Optimal Sample Complexity for Reinforcement Learning in Continuous-Space MDPs [56.237917407785545]
We consider the problem of learning an $varepsilon$-optimal policy in a general class of continuous-space Markov decision processes (MDPs) having smooth Bellman operators.
Key to our solution is a novel projection technique based on ideas from harmonic analysis.
Our result bridges the gap between two popular but conflicting perspectives on continuous-space MDPs.
arXiv Detail & Related papers (2024-05-10T09:58:47Z) - Hierarchical Partitioning Forecaster [0.571097144710995]
We consider a new family of algorithms for sequential prediction, Hierarchical Partitioning Forecasters (HPFs)
Our goal is to provide appealing theoretical - regret guarantees on a powerful model class.
arXiv Detail & Related papers (2023-05-22T14:25:46Z) - Simplifying and Understanding State Space Models with Diagonal Linear
RNNs [56.33053691749856]
This work disposes of the discretization step, and proposes a model based on vanilla Diagonal Linear RNNs.
We empirically show that, despite being conceptually much simpler, $mathrmDLR$ is as performant as previously-proposed SSMs.
We also characterize the expressivity of SSMs and attention-based models via a suite of $13$ synthetic sequence-to-sequence tasks.
arXiv Detail & Related papers (2022-12-01T18:53:06Z) - When are Local Queries Useful for Robust Learning? [25.832511407411637]
We study learning models where the learner is given more power through the use of local queries.
We give the first distribution-free algorithms that perform robust empirical risk minimization.
We finish by giving robust learning algorithms for halfspaces on $0,1n$ and then obtaining robustness guarantees for halfspaces in $mathbbRn$ against precision-bounded adversaries.
arXiv Detail & Related papers (2022-10-12T11:04:22Z) - The Power of Subsampling in Submodular Maximization [51.629656762796564]
We show that this approach leads to optimal/state-of-the-art results despite being much simpler than existing methods.
We empirically demonstrate the effectiveness of our algorithms on video summarization, location summarization, and movie recommendation tasks.
arXiv Detail & Related papers (2021-04-06T20:25:57Z) - Deep Learning Meets Projective Clustering [66.726500395069]
A common approach for compressing NLP networks is to encode the embedding layer as a matrix $AinmathbbRntimes d$.
Inspired by emphprojective clustering from computational geometry, we suggest replacing this subspace by a set of $k$ subspaces.
arXiv Detail & Related papers (2020-10-08T22:47:48Z) - From Two-Class Linear Discriminant Analysis to Interpretable Multilayer
Perceptron Design [31.446335485087758]
A closed-form solution exists in two-class linear discriminant analysis (LDA)
We interpret the multilayer perceptron (MLP) as a generalization of a two-class LDA system.
We present an automatic design that can specify the network architecture and all filter weights in a feedforward one-pass fashion.
arXiv Detail & Related papers (2020-09-09T17:43:39Z) - Provably Efficient Reinforcement Learning for Discounted MDPs with
Feature Mapping [99.59319332864129]
In this paper, we study reinforcement learning for discounted Decision (MDP)
We propose a novel algorithm that makes use of the feature mapping and obtains a $tilde O(dsqrtT/ (1-gamma)2)$ regret.
Our upper and lower bound results together suggest that the proposed reinforcement learning algorithm is near-optimal up to a $ (1-gamma)-0.5$ factor.
arXiv Detail & Related papers (2020-06-23T17:08:54Z) - Learning Halfspaces with Tsybakov Noise [50.659479930171585]
We study the learnability of halfspaces in the presence of Tsybakov noise.
We give an algorithm that achieves misclassification error $epsilon$ with respect to the true halfspace.
arXiv Detail & Related papers (2020-06-11T14:25:02Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.