Gradient Boosting Mapping for Dimensionality Reduction and Feature Extraction

• URL: http://arxiv.org/abs/2405.08486v1
• Date: Tue, 14 May 2024 10:23:57 GMT
• Title: Gradient Boosting Mapping for Dimensionality Reduction and Feature Extraction
• Authors: Anri Patron, Ayush Prasad, Hoang Phuc Hau Luu, Kai Puolamäki,
• Abstract summary: A fundamental problem in supervised learning is to find a good set of features or distance measures. We propose a supervised dimensionality reduction method, where the outputs of weak learners define the embedding. We show that the embedding coordinates provide better features for the supervised learning task.
• Score: 2.778647101651566
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: A fundamental problem in supervised learning is to find a good set of features or distance measures. If the new set of features is of lower dimensionality and can be obtained by a simple transformation of the original data, they can make the model understandable, reduce overfitting, and even help to detect distribution drift. We propose a supervised dimensionality reduction method Gradient Boosting Mapping (GBMAP), where the outputs of weak learners -- defined as one-layer perceptrons -- define the embedding. We show that the embedding coordinates provide better features for the supervised learning task, making simple linear models competitive with the state-of-the-art regressors and classifiers. We also use the embedding to find a principled distance measure between points. The features and distance measures automatically ignore directions irrelevant to the supervised learning task. We also show that we can reliably detect out-of-distribution data points with potentially large regression or classification errors. GBMAP is fast and works in seconds for dataset of million data points or hundreds of features. As a bonus, GBMAP provides a regression and classification performance comparable to the state-of-the-art supervised learning methods.

Related papers

• Enhancing Consistency and Mitigating Bias: A Data Replay Approach for Incremental Learning [100.7407460674153]
  Deep learning systems are prone to catastrophic forgetting when learning from a sequence of tasks. To mitigate the problem, a line of methods propose to replay the data of experienced tasks when learning new tasks. However, it is not expected in practice considering the memory constraint or data privacy issue. As a replacement, data-free data replay methods are proposed by inverting samples from the classification model.
  arXiv  Detail & Related papers  (2024-01-12T12:51:12Z)
• PointJEM: Self-supervised Point Cloud Understanding for Reducing Feature Redundancy via Joint Entropy Maximization [10.53900407467811]
  We propose PointJEM, a self-supervised representation learning method applied to the point cloud field. To reduce redundant information in the features, PointJEM maximizes the joint entropy between the different parts. PointJEM achieves competitive performance in downstream tasks such as classification and segmentation.
  arXiv  Detail & Related papers  (2023-12-06T08:21:42Z)
• Bidirectional Knowledge Reconfiguration for Lightweight Point Cloud Analysis [74.00441177577295]
  Point cloud analysis faces computational system overhead, limiting its application on mobile or edge devices. This paper explores feature distillation for lightweight point cloud models. We propose bidirectional knowledge reconfiguration to distill informative contextual knowledge from the teacher to the student.
  arXiv  Detail & Related papers  (2023-10-08T11:32:50Z)
• Adaptive Cross Batch Normalization for Metric Learning [75.91093210956116]
  Metric learning is a fundamental problem in computer vision. We show that it is equally important to ensure that the accumulated embeddings are up to date. In particular, it is necessary to circumvent the representational drift between the accumulated embeddings and the feature embeddings at the current training iteration.
  arXiv  Detail & Related papers  (2023-03-30T03:22:52Z)
• Model-Change Active Learning in Graph-Based Semi-Supervised Learning [7.208515071018781]
  "Model-change" active learning quantifies the resulting change incurred in the classifier by introducing the additional label(s) We consider a family of convex loss functions for which the acquisition function can be efficiently approximated using the Laplace approximation of the posterior distribution.
  arXiv  Detail & Related papers  (2021-10-14T21:47:10Z)
• Towards Open-World Feature Extrapolation: An Inductive Graph Learning Approach [80.8446673089281]
  We propose a new learning paradigm with graph representation and learning. Our framework contains two modules: 1) a backbone network (e.g., feedforward neural nets) as a lower model takes features as input and outputs predicted labels; 2) a graph neural network as an upper model learns to extrapolate embeddings for new features via message passing over a feature-data graph built from observed data.
  arXiv  Detail & Related papers  (2021-10-09T09:02:45Z)
• Finding Significant Features for Few-Shot Learning using Dimensionality Reduction [0.0]
  This module helps to improve the accuracy performance by allowing the similarity function, given by the metric learning method, to have more discriminative features for the classification. Our method outperforms the metric learning baselines in the miniImageNet dataset by around 2% in accuracy performance.
  arXiv  Detail & Related papers  (2021-07-06T16:36:57Z)
• MetricOpt: Learning to Optimize Black-Box Evaluation Metrics [21.608384691401238]
  We study the problem of optimizing arbitrary non-differentiable task evaluation metrics such as misclassification rate and recall. Our method, named MetricOpt, operates in a black-box setting where the computational details of the target metric are unknown. We achieve this by learning a differentiable value function, which maps compact task-specific model parameters to metric observations.
  arXiv  Detail & Related papers  (2021-04-21T16:50:01Z)
• Progressive Self-Guided Loss for Salient Object Detection [102.35488902433896]
  We present a progressive self-guided loss function to facilitate deep learning-based salient object detection in images. Our framework takes advantage of adaptively aggregated multi-scale features to locate and detect salient objects effectively.
  arXiv  Detail & Related papers  (2021-01-07T07:33:38Z)
• Learning What Makes a Difference from Counterfactual Examples and Gradient Supervision [57.14468881854616]
  We propose an auxiliary training objective that improves the generalization capabilities of neural networks. We use pairs of minimally-different examples with different labels, a.k.a counterfactual or contrasting examples, which provide a signal indicative of the underlying causal structure of the task. Models trained with this technique demonstrate improved performance on out-of-distribution test sets.
  arXiv  Detail & Related papers  (2020-04-20T02:47:49Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.