Attentional meta-learners are polythetic classifiers
- URL: http://arxiv.org/abs/2106.05317v1
- Date: Wed, 9 Jun 2021 18:11:54 GMT
- Title: Attentional meta-learners are polythetic classifiers
- Authors: Ben Day, Ramon Vi\~nas, Nikola Simidjievski, Pietro Li\`o
- Abstract summary: We show that threshold meta-learners require an embedding dimension that is exponential in the number of features to emulate these functions.
We propose a self-attention feature-selection mechanism that adaptively dilutes non-discriminative features.
- Score: 5.543867614999908
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Polythetic classifications, based on shared patterns of features that need
neither be universal nor constant among members of a class, are common in the
natural world and greatly outnumber monothetic classifications over a set of
features. We show that threshold meta-learners require an embedding dimension
that is exponential in the number of features to emulate these functions. In
contrast, attentional classifiers are polythetic by default and able to solve
these problems with a linear embedding dimension. However, we find that in the
presence of task-irrelevant features, inherent to meta-learning problems,
attentional models are susceptible to misclassification. To address this
challenge, we further propose a self-attention feature-selection mechanism that
adaptively dilutes non-discriminative features. We demonstrate the
effectiveness of our approach in meta-learning Boolean functions, and synthetic
and real-world few-shot learning tasks.
Related papers
- Class-specific feature selection for classification explainability [0.0]
This work first introduces a comprehensive review of the concept of class-specific, with a focus on feature selection and classification.
The fundamental idea of the class-specific concept resides in the understanding that the significance of each feature can vary from one class to another.
This class-specific perspective offers a more effective approach to classification tasks by recognizing and leveraging the unique characteristics of each class.
arXiv Detail & Related papers (2024-11-02T10:31:55Z) - Impossibility of Characterizing Distribution Learning -- a simple
solution to a long-standing problem [11.39656079889941]
We show that there is no notion of dimension that characterizes the sample complexity of learning distribution classes.
In particular, we show that there is no notion of characterizing dimension (or characterization of learnability) for any of the tasks.
arXiv Detail & Related papers (2023-04-18T03:23:39Z) - Reconnoitering the class distinguishing abilities of the features, to
know them better [6.026640792312181]
Explainability can allow end-users to have a transparent and humane reckoning of a machine learning scheme's capability and utility.
In this work, we explain the features on the basis of their class or category-distinguishing capabilities.
We validate the explainability given by our scheme empirically on several real-world, multi-class datasets.
arXiv Detail & Related papers (2022-11-23T08:39:41Z) - Equivariance with Learned Canonicalization Functions [77.32483958400282]
We show that learning a small neural network to perform canonicalization is better than using predefineds.
Our experiments show that learning the canonicalization function is competitive with existing techniques for learning equivariant functions across many tasks.
arXiv Detail & Related papers (2022-11-11T21:58:15Z) - Learning Debiased and Disentangled Representations for Semantic
Segmentation [52.35766945827972]
We propose a model-agnostic and training scheme for semantic segmentation.
By randomly eliminating certain class information in each training iteration, we effectively reduce feature dependencies among classes.
Models trained with our approach demonstrate strong results on multiple semantic segmentation benchmarks.
arXiv Detail & Related papers (2021-10-31T16:15:09Z) - Meta-learning Amidst Heterogeneity and Ambiguity [11.061517140668961]
We devise a novel meta-learning framework, called Meta-learning Amidst Heterogeneity and Ambiguity (MAHA)
By extensively conducting several experiments in regression and classification, we demonstrate the validity of our model.
arXiv Detail & Related papers (2021-07-05T18:54:31Z) - Intersection Regularization for Extracting Semantic Attributes [72.53481390411173]
We consider the problem of supervised classification, such that the features that the network extracts match an unseen set of semantic attributes.
For example, when learning to classify images of birds into species, we would like to observe the emergence of features that zoologists use to classify birds.
We propose training a neural network with discrete top-level activations, which is followed by a multi-layered perceptron (MLP) and a parallel decision tree.
arXiv Detail & Related papers (2021-03-22T14:32:44Z) - Learning Abstract Task Representations [0.6690874707758511]
We propose a method to induce new abstract meta-features as latent variables in a deep neural network.
We demonstrate our methodology using a deep neural network as a feature extractor.
arXiv Detail & Related papers (2021-01-19T20:31:02Z) - A Trainable Optimal Transport Embedding for Feature Aggregation and its
Relationship to Attention [96.77554122595578]
We introduce a parametrized representation of fixed size, which embeds and then aggregates elements from a given input set according to the optimal transport plan between the set and a trainable reference.
Our approach scales to large datasets and allows end-to-end training of the reference, while also providing a simple unsupervised learning mechanism with small computational cost.
arXiv Detail & Related papers (2020-06-22T08:35:58Z) - Towards Efficient Processing and Learning with Spikes: New Approaches
for Multi-Spike Learning [59.249322621035056]
We propose two new multi-spike learning rules which demonstrate better performance over other baselines on various tasks.
In the feature detection task, we re-examine the ability of unsupervised STDP with its limitations being presented.
Our proposed learning rules can reliably solve the task over a wide range of conditions without specific constraints being applied.
arXiv Detail & Related papers (2020-05-02T06:41:20Z) - Learning Class Regularized Features for Action Recognition [68.90994813947405]
We introduce a novel method named Class Regularization that performs class-based regularization of layer activations.
We show that using Class Regularization blocks in state-of-the-art CNN architectures for action recognition leads to systematic improvement gains of 1.8%, 1.2% and 1.4% on the Kinetics, UCF-101 and HMDB-51 datasets, respectively.
arXiv Detail & Related papers (2020-02-07T07:27:49Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.