Implicit ZCA Whitening Effects of Linear Autoencoders for Recommendation
- URL: http://arxiv.org/abs/2308.13536v1
- Date: Tue, 15 Aug 2023 07:58:22 GMT
- Title: Implicit ZCA Whitening Effects of Linear Autoencoders for Recommendation
- Authors: Katsuhiko Hayashi and Kazuma Onishi
- Abstract summary: We show a connection between a linear autoencoder model and ZCA whitening for recommendation data.
We also show the correctness of applying a linear autoencoder to low-dimensional item vectors obtained using embedding methods.
- Score: 10.374400063702392
- License: http://creativecommons.org/licenses/by-sa/4.0/
- Abstract: Recently, in the field of recommendation systems, linear regression
(autoencoder) models have been investigated as a way to learn item similarity.
In this paper, we show a connection between a linear autoencoder model and ZCA
whitening for recommendation data. In particular, we show that the dual form
solution of a linear autoencoder model actually has ZCA whitening effects on
feature vectors of items, while items are considered as input features in the
primal problem of the autoencoder/regression model. We also show the
correctness of applying a linear autoencoder to low-dimensional item vectors
obtained using embedding methods such as Item2vec to estimate item-item
similarities. Our experiments provide preliminary results indicating the
effectiveness of whitening low-dimensional item embeddings.
Related papers
- Loss-Free Machine Unlearning [51.34904967046097]
We present a machine unlearning approach that is both retraining- and label-free.
Retraining-free approaches often utilise Fisher information, which is derived from the loss and requires labelled data which may not be available.
We present an extension to the Selective Synaptic Dampening algorithm, substituting the diagonal of the Fisher information matrix for the gradient of the l2 norm of the model output to approximate sensitivity.
arXiv Detail & Related papers (2024-02-29T16:15:34Z) - Fast Dual-Regularized Autoencoder for Sparse Biological Data [65.268245109828]
We develop a shallow autoencoder for the dual neighborhood-regularized matrix completion problem.
We demonstrate the speed and accuracy advantage of our approach over the existing state-of-the-art in predicting drug-target interactions and drug-disease associations.
arXiv Detail & Related papers (2024-01-30T01:28:48Z) - It's Enough: Relaxing Diagonal Constraints in Linear Autoencoders for
Recommendation [4.8802420827610025]
This paper aims to theoretically understand the properties of two terms in linear autoencoders.
We propose simple-yet-effective linear autoencoder models using diagonal inequality constraints, called Relaxed Linear AutoEncoder (RLAE) and Relaxed Denoising Linear AutoEncoder (RDLAE)
Experimental results demonstrate that our models are comparable or superior to state-of-the-art linear and non-linear models on six benchmark datasets.
arXiv Detail & Related papers (2023-05-22T11:09:49Z) - Fundamental Limits of Two-layer Autoencoders, and Achieving Them with
Gradient Methods [91.54785981649228]
This paper focuses on non-linear two-layer autoencoders trained in the challenging proportional regime.
Our results characterize the minimizers of the population risk, and show that such minimizers are achieved by gradient methods.
For the special case of a sign activation function, our analysis establishes the fundamental limits for the lossy compression of Gaussian sources via (shallow) autoencoders.
arXiv Detail & Related papers (2022-12-27T12:37:34Z) - Efficient Nearest Neighbor Search for Cross-Encoder Models using Matrix
Factorization [60.91600465922932]
We present an approach that avoids the use of a dual-encoder for retrieval, relying solely on the cross-encoder.
Our approach provides test-time recall-vs-computational cost trade-offs superior to the current widely-used methods.
arXiv Detail & Related papers (2022-10-23T00:32:04Z) - On the Regularization of Autoencoders [14.46779433267854]
We show that the unsupervised setting by itself induces strong additional regularization, i.e., a severe reduction in the model-capacity of the learned autoencoder.
We derive that a deep nonlinear autoencoder cannot fit the training data more accurately than a linear autoencoder does if both models have the same dimensionality in their last layer.
We demonstrate that it is an accurate approximation across all model-ranks in our experiments on three well-known data sets.
arXiv Detail & Related papers (2021-10-21T18:28:25Z) - Unsupervised detection of mouse behavioural anomalies using two-stream
convolutional autoencoders [1.1236899956615454]
This paper explores the application of unsupervised learning to detecting anomalies in mouse video data.
The two models presented are a dual-stream, 3D convolutional autoencoder and a dual-stream, 2D convolutional autoencoder.
arXiv Detail & Related papers (2021-05-28T16:30:09Z) - Autoencoding Variational Autoencoder [56.05008520271406]
We study the implications of this behaviour on the learned representations and also the consequences of fixing it by introducing a notion of self consistency.
We show that encoders trained with our self-consistency approach lead to representations that are robust (insensitive) to perturbations in the input introduced by adversarial attacks.
arXiv Detail & Related papers (2020-12-07T14:16:14Z) - Dual Adversarial Auto-Encoders for Clustering [152.84443014554745]
We propose Dual Adversarial Auto-encoder (Dual-AAE) for unsupervised clustering.
By performing variational inference on the objective function of Dual-AAE, we derive a new reconstruction loss which can be optimized by training a pair of Auto-encoders.
Experiments on four benchmarks show that Dual-AAE achieves superior performance over state-of-the-art clustering methods.
arXiv Detail & Related papers (2020-08-23T13:16:34Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.