Rethinking Positional Encoding
- URL: http://arxiv.org/abs/2107.02561v1
- Date: Tue, 6 Jul 2021 12:04:04 GMT
- Title: Rethinking Positional Encoding
- Authors: Jianqiao Zheng, Sameera Ramasinghe, Simon Lucey
- Abstract summary: We show that alternative non-Fourier embedding functions can indeed be used for positional encoding.
We show that their performance is entirely determined by a trade-off between the stable rank of the embedded matrix and the distance preservation between embedded coordinates.
We present a more general theory to analyze positional encoding in terms of shifted basis functions.
- Score: 31.80055086317266
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: It is well noted that coordinate based MLPs benefit greatly -- in terms of
preserving high-frequency information -- through the encoding of coordinate
positions as an array of Fourier features. Hitherto, the rationale for the
effectiveness of these positional encodings has been solely studied through a
Fourier lens. In this paper, we strive to broaden this understanding by showing
that alternative non-Fourier embedding functions can indeed be used for
positional encoding. Moreover, we show that their performance is entirely
determined by a trade-off between the stable rank of the embedded matrix and
the distance preservation between embedded coordinates. We further establish
that the now ubiquitous Fourier feature mapping of position is a special case
that fulfills these conditions. Consequently, we present a more general theory
to analyze positional encoding in terms of shifted basis functions. To this
end, we develop the necessary theoretical formulae and empirically verify that
our theoretical claims hold in practice. Codes available at
https://github.com/osiriszjq/Rethinking-positional-encoding.
Related papers
- Improving Transformers using Faithful Positional Encoding [55.30212768657544]
We propose a new positional encoding method for a neural network architecture called the Transformer.
Unlike the standard sinusoidal positional encoding, our approach has a guarantee of not losing information about the positional order of the input sequence.
arXiv Detail & Related papers (2024-05-15T03:17:30Z) - Positional Encoding Helps Recurrent Neural Networks Handle a Large Vocabulary [1.4594704809280983]
Positional encoding is a high-dimensional representation of time indices on input data.
RNNs can encode the temporal information of data points on their own, rendering their use of positional encoding seemingly redundant/unnecessary.
arXiv Detail & Related papers (2024-01-31T23:32:20Z) - Coordinate Quantized Neural Implicit Representations for Multi-view
Reconstruction [28.910183274743872]
We introduce neural implicit representations with quantized coordinates, which reduces the uncertainty and ambiguity in the field during optimization.
We use discrete coordinates and their positional encodings to learn implicit functions through volume rendering.
Our evaluations under the widely used benchmarks show our superiority over the state-of-the-art.
arXiv Detail & Related papers (2023-08-21T20:27:33Z) - Generalized Laplacian Positional Encoding for Graph Representation
Learning [15.723716197068574]
Graph neural networks (GNNs) are the primary tool for processing graph-structured data.
Recent works have adapted the idea of positional encodings to graph data.
This paper draws inspiration from the recent success of Laplacian-based positional encoding.
arXiv Detail & Related papers (2022-10-28T07:21:57Z) - Trading Positional Complexity vs. Deepness in Coordinate Networks [33.90893096003318]
We show that alternative non-Fourier embedding functions can indeed be used for positional encoding.
Their performance is entirely determined by a trade-off between the stable rank of the embedded matrix and the distance preservation between embedded coordinates.
We argue that employing a more complex positional encoding -- that scales exponentially with the number of modes -- requires only a linear (rather than deep) coordinate function to achieve comparable performance.
arXiv Detail & Related papers (2022-05-18T15:17:09Z) - Dense Coding with Locality Restriction for Decoder: Quantum Encoders vs.
Super-Quantum Encoders [67.12391801199688]
We investigate dense coding by imposing various locality restrictions to our decoder.
In this task, the sender Alice and the receiver Bob share an entangled state.
arXiv Detail & Related papers (2021-09-26T07:29:54Z) - Learnable Fourier Features for Multi-DimensionalSpatial Positional
Encoding [96.9752763607738]
We propose a novel positional encoding method based on learnable Fourier features.
Our experiments show that our learnable feature representation for multi-dimensional positional encoding outperforms existing methods.
arXiv Detail & Related papers (2021-06-05T04:40:18Z) - Positional Encoding as Spatial Inductive Bias in GANs [97.6622154941448]
SinGAN shows impressive capability in learning internal patch distribution despite its limited effective receptive field.
In this work, we show that such capability, to a large extent, is brought by the implicit positional encoding when using zero padding in the generators.
We propose a new multi-scale training strategy and demonstrate its effectiveness in the state-of-the-art unconditional generator StyleGAN2.
arXiv Detail & Related papers (2020-12-09T18:27:16Z) - MetaSDF: Meta-learning Signed Distance Functions [85.81290552559817]
Generalizing across shapes with neural implicit representations amounts to learning priors over the respective function space.
We formalize learning of a shape space as a meta-learning problem and leverage gradient-based meta-learning algorithms to solve this task.
arXiv Detail & Related papers (2020-06-17T05:14:53Z) - A Transformer-based Approach for Source Code Summarization [86.08359401867577]
We learn code representation for summarization by modeling the pairwise relationship between code tokens.
We show that despite the approach is simple, it outperforms the state-of-the-art techniques by a significant margin.
arXiv Detail & Related papers (2020-05-01T23:29:36Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.