SHAPE: Shifted Absolute Position Embedding for Transformers

• URL: http://arxiv.org/abs/2109.05644v1
• Date: Mon, 13 Sep 2021 00:10:02 GMT
• Title: SHAPE: Shifted Absolute Position Embedding for Transformers
• Authors: Shun Kiyono, Sosuke Kobayashi, Jun Suzuki, Kentaro Inui
• Abstract summary: Existing position representations suffer from a lack of generalization to test data with unseen lengths or high computational cost. We investigate shifted absolute position embedding (SHAPE) to address both issues.
• Score: 59.03597635990196
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Position representation is crucial for building position-aware representations in Transformers. Existing position representations suffer from a lack of generalization to test data with unseen lengths or high computational cost. We investigate shifted absolute position embedding (SHAPE) to address both issues. The basic idea of SHAPE is to achieve shift invariance, which is a key property of recent successful position representations, by randomly shifting absolute positions during training. We demonstrate that SHAPE is empirically comparable to its counterpart while being simpler and faster.

Related papers

• EipFormer: Emphasizing Instance Positions in 3D Instance Segmentation [51.996943482875366]
  We present a novel Transformer-based architecture, EipFormer, which comprises progressive aggregation and dual position embedding. EipFormer achieves superior or comparable performance compared to state-of-the-art approaches.
  arXiv  Detail & Related papers  (2023-12-09T16:08:47Z)
• DropPos: Pre-Training Vision Transformers by Reconstructing Dropped Positions [63.61970125369834]
  We present DropPos, a novel pretext task designed to reconstruct Dropped Positions. The code is publicly available at https://github.com/Haochen-Wang409/DropPos.
  arXiv  Detail & Related papers  (2023-09-07T09:12:02Z)
• Open-World Pose Transfer via Sequential Test-Time Adaption [92.67291699304992]
  A typical pose transfer framework usually employs representative datasets to train a discriminative model. Test-time adaption (TTA) offers a feasible solution for OOD data by using a pre-trained model that learns essential features with self-supervision. In our experiment, we first show that pose transfer can be applied to open-world applications, including Tiktok reenactment and celebrity motion synthesis.
  arXiv  Detail & Related papers  (2023-03-20T09:01:23Z)
• The Curious Case of Absolute Position Embeddings [65.13827063579728]
  Transformer language models encode the notion of word order using positional information. In natural language, it is not absolute position that matters, but relative position, and the extent to which APEs can capture this type of information has not been investigated. We observe that models trained with APE over-rely on positional information to the point that they break-down when subjected to sentences with shifted position information.
  arXiv  Detail & Related papers  (2022-10-23T00:00:04Z)
• SSP-Pose: Symmetry-Aware Shape Prior Deformation for Direct Category-Level Object Pose Estimation [77.88624073105768]
  Category-level pose estimation is a challenging problem due to intra-class shape variations. We propose an end-to-end trainable network SSP-Pose for category-level pose estimation. SSP-Pose produces superior performance compared with competitors with a real-time inference speed at about 25Hz.
  arXiv  Detail & Related papers  (2022-08-13T14:37:31Z)
• CAPE: Encoding Relative Positions with Continuous Augmented Positional Embeddings [33.87449556591022]
  We propose an augmentation-based approach (CAPE) for absolute positional embeddings. CAPE keeps the advantages of both absolute (simplicity and speed) and relative position embeddings (better generalization)
  arXiv  Detail & Related papers  (2021-06-06T14:54:55Z)
• What Do Position Embeddings Learn? An Empirical Study of Pre-Trained Language Model Positional Encoding [42.011175069706816]
  This paper focuses on providing a new insight of pre-trained position embeddings through feature-level analysis and empirical experiments on most of iconic NLP tasks. It is believed that our experimental results can guide the future work to choose the suitable positional encoding function for specific tasks given the application property.
  arXiv  Detail & Related papers  (2020-10-10T05:03:14Z)
• Improve Transformer Models with Better Relative Position Embeddings [18.59434691153783]
  Transformer architectures rely on explicit position encodings to preserve a notion of word order. We argue that existing work does not fully utilize position information. We propose new techniques that encourage increased interaction between query, key and relative position embeddings.
  arXiv  Detail & Related papers  (2020-09-28T22:18:58Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.