Discrete Cosine Transform as Universal Sentence Encoder

• URL: http://arxiv.org/abs/2106.00934v1
• Date: Wed, 2 Jun 2021 04:43:54 GMT
• Title: Discrete Cosine Transform as Universal Sentence Encoder
• Authors: Nada Almarwani and Mona Diab
• Abstract summary: We use Discrete Cosine Transform (DCT) to generate universal sentence representation for different languages. The experimental results clearly show the superior effectiveness of DCT encoding.
• Score: 10.355894890759377
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Modern sentence encoders are used to generate dense vector representations that capture the underlying linguistic characteristics for a sequence of words, including phrases, sentences, or paragraphs. These kinds of representations are ideal for training a classifier for an end task such as sentiment analysis, question answering and text classification. Different models have been proposed to efficiently generate general purpose sentence representations to be used in pretraining protocols. While averaging is the most commonly used efficient sentence encoder, Discrete Cosine Transform (DCT) was recently proposed as an alternative that captures the underlying syntactic characteristics of a given text without compromising practical efficiency compared to averaging. However, as with most other sentence encoders, the DCT sentence encoder was only evaluated in English. To this end, we utilize DCT encoder to generate universal sentence representation for different languages such as German, French, Spanish and Russian. The experimental results clearly show the superior effectiveness of DCT encoding in which consistent performance improvements are achieved over strong baselines on multiple standardized datasets.

Related papers

• SenTest: Evaluating Robustness of Sentence Encoders [0.4194295877935868]
  This work focuses on evaluating the robustness of the sentence encoders. We employ several adversarial attacks to evaluate its robustness. The results of the experiments strongly undermine the robustness of sentence encoders.
  arXiv  Detail & Related papers  (2023-11-29T15:21:35Z)
• Sub-Sentence Encoder: Contrastive Learning of Propositional Semantic Representations [102.05351905494277]
  Sub-sentence encoder is a contrastively-learned contextual embedding model for fine-grained semantic representation of text. We show that sub-sentence encoders keep the same level of inference cost and space complexity compared to sentence encoders.
  arXiv  Detail & Related papers  (2023-11-07T20:38:30Z)
• On the Suitability of Representations for Quality Diversity Optimization of Shapes [77.34726150561087]
  The representation, or encoding, utilized in evolutionary algorithms has a substantial effect on their performance. This study compares the impact of several representations, including direct encoding, a dictionary-based representation, parametric encoding, compositional pattern producing networks, and cellular automata, on the generation of voxelized meshes.
  arXiv  Detail & Related papers  (2023-04-07T07:34:23Z)
• Hierarchical Sketch Induction for Paraphrase Generation [79.87892048285819]
  We introduce Hierarchical Refinement Quantized Variational Autoencoders (HRQ-VAE), a method for learning decompositions of dense encodings. We use HRQ-VAE to encode the syntactic form of an input sentence as a path through the hierarchy, allowing us to more easily predict syntactic sketches at test time.
  arXiv  Detail & Related papers  (2022-03-07T15:28:36Z)
• Sentence Bottleneck Autoencoders from Transformer Language Models [53.350633961266375]
  We build a sentence-level autoencoder from a pretrained, frozen transformer language model. We adapt the masked language modeling objective as a generative, denoising one, while only training a sentence bottleneck and a single-layer modified transformer decoder. We demonstrate that the sentence representations discovered by our model achieve better quality than previous methods that extract representations from pretrained transformers on text similarity tasks, style transfer, and single-sentence classification tasks in the GLUE benchmark, while using fewer parameters than large pretrained models.
  arXiv  Detail & Related papers  (2021-08-31T19:39:55Z)
• Transition based Graph Decoder for Neural Machine Translation [41.7284715234202]
  We propose a general Transformer-based approach for tree and graph decoding based on generating a sequence of transitions. We show improved performance over the standard Transformer decoder, as well as over ablated versions of the model.
  arXiv  Detail & Related papers  (2021-01-29T15:20:45Z)
• Adapting Pretrained Transformer to Lattices for Spoken Language Understanding [39.50831917042577]
  It is shown that encoding lattices as opposed to 1-best results generated by automatic speech recognizer (ASR) boosts the performance of spoken language understanding (SLU) This paper aims at adapting pretrained transformers to lattice inputs in order to perform understanding tasks specifically for spoken language.
  arXiv  Detail & Related papers  (2020-11-02T07:14:34Z)
• Cross-Thought for Sentence Encoder Pre-training [89.32270059777025]
  Cross-Thought is a novel approach to pre-training sequence encoder. We train a Transformer-based sequence encoder over a large set of short sequences. Experiments on question answering and textual entailment tasks demonstrate that our pre-trained encoder can outperform state-of-the-art encoders.
  arXiv  Detail & Related papers  (2020-10-07T21:02:41Z)
• Discovering Useful Sentence Representations from Large Pretrained Language Models [8.212920842986689]
  We explore the question of whether pretrained language models can be adapted to be used as universal decoders. For large transformer-based language models trained on vast amounts of English text, we investigate whether such representations can be easily discovered. We present and compare three representation injection techniques for transformer-based models and three accompanying methods which map sentences to and from this representation space.
  arXiv  Detail & Related papers  (2020-08-20T16:03:51Z)
• Improve Variational Autoencoder for Text Generationwith Discrete Latent Bottleneck [52.08901549360262]
  Variational autoencoders (VAEs) are essential tools in end-to-end representation learning. VAEs tend to ignore latent variables with a strong auto-regressive decoder. We propose a principled approach to enforce an implicit latent feature matching in a more compact latent space.
  arXiv  Detail & Related papers  (2020-04-22T14:41:37Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.