Discrete representations in neural models of spoken language

• URL: http://arxiv.org/abs/2105.05582v1
• Date: Wed, 12 May 2021 11:02:02 GMT
• Title: Discrete representations in neural models of spoken language
• Authors: Bertrand Higy, Lieke Gelderloos, Afra Alishahi and Grzegorz Chrupa{\l}a
• Abstract summary: We compare the merits of four commonly used metrics in the context of weakly supervised models of spoken language. We find that the different evaluation metrics can give inconsistent results.
• Score: 56.29049879393466
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The distributed and continuous representations used by neural networks are at odds with representations employed in linguistics, which are typically symbolic. Vector quantization has been proposed as a way to induce discrete neural representations that are closer in nature to their linguistic counterparts. However, it is not clear which metrics are the best-suited to analyze such discrete representations. We compare the merits of four commonly used metrics in the context of weakly supervised models of spoken language. We perform a systematic analysis of the impact of (i) architectural choices, (ii) the learning objective and training dataset, and (iii) the evaluation metric. We find that the different evaluation metrics can give inconsistent results. In particular, we find that the use of minimal pairs of phoneme triples as stimuli during evaluation disadvantages larger embeddings, unlike metrics applied to complete utterances.

Related papers

• Enhancing Idiomatic Representation in Multiple Languages via an Adaptive Contrastive Triplet Loss [9.807885676930308]
  We propose an approach to model idiomaticity using a triplet loss that incorporates the asymmetric contribution of components words to an idiomatic meaning for training language models. Our proposed method is evaluated on a SemEval challenge and outperforms previous alternatives significantly in many metrics.
  arXiv  Detail & Related papers  (2024-06-21T14:21:41Z)
• How well do distributed representations convey contextual lexical semantics: a Thesis Proposal [3.3585951129432323]
  In this thesis, we examine the efficacy of distributed representations from modern neural networks in encoding lexical meaning. We identify four sources of ambiguity based on the relatedness and similarity of meanings influenced by context. We then aim to evaluate these sources by collecting or constructing multilingual datasets, leveraging various language models, and employing linguistic analysis tools.
  arXiv  Detail & Related papers  (2024-06-02T14:08:51Z)
• Constructing Word-Context-Coupled Space Aligned with Associative Knowledge Relations for Interpretable Language Modeling [0.0]
  The black-box structure of the deep neural network in pre-trained language models seriously limits the interpretability of the language modeling process. A Word-Context-Coupled Space (W2CSpace) is proposed by introducing the alignment processing between uninterpretable neural representation and interpretable statistical logic. Our language model can achieve better performance and highly credible interpretable ability compared to related state-of-the-art methods.
  arXiv  Detail & Related papers  (2023-05-19T09:26:02Z)
• A Latent-Variable Model for Intrinsic Probing [93.62808331764072]
  We propose a novel latent-variable formulation for constructing intrinsic probes. We find empirical evidence that pre-trained representations develop a cross-lingually entangled notion of morphosyntax.
  arXiv  Detail & Related papers  (2022-01-20T15:01:12Z)
• Idiomatic Expression Identification using Semantic Compatibility [8.355785779504869]
  We study the task of detecting whether a sentence has an idiomatic expression and localizing it. We propose a multi-stage neural architecture with the attention flow mechanism for identifying these expressions. A salient feature of the model is its ability to identify idioms unseen during training with gains from 1.4% to 30.8% over competitive baselines.
  arXiv  Detail & Related papers  (2021-10-19T15:44:28Z)
• On The Ingredients of an Effective Zero-shot Semantic Parser [95.01623036661468]
  We analyze zero-shot learning by paraphrasing training examples of canonical utterances and programs from a grammar. We propose bridging these gaps using improved grammars, stronger paraphrasers, and efficient learning methods. Our model achieves strong performance on two semantic parsing benchmarks (Scholar, Geo) with zero labeled data.
  arXiv  Detail & Related papers  (2021-10-15T21:41:16Z)
• Preliminary study on using vector quantization latent spaces for TTS/VC systems with consistent performance [55.10864476206503]
  We investigate the use of quantized vectors to model the latent linguistic embedding. By enforcing different policies over the latent spaces in the training, we are able to obtain a latent linguistic embedding. Our experiments show that the voice cloning system built with vector quantization has only a small degradation in terms of perceptive evaluations.
  arXiv  Detail & Related papers  (2021-06-25T07:51:35Z)
• Infusing Finetuning with Semantic Dependencies [62.37697048781823]
  We show that, unlike syntax, semantics is not brought to the surface by today's pretrained models. We then use convolutional graph encoders to explicitly incorporate semantic parses into task-specific finetuning.
  arXiv  Detail & Related papers  (2020-12-10T01:27:24Z)
• Mechanisms for Handling Nested Dependencies in Neural-Network Language Models and Humans [75.15855405318855]
  We studied whether a modern artificial neural network trained with "deep learning" methods mimics a central aspect of human sentence processing. Although the network was solely trained to predict the next word in a large corpus, analysis showed the emergence of specialized units that successfully handled local and long-distance syntactic agreement. We tested the model's predictions in a behavioral experiment where humans detected violations in number agreement in sentences with systematic variations in the singular/plural status of multiple nouns.
  arXiv  Detail & Related papers  (2020-06-19T12:00:05Z)
• Do Neural Models Learn Systematicity of Monotonicity Inference in Natural Language? [41.649440404203595]
  We introduce a method for evaluating whether neural models can learn systematicity of monotonicity inference in natural language. We consider four aspects of monotonicity inferences and test whether the models can systematically interpret lexical and logical phenomena on different training/test splits.
  arXiv  Detail & Related papers  (2020-04-30T14:48:39Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.