IsoVec: Controlling the Relative Isomorphism of Word Embedding Spaces

• URL: http://arxiv.org/abs/2210.05098v3
• Date: Tue, 4 Jul 2023 18:32:01 GMT
• Title: IsoVec: Controlling the Relative Isomorphism of Word Embedding Spaces
• Authors: Kelly Marchisio, Neha Verma, Kevin Duh, Philipp Koehn
• Abstract summary: We address the root-cause of faulty cross-lingual mapping: that word embedding training resulted in the underlying spaces being non-isomorphic. We incorporate global measures of isomorphism directly into the Skip-gram loss function, successfully increasing the relative isomorphism of trained word embedding spaces.
• Score: 24.256732557154486
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The ability to extract high-quality translation dictionaries from monolingual word embedding spaces depends critically on the geometric similarity of the spaces -- their degree of "isomorphism." We address the root-cause of faulty cross-lingual mapping: that word embedding training resulted in the underlying spaces being non-isomorphic. We incorporate global measures of isomorphism directly into the Skip-gram loss function, successfully increasing the relative isomorphism of trained word embedding spaces and improving their ability to be mapped to a shared cross-lingual space. The result is improved bilingual lexicon induction in general data conditions, under domain mismatch, and with training algorithm dissimilarities. We release IsoVec at https://github.com/kellymarchisio/isovec.

Related papers

• GARI: Graph Attention for Relative Isomorphism of Arabic Word Embeddings [10.054788741823627]
  Lexical Induction (BLI) is a core challenge in NLP, it relies on the relative isomorphism of individual embedding spaces. Existing attempts aimed at controlling the relative isomorphism of different embedding spaces fail to incorporate the impact of semantically related words. We propose GARI that combines the distributional training objectives with multiple isomorphism losses guided by the graph attention network.
  arXiv  Detail & Related papers  (2023-10-19T18:08:22Z)
• GRI: Graph-based Relative Isomorphism of Word Embedding Spaces [10.984134369344117]
  Automated construction of bilingual dictionaries using monolingual embedding spaces is a core challenge in machine translation. Existing attempts aimed at controlling the relative isomorphism of different spaces fail to incorporate the impact of semantically related words in the training objective. We propose GRI that combines the distributional training objectives with attentive graph convolutions to unanimously consider the impact of semantically similar words.
  arXiv  Detail & Related papers  (2023-10-18T22:10:47Z)
• Robust Unsupervised Cross-Lingual Word Embedding using Domain Flow Interpolation [48.32604585839687]
  Previous adversarial approaches have shown promising results in inducing cross-lingual word embedding without parallel data. We propose to make use of a sequence of intermediate spaces for smooth bridging.
  arXiv  Detail & Related papers  (2022-10-07T04:37:47Z)
• Multilingual Extraction and Categorization of Lexical Collocations with Graph-aware Transformers [86.64972552583941]
  We put forward a sequence tagging BERT-based model enhanced with a graph-aware transformer architecture, which we evaluate on the task of collocation recognition in context. Our results suggest that explicitly encoding syntactic dependencies in the model architecture is helpful, and provide insights on differences in collocation typification in English, Spanish and French.
  arXiv  Detail & Related papers  (2022-05-23T16:47:37Z)
• Cross-Lingual BERT Contextual Embedding Space Mapping with Isotropic and Isometric Conditions [7.615096161060399]
  We investigate a context-aware and dictionary-free mapping approach by leveraging parallel corpora. Our findings unfold the tight relationship between isotropy, isometry, and isomorphism in normalized contextual embedding spaces.
  arXiv  Detail & Related papers  (2021-07-19T22:57:36Z)
• Zero-Shot Cross-Lingual Dependency Parsing through Contextual Embedding Transformation [7.615096161060399]
  Cross-lingual embedding space mapping is usually studied in static word-level embeddings. We investigate a contextual embedding alignment approach which is sense-level and dictionary-free. Experiments on zero-shot dependency parsing through the concept-shared space built by our embedding transformation substantially outperform state-of-the-art methods using multilingual embeddings.
  arXiv  Detail & Related papers  (2021-03-03T06:50:43Z)
• Simultaneous Semantic Alignment Network for Heterogeneous Domain Adaptation [67.37606333193357]
  We propose aSimultaneous Semantic Alignment Network (SSAN) to simultaneously exploit correlations among categories and align the centroids for each category across domains. By leveraging target pseudo-labels, a robust triplet-centroid alignment mechanism is explicitly applied to align feature representations for each category. Experiments on various HDA tasks across text-to-image, image-to-image and text-to-text successfully validate the superiority of our SSAN against state-of-the-art HDA methods.
  arXiv  Detail & Related papers  (2020-08-04T16:20:37Z)
• APo-VAE: Text Generation in Hyperbolic Space [116.11974607497986]
  In this paper, we investigate text generation in a hyperbolic latent space to learn continuous hierarchical representations. An Adrial Poincare Variversaational Autoencoder (APo-VAE) is presented, where both the prior and variational posterior of latent variables are defined over a Poincare ball via wrapped normal distributions. Experiments in language modeling and dialog-response generation tasks demonstrate the winning effectiveness of the proposed APo-VAE model.
  arXiv  Detail & Related papers  (2020-04-30T19:05:41Z)
• Are All Good Word Vector Spaces Isomorphic? [79.04509759167952]
  We show that variance in performance across language pairs is not only due to typological differences, but can mostly be attributed to the size of the monolingual resources available.
  arXiv  Detail & Related papers  (2020-04-08T15:49:19Z)
• Robust Cross-lingual Embeddings from Parallel Sentences [65.85468628136927]
  We propose a bilingual extension of the CBOW method which leverages sentence-aligned corpora to obtain robust cross-lingual word representations. Our approach significantly improves crosslingual sentence retrieval performance over all other approaches. It also achieves parity with a deep RNN method on a zero-shot cross-lingual document classification task.
  arXiv  Detail & Related papers  (2019-12-28T16:18:33Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.