Cross-Lingual BERT Contextual Embedding Space Mapping with Isotropic and Isometric Conditions

• URL: http://arxiv.org/abs/2107.09186v1
• Date: Mon, 19 Jul 2021 22:57:36 GMT
• Title: Cross-Lingual BERT Contextual Embedding Space Mapping with Isotropic and Isometric Conditions
• Authors: Haoran Xu and Philipp Koehn
• Abstract summary: We investigate a context-aware and dictionary-free mapping approach by leveraging parallel corpora. Our findings unfold the tight relationship between isotropy, isometry, and isomorphism in normalized contextual embedding spaces.
• Score: 7.615096161060399
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Typically, a linearly orthogonal transformation mapping is learned by aligning static type-level embeddings to build a shared semantic space. In view of the analysis that contextual embeddings contain richer semantic features, we investigate a context-aware and dictionary-free mapping approach by leveraging parallel corpora. We illustrate that our contextual embedding space mapping significantly outperforms previous multilingual word embedding methods on the bilingual dictionary induction (BDI) task by providing a higher degree of isomorphism. To improve the quality of mapping, we also explore sense-level embeddings that are split from type-level representations, which can align spaces in a finer resolution and yield more precise mapping. Moreover, we reveal that contextual embedding spaces suffer from their natural properties -- anisotropy and anisometry. To mitigate these two problems, we introduce the iterative normalization algorithm as an imperative preprocessing step. Our findings unfold the tight relationship between isotropy, isometry, and isomorphism in normalized contextual embedding spaces.

Related papers

• Relative Representations: Topological and Geometric Perspectives [53.88896255693922]
  Relative representations are an established approach to zero-shot model stitching. We introduce a normalization procedure in the relative transformation, resulting in invariance to non-isotropic rescalings and permutations. Second, we propose to deploy topological densification when fine-tuning relative representations, a topological regularization loss encouraging clustering within classes.
  arXiv  Detail & Related papers  (2024-09-17T08:09:22Z)
• EmerDiff: Emerging Pixel-level Semantic Knowledge in Diffusion Models [52.3015009878545]
  We develop an image segmentor capable of generating fine-grained segmentation maps without any additional training. Our framework identifies semantic correspondences between image pixels and spatial locations of low-dimensional feature maps. In extensive experiments, the produced segmentation maps are demonstrated to be well delineated and capture detailed parts of the images.
  arXiv  Detail & Related papers  (2024-01-22T07:34:06Z)
• IsoVec: Controlling the Relative Isomorphism of Word Embedding Spaces [24.256732557154486]
  We address the root-cause of faulty cross-lingual mapping: that word embedding training resulted in the underlying spaces being non-isomorphic. We incorporate global measures of isomorphism directly into the Skip-gram loss function, successfully increasing the relative isomorphism of trained word embedding spaces.
  arXiv  Detail & Related papers  (2022-10-11T02:29:34Z)
• Graph Adaptive Semantic Transfer for Cross-domain Sentiment Classification [68.06496970320595]
  Cross-domain sentiment classification (CDSC) aims to use the transferable semantics learned from the source domain to predict the sentiment of reviews in the unlabeled target domain. We present Graph Adaptive Semantic Transfer (GAST) model, an adaptive syntactic graph embedding method that is able to learn domain-invariant semantics from both word sequences and syntactic graphs.
  arXiv  Detail & Related papers  (2022-05-18T07:47:01Z)
• Marginal Contrastive Correspondence for Guided Image Generation [58.0605433671196]
  Exemplar-based image translation establishes dense correspondences between a conditional input and an exemplar from two different domains. Existing work builds the cross-domain correspondences implicitly by minimizing feature-wise distances across the two domains. We design a Marginal Contrastive Learning Network (MCL-Net) that explores contrastive learning to learn domain-invariant features for realistic exemplar-based image translation.
  arXiv  Detail & Related papers  (2022-04-01T13:55:44Z)
• The Low-Dimensional Linear Geometry of Contextualized Word Representations [27.50785941238007]
  We study the linear geometry of contextualized word representations in ELMO and BERT. We show that a variety of linguistic features are encoded in low-dimensional subspaces.
  arXiv  Detail & Related papers  (2021-05-15T00:58:08Z)
• Zero-Shot Cross-Lingual Dependency Parsing through Contextual Embedding Transformation [7.615096161060399]
  Cross-lingual embedding space mapping is usually studied in static word-level embeddings. We investigate a contextual embedding alignment approach which is sense-level and dictionary-free. Experiments on zero-shot dependency parsing through the concept-shared space built by our embedding transformation substantially outperform state-of-the-art methods using multilingual embeddings.
  arXiv  Detail & Related papers  (2021-03-03T06:50:43Z)
• Introducing Orthogonal Constraint in Structural Probes [0.2538209532048867]
  We decompose a linear projection of language vector space into isomorphic space rotation and linear scaling directions. We experimentally show that our approach can be performed in a multitask setting.
  arXiv  Detail & Related papers  (2020-12-30T17:14:25Z)
• RatE: Relation-Adaptive Translating Embedding for Knowledge Graph Completion [51.64061146389754]
  We propose a relation-adaptive translation function built upon a novel weighted product in complex space. We then present our Relation-adaptive translating Embedding (RatE) approach to score each graph triple.
  arXiv  Detail & Related papers  (2020-10-10T01:30:30Z)
• A Comparative Study on Structural and Semantic Properties of Sentence Embeddings [77.34726150561087]
  We propose a set of experiments using a widely-used large-scale data set for relation extraction. We show that different embedding spaces have different degrees of strength for the structural and semantic properties. These results provide useful information for developing embedding-based relation extraction methods.
  arXiv  Detail & Related papers  (2020-09-23T15:45:32Z)
• Filtered Inner Product Projection for Crosslingual Embedding Alignment [28.72288652451881]
  Filtered Inner Product Projection (FIPP) is a method for mapping embeddings to a common representation space. FIPP is applicable even when the source and target embeddings are of differing dimensionalities. We show that our approach outperforms existing methods on the MUSE dataset for various language pairs.
  arXiv  Detail & Related papers  (2020-06-05T19:53:30Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.