Bridging Continuous and Discrete Spaces: Interpretable Sentence Representation Learning via Compositional Operations

• URL: http://arxiv.org/abs/2305.14599v2
• Date: Sun, 5 Nov 2023 06:03:59 GMT
• Title: Bridging Continuous and Discrete Spaces: Interpretable Sentence Representation Learning via Compositional Operations
• Authors: James Y. Huang, Wenlin Yao, Kaiqiang Song, Hongming Zhang, Muhao Chen, Dong Yu
• Abstract summary: It is unclear whether the compositional semantics of sentences can be directly reflected as compositional operations in the embedding space. We propose InterSent, an end-to-end framework for learning interpretable sentence embeddings.
• Score: 80.45474362071236
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Traditional sentence embedding models encode sentences into vector representations to capture useful properties such as the semantic similarity between sentences. However, in addition to similarity, sentence semantics can also be interpreted via compositional operations such as sentence fusion or difference. It is unclear whether the compositional semantics of sentences can be directly reflected as compositional operations in the embedding space. To more effectively bridge the continuous embedding and discrete text spaces, we explore the plausibility of incorporating various compositional properties into the sentence embedding space that allows us to interpret embedding transformations as compositional sentence operations. We propose InterSent, an end-to-end framework for learning interpretable sentence embeddings that supports compositional sentence operations in the embedding space. Our method optimizes operator networks and a bottleneck encoder-decoder model to produce meaningful and interpretable sentence embeddings. Experimental results demonstrate that our method significantly improves the interpretability of sentence embeddings on four textual generation tasks over existing approaches while maintaining strong performance on traditional semantic similarity tasks.

Related papers

• Sub-Sentence Encoder: Contrastive Learning of Propositional Semantic Representations [102.05351905494277]
  Sub-sentence encoder is a contrastively-learned contextual embedding model for fine-grained semantic representation of text. We show that sub-sentence encoders keep the same level of inference cost and space complexity compared to sentence encoders.
  arXiv  Detail & Related papers  (2023-11-07T20:38:30Z)
• Relational Sentence Embedding for Flexible Semantic Matching [86.21393054423355]
  We present Sentence Embedding (RSE), a new paradigm to discover further the potential of sentence embeddings. RSE is effective and flexible in modeling sentence relations and outperforms a series of state-of-the-art embedding methods.
  arXiv  Detail & Related papers  (2022-12-17T05:25:17Z)
• Subspace Representations for Soft Set Operations and Sentence Similarities [17.52824249186434]
  We realize the representation of word sets and corresponding set operations within pre-trained word embedding spaces. By grounding our approach in the linear subspaces, we enable efficient computation of various set operations. We show that our subspace-based set operations consistently outperform vector-based ones in both sentence similarity and set retrieval tasks.
  arXiv  Detail & Related papers  (2022-10-24T08:34:10Z)
• Towards Structure-aware Paraphrase Identification with Phrase Alignment Using Sentence Encoders [4.254099382808598]
  We propose to combine sentence encoders with an alignment component by representing each sentence as a list of predicate-argument spans. Empirical results show that the alignment component brings in both improved performance and interpretability for various sentence encoders.
  arXiv  Detail & Related papers  (2022-10-11T09:52:52Z)
• A Sentence is Worth 128 Pseudo Tokens: A Semantic-Aware Contrastive Learning Framework for Sentence Embeddings [28.046786376565123]
  We propose a semantics-aware contrastive learning framework for sentence embeddings, termed Pseudo-Token BERT (PT-BERT) We exploit the pseudo-token space (i.e., latent semantic space) representation of a sentence while eliminating the impact of superficial features such as sentence length and syntax. Our model outperforms the state-of-the-art baselines on six standard semantic textual similarity (STS) tasks.
  arXiv  Detail & Related papers  (2022-03-11T12:29:22Z)
• Clustering and Network Analysis for the Embedding Spaces of Sentences and Sub-Sentences [69.3939291118954]
  This paper reports research on a set of comprehensive clustering and network analyses targeting sentence and sub-sentence embedding spaces. Results show that one method generates the most clusterable embeddings. In general, the embeddings of span sub-sentences have better clustering properties than the original sentences.
  arXiv  Detail & Related papers  (2021-10-02T00:47:35Z)
• Unsupervised Distillation of Syntactic Information from Contextualized Word Representations [62.230491683411536]
  We tackle the task of unsupervised disentanglement between semantics and structure in neural language representations. To this end, we automatically generate groups of sentences which are structurally similar but semantically different. We demonstrate that our transformation clusters vectors in space by structural properties, rather than by lexical semantics.
  arXiv  Detail & Related papers  (2020-10-11T15:13:18Z)
• A Comparative Study on Structural and Semantic Properties of Sentence Embeddings [77.34726150561087]
  We propose a set of experiments using a widely-used large-scale data set for relation extraction. We show that different embedding spaces have different degrees of strength for the structural and semantic properties. These results provide useful information for developing embedding-based relation extraction methods.
  arXiv  Detail & Related papers  (2020-09-23T15:45:32Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.