An enhanced Tree-LSTM architecture for sentence semantic modeling using typed dependencies

• URL: http://arxiv.org/abs/2002.07775v2
• Date: Fri, 25 Sep 2020 09:45:26 GMT
• Title: An enhanced Tree-LSTM architecture for sentence semantic modeling using typed dependencies
• Authors: Jeena Kleenankandy, K. A. Abdul Nazeer (Department of Computer Science and Engineering, National Institute of Technology Calicut, Kerala, India)
• Abstract summary: Tree-based Long short term memory (LSTM) network has become state-of-the-art for modeling the meaning of language texts. This paper proposes an enhanced LSTM architecture, called relation gated LSTM, which can model the relationship between two inputs of a sequence. We also introduce a Tree-LSTM model called Typed Dependency Tree-LSTM that uses the sentence dependency parse structure and the dependency type to embed sentence meaning into a dense vector.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Tree-based Long short term memory (LSTM) network has become state-of-the-art for modeling the meaning of language texts as they can effectively exploit the grammatical syntax and thereby non-linear dependencies among words of the sentence. However, most of these models cannot recognize the difference in meaning caused by a change in semantic roles of words or phrases because they do not acknowledge the type of grammatical relations, also known as typed dependencies, in sentence structure. This paper proposes an enhanced LSTM architecture, called relation gated LSTM, which can model the relationship between two inputs of a sequence using a control input. We also introduce a Tree-LSTM model called Typed Dependency Tree-LSTM that uses the sentence dependency parse structure as well as the dependency type to embed sentence meaning into a dense vector. The proposed model outperformed its type-unaware counterpart in two typical NLP tasks - Semantic Relatedness Scoring and Sentiment Analysis, in a lesser number of training epochs. The results were comparable or competitive with other state-of-the-art models. Qualitative analysis showed that changes in the voice of sentences had little effect on the model's predicted scores, while changes in nominal (noun) words had a more significant impact. The model recognized subtle semantic relationships in sentence pairs. The magnitudes of learned typed dependencies embeddings were also in agreement with human intuitions. The research findings imply the significance of grammatical relations in sentence modeling. The proposed models would serve as a base for future researches in this direction.

Related papers

• Syntax and Semantics Meet in the "Middle": Probing the Syntax-Semantics Interface of LMs Through Agentivity [68.8204255655161]
  We present the semantic notion of agentivity as a case study for probing such interactions. This suggests LMs may potentially serve as more useful tools for linguistic annotation, theory testing, and discovery.
  arXiv  Detail & Related papers  (2023-05-29T16:24:01Z)
• Recognizing semantic relation in sentence pairs using Tree-RNNs and Typed dependencies [0.0]
  This work proposes an improvement to Dependency Tree-RNN (DT-RNN) using the grammatical relationship type identified in the dependency parse. Experiments on semantic relatedness scoring (SRS) and recognizing textual entailment (RTE) in sentence pairs using SICK dataset show encouraging results.
  arXiv  Detail & Related papers  (2022-01-13T06:59:27Z)
• Introducing Syntactic Structures into Target Opinion Word Extraction with Deep Learning [89.64620296557177]
  We propose to incorporate the syntactic structures of the sentences into the deep learning models for targeted opinion word extraction. We also introduce a novel regularization technique to improve the performance of the deep learning models. The proposed model is extensively analyzed and achieves the state-of-the-art performance on four benchmark datasets.
  arXiv  Detail & Related papers  (2020-10-26T07:13:17Z)
• High-order Semantic Role Labeling [86.29371274587146]
  This paper introduces a high-order graph structure for the neural semantic role labeling model. It enables the model to explicitly consider not only the isolated predicate-argument pairs but also the interaction between the predicate-argument pairs. Experimental results on 7 languages of the CoNLL-2009 benchmark show that the high-order structural learning techniques are beneficial to the strong performing SRL models.
  arXiv  Detail & Related papers  (2020-10-09T15:33:54Z)
• Multi-timescale Representation Learning in LSTM Language Models [69.98840820213937]
  Language models must capture statistical dependencies between words at timescales ranging from very short to very long. We derived a theory for how the memory gating mechanism in long short-term memory language models can capture power law decay. Experiments showed that LSTM language models trained on natural English text learn to approximate this theoretical distribution.
  arXiv  Detail & Related papers  (2020-09-27T02:13:38Z)
• A Comparative Study of Lexical Substitution Approaches based on Neural Language Models [117.96628873753123]
  We present a large-scale comparative study of popular neural language and masked language models. We show that already competitive results achieved by SOTA LMs/MLMs can be further improved if information about the target word is injected properly.
  arXiv  Detail & Related papers  (2020-05-29T18:43:22Z)
• Exploiting Syntactic Structure for Better Language Modeling: A Syntactic Distance Approach [78.77265671634454]
  We make use of a multi-task objective, i.e., the models simultaneously predict words as well as ground truth parse trees in a form called "syntactic distances" Experimental results on the Penn Treebank and Chinese Treebank datasets show that when ground truth parse trees are provided as additional training signals, the model is able to achieve lower perplexity and induce trees with better quality.
  arXiv  Detail & Related papers  (2020-05-12T15:35:00Z)
• Influence Paths for Characterizing Subject-Verb Number Agreement in LSTM Language Models [22.826154706036995]
  LSTM-based recurrent neural networks are the state-of-the-art for many natural language processing (NLP) tasks. Lacking this understanding, the generality of LSTM performance on this task and their suitability for related tasks remains uncertain. We introduce *influence paths*, a causal account of structural properties as carried by paths across gates and neurons of a recurrent neural network.
  arXiv  Detail & Related papers  (2020-05-03T21:10:31Z)
• Attribution Analysis of Grammatical Dependencies in LSTMs [0.043512163406551986]
  LSTM language models have been shown to capture syntax-sensitive grammatical dependencies with a high degree of accuracy. We show that LSTM performance on number agreement is directly correlated with the model's ability to distinguish subjects from other nouns. Our results suggest that LSTM language models are able to infer robust representations of syntactic dependencies.
  arXiv  Detail & Related papers  (2020-04-30T19:19:37Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.