Structured Tree Alignment for Evaluation of (Speech) Constituency Parsing

• URL: http://arxiv.org/abs/2402.13433v2
• Date: Wed, 19 Jun 2024 18:33:17 GMT
• Title: Structured Tree Alignment for Evaluation of (Speech) Constituency Parsing
• Authors: Freda Shi, Kevin Gimpel, Karen Livescu,
• Abstract summary: We present the structured average intersection-over-union ratio (STRUCT-IOU), a similarity metric between constituency parse trees motivated by the problem of evaluating speechs. To compute the metric, we project the ground-truth parse tree to the speech domain by forced alignment, align the projected ground-truth constituents with the predicted ones under certain structured constraints, and calculate the average IOU score across all aligned constituent pairs.
• Score: 43.758912958903494
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We present the structured average intersection-over-union ratio (STRUCT-IOU), a similarity metric between constituency parse trees motivated by the problem of evaluating speech parsers. STRUCT-IOU enables comparison between a constituency parse tree (over automatically recognized spoken word boundaries) with the ground-truth parse (over written words). To compute the metric, we project the ground-truth parse tree to the speech domain by forced alignment, align the projected ground-truth constituents with the predicted ones under certain structured constraints, and calculate the average IOU score across all aligned constituent pairs. STRUCT-IOU takes word boundaries into account and overcomes the challenge that the predicted words and ground truth may not have perfect one-to-one correspondence. Extending to the evaluation of text constituency parsing, we demonstrate that STRUCT-IOU can address token-mismatch issues, and shows higher tolerance to syntactically plausible parses than PARSEVAL (Black et al., 1991).

Related papers

• Growing Trees on Sounds: Assessing Strategies for End-to-End Dependency Parsing of Speech [8.550564152063522]
  We report on a set of experiments aiming at assessing the performance of two parsing paradigms on speech parsing. We perform this evaluation on a large treebank of spoken French, featuring realistic spontaneous conversations. Our findings show that (i) the graph based approach obtains better results across the board (ii) parsing directly from speech outperforms a pipeline approach, despite having 30% fewer parameters.
  arXiv  Detail & Related papers  (2024-06-18T13:46:10Z)
• Cascading and Direct Approaches to Unsupervised Constituency Parsing on Spoken Sentences [67.37544997614646]
  We present the first study on unsupervised spoken constituency parsing. The goal is to determine the spoken sentences' hierarchical syntactic structure in the form of constituency parse trees. We show that accurate segmentation alone may be sufficient to parse spoken sentences accurately.
  arXiv  Detail & Related papers  (2023-03-15T17:57:22Z)
• CPTAM: Constituency Parse Tree Aggregation Method [6.011216641982612]
  This paper adopts the truth discovery idea to aggregate constituency parse trees from different distances. We formulate the constituency parse tree aggregation problem in two steps, structure aggregation and constituent label aggregation. Experiments are conducted on benchmark datasets in different languages and domains.
  arXiv  Detail & Related papers  (2022-01-19T23:05:37Z)
• Contextualized Semantic Distance between Highly Overlapped Texts [85.1541170468617]
  Overlapping frequently occurs in paired texts in natural language processing tasks like text editing and semantic similarity evaluation. This paper aims to address the issue with a mask-and-predict strategy. We take the words in the longest common sequence as neighboring words and use masked language modeling (MLM) to predict the distributions on their positions. Experiments on Semantic Textual Similarity show NDD to be more sensitive to various semantic differences, especially on highly overlapped paired texts.
  arXiv  Detail & Related papers  (2021-10-04T03:59:15Z)
• A Conditional Splitting Framework for Efficient Constituency Parsing [14.548146390081778]
  We introduce a generic seq2seq parsing framework that casts constituency parsing problems (syntactic and discourse parsing) into a series of conditional splitting decisions. Our parsing model estimates the conditional probability distribution of possible splitting points in a given text span and supports efficient top-down decoding. For discourse analysis we show that in our formulation, discourse segmentation can be framed as a special case of parsing.
  arXiv  Detail & Related papers  (2021-06-30T00:36:34Z)
• RST Parsing from Scratch [14.548146390081778]
  We introduce a novel end-to-end formulation of document-level discourse parsing in the Rhetorical Structure Theory (RST) framework. Our framework facilitates discourse parsing from scratch without requiring discourse segmentation as a prerequisite. Our unified parsing model adopts a beam search to decode the best tree structure by searching through a space of high-scoring trees.
  arXiv  Detail & Related papers  (2021-05-23T06:19:38Z)
• Syntactic representation learning for neural network based TTS with syntactic parse tree traversal [49.05471750563229]
  We propose a syntactic representation learning method based on syntactic parse tree to automatically utilize the syntactic structure information. Experimental results demonstrate the effectiveness of our proposed approach. For sentences with multiple syntactic parse trees, prosodic differences can be clearly perceived from the synthesized speeches.
  arXiv  Detail & Related papers  (2020-12-13T05:52:07Z)
• Unsupervised Parsing via Constituency Tests [49.42244463346612]
  We propose a method for unsupervised parsing based on the linguistic notion of a constituency test. To produce a tree given a sentence, we score each span by aggregating its constituency test judgments, and we choose the binary tree with the highest total score. The refined model achieves 62.8 F1 on the Penn Treebank test set, an absolute improvement of 7.6 points over the previous best published result.
  arXiv  Detail & Related papers  (2020-10-07T04:05:01Z)
• Pareto Probing: Trading Off Accuracy for Complexity [87.09294772742737]
  We argue for a probe metric that reflects the fundamental trade-off between probe complexity and performance. Our experiments with dependency parsing reveal a wide gap in syntactic knowledge between contextual and non-contextual representations.
  arXiv  Detail & Related papers  (2020-10-05T17:27:31Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.