How Good is Automatic Segmentation as a Multimodal Discourse Annotation Aid?

• URL: http://arxiv.org/abs/2305.17350v1
• Date: Sat, 27 May 2023 03:06:15 GMT
• Title: How Good is Automatic Segmentation as a Multimodal Discourse Annotation Aid?
• Authors: Corbin Terpstra, Ibrahim Khebour, Mariah Bradford, Brett Wisniewski, Nikhil Krishnaswamy, Nathaniel Blanchard
• Abstract summary: We assess the quality of different utterance segmentation techniques as an aid in annotating Collaborative Problem Solving. We show that the oracle utterances have minimal correspondence to automatically segmented speech, and that automatically segmented speech using different segmentation methods is also inconsistent.
• Score: 3.3861948721202233
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Collaborative problem solving (CPS) in teams is tightly coupled with the creation of shared meaning between participants in a situated, collaborative task. In this work, we assess the quality of different utterance segmentation techniques as an aid in annotating CPS. We (1) manually transcribe utterances in a dataset of triads collaboratively solving a problem involving dialogue and physical object manipulation, (2) annotate collaborative moves according to these gold-standard transcripts, and then (3) apply these annotations to utterances that have been automatically segmented using toolkits from Google and OpenAI's Whisper. We show that the oracle utterances have minimal correspondence to automatically segmented speech, and that automatically segmented speech using different segmentation methods is also inconsistent. We also show that annotating automatically segmented speech has distinct implications compared with annotating oracle utterances--since most annotation schemes are designed for oracle cases, when annotating automatically-segmented utterances, annotators must invoke other information to make arbitrary judgments which other annotators may not replicate. We conclude with a discussion of how future annotation specs can account for these needs.

Related papers

• Dude, where's my utterance? Evaluating the effects of automatic segmentation and transcription on CPS detection [0.27309692684728604]
  Collaborative Problem-Solving markers capture key aspects of effective teamwork.<n>An AI system that reliably detects these markers could help teachers identify when a group is struggling or demonstrating productive collaboration.<n>We evaluate how CPS detection is impacted by automating two critical components: transcription and speech segmentation.
  arXiv  Detail & Related papers  (2025-07-06T16:25:18Z)
• MockConf: A Student Interpretation Dataset: Analysis, Word- and Span-level Alignment and Baselines [11.037522635949939]
  We introduce MockConf, a student interpreting dataset that was collected from Mock Conferences run as part of the students' curriculum.<n>This dataset contains 7 hours of recordings in 5 European languages, transcribed and aligned at the level of spans and words.<n>We further implement and release InterAlign, a modern web-based annotation tool for parallel word and span annotations on long inputs, suitable for aligning simultaneous interpreting.
  arXiv  Detail & Related papers  (2025-06-05T10:16:15Z)
• Empowering Whisper as a Joint Multi-Talker and Target-Talker Speech Recognition System [73.34663391495616]
  We propose a pioneering approach to tackle joint multi-talker and target-talker speech recognition tasks. Specifically, we freeze Whisper and plug a Sidecar separator into its encoder to separate mixed embedding for multiple talkers. We deliver acceptable zero-shot performance on multi-talker ASR on AishellMix Mandarin dataset.
  arXiv  Detail & Related papers  (2024-07-13T09:28:24Z)
• Multi-turn Dialogue Comprehension from a Topic-aware Perspective [70.37126956655985]
  This paper proposes to model multi-turn dialogues from a topic-aware perspective. We use a dialogue segmentation algorithm to split a dialogue passage into topic-concentrated fragments in an unsupervised way. We also present a novel model, Topic-Aware Dual-Attention Matching (TADAM) Network, which takes topic segments as processing elements.
  arXiv  Detail & Related papers  (2023-09-18T11:03:55Z)
• SuperDialseg: A Large-scale Dataset for Supervised Dialogue Segmentation [55.82577086422923]
  We provide a feasible definition of dialogue segmentation points with the help of document-grounded dialogues. We release a large-scale supervised dataset called SuperDialseg, containing 9,478 dialogues. We also provide a benchmark including 18 models across five categories for the dialogue segmentation task.
  arXiv  Detail & Related papers  (2023-05-15T06:08:01Z)
• Unsupervised Dialogue Topic Segmentation with Topic-aware Utterance Representation [51.22712675266523]
  Dialogue Topic (DTS) plays an essential role in a variety of dialogue modeling tasks. We propose a novel unsupervised DTS framework, which learns topic-aware utterance representations from unlabeled dialogue data.
  arXiv  Detail & Related papers  (2023-05-04T11:35:23Z)
• Analysis of Joint Speech-Text Embeddings for Semantic Matching [3.6423306784901235]
  We study a joint speech-text embedding space trained for semantic matching by minimizing the distance between paired utterance and transcription inputs. We extend our method to incorporate automatic speech recognition through both pretraining and multitask scenarios.
  arXiv  Detail & Related papers  (2022-04-04T04:50:32Z)
• A combined approach to the analysis of speech conversations in a contact center domain [2.575030923243061]
  We describe an experimentation with a speech analytics process for an Italian contact center, that deals with call recordings extracted from inbound or outbound flows. First, we illustrate in detail the development of an in-house speech-to-text solution, based on Kaldi framework. Then, we evaluate and compare different approaches to the semantic tagging of call transcripts. Finally, a decision tree inducer, called J48S, is applied to the problem of tagging.
  arXiv  Detail & Related papers  (2022-03-12T10:03:20Z)
• What Helps Transformers Recognize Conversational Structure? Importance of Context, Punctuation, and Labels in Dialog Act Recognition [41.1669799542627]
  We apply two pre-trained transformer models to structure a conversational transcript as a sequence of dialog acts. We find that the inclusion of a broader conversational context helps disambiguate many dialog act classes. A detailed analysis reveals specific segmentation patterns observed in its absence.
  arXiv  Detail & Related papers  (2021-07-05T21:56:00Z)
• Unsupervised Summarization for Chat Logs with Topic-Oriented Ranking and Context-Aware Auto-Encoders [59.038157066874255]
  We propose a novel framework called RankAE to perform chat summarization without employing manually labeled data. RankAE consists of a topic-oriented ranking strategy that selects topic utterances according to centrality and diversity simultaneously. A denoising auto-encoder is designed to generate succinct but context-informative summaries based on the selected utterances.
  arXiv  Detail & Related papers  (2020-12-14T07:31:17Z)
• Topic-Aware Multi-turn Dialogue Modeling [91.52820664879432]
  This paper presents a novel solution for multi-turn dialogue modeling, which segments and extracts topic-aware utterances in an unsupervised way. Our topic-aware modeling is implemented by a newly proposed unsupervised topic-aware segmentation algorithm and Topic-Aware Dual-attention Matching (TADAM) Network.
  arXiv  Detail & Related papers  (2020-09-26T08:43:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.