Do Visual Features Improve Other-Initiated Repair Detection? A Dyadic Multimodal Approach

• URL: http://arxiv.org/abs/2607.23845v1
• Date: Sun, 26 Jul 2026 21:16:02 GMT
• Title: Do Visual Features Improve Other-Initiated Repair Detection? A Dyadic Multimodal Approach
• Abstract summary: Other-initiated Self-repair, or in short Other-initiated Repair, is an essential mechanism in conversational interaction.<n>This paper introduces a novel multimodal model for OIR detection and classification, incorporating a set of visual features drawn from conversation analysis.
• Score: 8.813506953164778
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Other-initiated Self-repair, or in short Other-initiated Repair (OIR), is an essential mechanism in conversational interaction, whereby a recipient signals a problem in speaking, hearing, or understanding, prompting the previous speaker to resolve it. In the case of conversational agents, it is essential to accurately identify these repair initiation strategies to address communication breakdowns efficiently. While conversational analysis studies have shown that OIR initiation is accompanied by both verbal and non-verbal signals such as gaze shifts, facial expressions, body postures, and hand gestures, existing computational approaches rely mainly on text and audio. This paper introduces a novel multimodal model for OIR detection and classification, incorporating a set of visual features drawn from conversation analysis. We evaluate our approach on two corpora with distinct languages and interaction settings. Results demonstrate that visual information consistently improves performance over text and audio baselines, and provide insights into cross-modal feature contributions across two corpora.

Related papers

• "Mm, Wat?" Detecting Other-initiated Repair Requests in Dialogue [1.0616273526777913]
  This work proposes a multimodal model to automatically detect repair initiation in Dutch dialogues.<n>The results show that prosodic cues complement linguistic features and significantly improve the results of pretrained text and audio embeddings.
  arXiv  Detail & Related papers  (2025-10-28T16:58:26Z)
• Computational Analysis of Conversation Dynamics through Participant Responsivity [18.116125865284666]
  We develop and evaluate methods for quantifying responsivity.<n>We evaluate both methods against a ground truth set of human-annotated conversations.<n>We then develop conversation-level derived metrics to address various aspects of conversational discourse.
  arXiv  Detail & Related papers  (2025-09-19T23:13:13Z)
• Towards Inclusive Communication: A Unified Framework for Generating Spoken Language from Sign, Lip, and Audio [52.859261069569165]
  We propose the first unified framework capable of handling diverse combinations of sign language, lip movements, and audio for spoken-language text generation.<n>We focus on three main objectives: (i) designing a unified, modality-agnostic architecture capable of effectively processing heterogeneous inputs; (ii) exploring the underexamined synergy among modalities, particularly the role of lip movements as non-manual cues in sign language comprehension; and (iii) achieving performance on par with or better than state-of-the-art models specialized for individual tasks.
  arXiv  Detail & Related papers  (2025-08-28T06:51:42Z)
• VocalBench: Benchmarking the Vocal Conversational Abilities for Speech Interaction Models [31.584937435966253]
  We propose VocalBench to assess the speech conversational abilities.<n>It comprises 9,400 carefully curated instances across four key dimensions.<n>It covers a broad range of fundamental skills essential for effective vocal interactions.
  arXiv  Detail & Related papers  (2025-05-21T16:34:07Z)
• Nonverbal Interaction Detection [83.40522919429337]
  This work addresses a new challenge of understanding human nonverbal interaction in social contexts. We contribute a novel large-scale dataset, called NVI, which is meticulously annotated to include bounding boxes for humans and corresponding social groups. Second, we establish a new task NVI-DET for nonverbal interaction detection, which is formalized as identifying triplets in the form individual, group, interaction> from images. Third, we propose a nonverbal interaction detection hypergraph (NVI-DEHR), a new approach that explicitly models high-order nonverbal interactions using hypergraphs.
  arXiv  Detail & Related papers  (2024-07-11T02:14:06Z)
• Enhancing HOI Detection with Contextual Cues from Large Vision-Language Models [56.257840490146]
  ConCue is a novel approach for improving visual feature extraction in HOI detection. We develop a transformer-based feature extraction module with a multi-tower architecture that integrates contextual cues into both instance and interaction detectors.
  arXiv  Detail & Related papers  (2023-11-26T09:11:32Z)
• Re-mine, Learn and Reason: Exploring the Cross-modal Semantic Correlations for Language-guided HOI detection [57.13665112065285]
  Human-Object Interaction (HOI) detection is a challenging computer vision task. We present a framework that enhances HOI detection by incorporating structured text knowledge.
  arXiv  Detail & Related papers  (2023-07-25T14:20:52Z)
• Filling the Gap of Utterance-aware and Speaker-aware Representation for Multi-turn Dialogue [76.88174667929665]
  A multi-turn dialogue is composed of multiple utterances from two or more different speaker roles. In the existing retrieval-based multi-turn dialogue modeling, the pre-trained language models (PrLMs) as encoder represent the dialogues coarsely. We propose a novel model to fill such a gap by modeling the effective utterance-aware and speaker-aware representations entailed in a dialogue history.
  arXiv  Detail & Related papers  (2020-09-14T15:07:19Z)
• Multi-Stage Conversational Passage Retrieval: An Approach to Fusing Term Importance Estimation and Neural Query Rewriting [56.268862325167575]
  We tackle conversational passage retrieval (ConvPR) with query reformulation integrated into a multi-stage ad-hoc IR system. We propose two conversational query reformulation (CQR) methods: (1) term importance estimation and (2) neural query rewriting. For the former, we expand conversational queries using important terms extracted from the conversational context with frequency-based signals. For the latter, we reformulate conversational queries into natural, standalone, human-understandable queries with a pretrained sequence-tosequence model.
  arXiv  Detail & Related papers  (2020-05-05T14:30:20Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.