Supporting Experts with a Multimodal Machine-Learning-Based Tool for
Human Behavior Analysis of Conversational Videos
- URL: http://arxiv.org/abs/2402.11145v1
- Date: Sat, 17 Feb 2024 00:27:04 GMT
- Title: Supporting Experts with a Multimodal Machine-Learning-Based Tool for
Human Behavior Analysis of Conversational Videos
- Authors: Riku Arakawa and Kiyosu Maeda and Hiromu Yakura
- Abstract summary: We developed Providence, a visual-programming-based tool based on design considerations derived from a formative study with experts.
It enables experts to combine various machine learning algorithms to capture human behavioral cues without writing code.
Our study showed its preferable usability and satisfactory output with less cognitive load imposed in accomplishing scene search tasks of conversations.
- Score: 40.30407535831779
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Multimodal scene search of conversations is essential for unlocking valuable
insights into social dynamics and enhancing our communication. While experts in
conversational analysis have their own knowledge and skills to find key scenes,
a lack of comprehensive, user-friendly tools that streamline the processing of
diverse multimodal queries impedes efficiency and objectivity. To solve it, we
developed Providence, a visual-programming-based tool based on design
considerations derived from a formative study with experts. It enables experts
to combine various machine learning algorithms to capture human behavioral cues
without writing code. Our study showed its preferable usability and
satisfactory output with less cognitive load imposed in accomplishing scene
search tasks of conversations, verifying the importance of its customizability
and transparency. Furthermore, through the in-the-wild trial, we confirmed the
objectivity and reusability of the tool transform experts' workflow, suggesting
the advantage of expert-AI teaming in a highly human-contextual domain.
Related papers
- Interactive Multi-Objective Evolutionary Optimization of Software
Architectures [0.0]
Putting the human in the loop brings new challenges to the search-based software engineering field.
This paper explores how the interactive evolutionary computation can serve as a basis for integrating the human's judgment into the search process.
arXiv Detail & Related papers (2024-01-08T19:15:40Z) - RLIF: Interactive Imitation Learning as Reinforcement Learning [56.997263135104504]
We show how off-policy reinforcement learning can enable improved performance under assumptions that are similar but potentially even more practical than those of interactive imitation learning.
Our proposed method uses reinforcement learning with user intervention signals themselves as rewards.
This relaxes the assumption that intervening experts in interactive imitation learning should be near-optimal and enables the algorithm to learn behaviors that improve over the potential suboptimal human expert.
arXiv Detail & Related papers (2023-11-21T21:05:21Z) - Re-mine, Learn and Reason: Exploring the Cross-modal Semantic
Correlations for Language-guided HOI detection [57.13665112065285]
Human-Object Interaction (HOI) detection is a challenging computer vision task.
We present a framework that enhances HOI detection by incorporating structured text knowledge.
arXiv Detail & Related papers (2023-07-25T14:20:52Z) - Interactive Natural Language Processing [67.87925315773924]
Interactive Natural Language Processing (iNLP) has emerged as a novel paradigm within the field of NLP.
This paper offers a comprehensive survey of iNLP, starting by proposing a unified definition and framework of the concept.
arXiv Detail & Related papers (2023-05-22T17:18:29Z) - An Interdisciplinary Perspective on Evaluation and Experimental Design
for Visual Text Analytics: Position Paper [24.586485898038312]
In this paper, we focus on the issues of evaluating visual text analytics approaches.
We identify four key groups of challenges for evaluating visual text analytics approaches.
arXiv Detail & Related papers (2022-09-23T11:47:37Z) - Co-Located Human-Human Interaction Analysis using Nonverbal Cues: A
Survey [71.43956423427397]
We aim to identify the nonverbal cues and computational methodologies resulting in effective performance.
This survey differs from its counterparts by involving the widest spectrum of social phenomena and interaction settings.
Some major observations are: the most often used nonverbal cue, computational method, interaction environment, and sensing approach are speaking activity, support vector machines, and meetings composed of 3-4 persons equipped with microphones and cameras, respectively.
arXiv Detail & Related papers (2022-07-20T13:37:57Z) - SOCIOFILLMORE: A Tool for Discovering Perspectives [10.189255026322996]
SOCIOFILLMORE is a tool which helps to bring to the fore the perspective that a text expresses in depicting an event.
Our tool, whose rationale we also support through a large collection of human judgements, is theoretically grounded on frame semantics and cognitive linguistics.
arXiv Detail & Related papers (2022-03-07T14:42:22Z) - Estimating Presentation Competence using Multimodal Nonverbal Behavioral
Cues [7.340483819263093]
Public speaking and presentation competence plays an essential role in many areas of social interaction.
One approach that can promote efficient development of presentation competence is the automated analysis of human behavior during a speech.
In this work, we investigate the contribution of different nonverbal behavioral cues, namely, facial, body pose-based, and audio-related features, to estimate presentation competence.
arXiv Detail & Related papers (2021-05-06T13:09:41Z) - On Interactive Machine Learning and the Potential of Cognitive Feedback [2.320417845168326]
We introduce interactive machine learning and explain its advantages and limitations within the context of defense applications.
We define the three techniques by which cognitive feedback may be employed: self reporting, implicit cognitive feedback, and modeled cognitive feedback.
arXiv Detail & Related papers (2020-03-23T16:28:14Z) - A Review on Intelligent Object Perception Methods Combining
Knowledge-based Reasoning and Machine Learning [60.335974351919816]
Object perception is a fundamental sub-field of Computer Vision.
Recent works seek ways to integrate knowledge engineering in order to expand the level of intelligence of the visual interpretation of objects.
arXiv Detail & Related papers (2019-12-26T13:26:49Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.