Unpacking Human-AI interactions: From interaction primitives to a design
space
- URL: http://arxiv.org/abs/2401.05115v1
- Date: Wed, 10 Jan 2024 12:27:18 GMT
- Title: Unpacking Human-AI interactions: From interaction primitives to a design
space
- Authors: Kostas Tsiakas and Dave Murray-Rust
- Abstract summary: We show how these primitives can be combined into a set of interaction patterns.
The motivation behind this is to provide a compact generalisation of existing practices.
We discuss how this approach can be used towards a design space for Human-AI interactions.
- Score: 6.778055454461106
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: This paper aims to develop a semi-formal design space for Human-AI
interactions, by building a set of interaction primitives which specify the
communication between users and AI systems during their interaction. We show
how these primitives can be combined into a set of interaction patterns which
can provide an abstract specification for exchanging messages between humans
and AI/ML models to carry out purposeful interactions. The motivation behind
this is twofold: firstly, to provide a compact generalisation of existing
practices, that highlights the similarities and differences between systems in
terms of their interaction behaviours; and secondly, to support the creation of
new systems, in particular by opening the space of possibilities for
interactions with models. We present a short literature review on frameworks,
guidelines and taxonomies related to the design and implementation of HAI
interactions, including human-in-the-loop, explainable AI, as well as hybrid
intelligence and collaborative learning approaches. From the literature review,
we define a vocabulary for describing information exchanges in terms of
providing and requesting particular model-specific data types. Based on this
vocabulary, a message passing model for interactions between humans and models
is presented, which we demonstrate can account for existing systems and
approaches. Finally, we build this into design patterns as mid-level constructs
that capture common interactional structures. We discuss how this approach can
be used towards a design space for Human-AI interactions that creates new
possibilities for designs as well as keeping track of implementation issues and
concerns.
Related papers
- Accounting for AI and Users Shaping One Another: The Role of Mathematical Models [17.89344451611069]
We argue for the development of formal interaction models which mathematically specify how AI and users shape one another.
We call for the community to leverage formal interaction models when designing, evaluating, or auditing any AI system which interacts with users.
arXiv Detail & Related papers (2024-04-18T17:49:02Z) - THOR: Text to Human-Object Interaction Diffusion via Relation Intervention [51.02435289160616]
We propose a novel Text-guided Human-Object Interaction diffusion model with Relation Intervention (THOR)
In each diffusion step, we initiate text-guided human and object motion and then leverage human-object relations to intervene in object motion.
We construct Text-BEHAVE, a Text2HOI dataset that seamlessly integrates textual descriptions with the currently largest publicly available 3D HOI dataset.
arXiv Detail & Related papers (2024-03-17T13:17:25Z) - Detecting Any Human-Object Interaction Relationship: Universal HOI
Detector with Spatial Prompt Learning on Foundation Models [55.20626448358655]
This study explores the universal interaction recognition in an open-world setting through the use of Vision-Language (VL) foundation models and large language models (LLMs)
Our design includes an HO Prompt-guided Decoder (HOPD), facilitates the association of high-level relation representations in the foundation model with various HO pairs within the image.
For open-category interaction recognition, our method supports either of two input types: interaction phrase or interpretive sentence.
arXiv Detail & Related papers (2023-11-07T08:27:32Z) - A Grammatical Compositional Model for Video Action Detection [24.546886938243393]
We present a novel Grammatical Compositional Model (GCM) for action detection based on typical And-Or graphs.
Our model exploits the intrinsic structures and latent relationships of actions in a hierarchical manner to harness both the compositionality of grammar models and the capability of expressing rich features of DNNs.
arXiv Detail & Related papers (2023-10-04T15:24:00Z) - Foundational Models Defining a New Era in Vision: A Survey and Outlook [151.49434496615427]
Vision systems to see and reason about the compositional nature of visual scenes are fundamental to understanding our world.
The models learned to bridge the gap between such modalities coupled with large-scale training data facilitate contextual reasoning, generalization, and prompt capabilities at test time.
The output of such models can be modified through human-provided prompts without retraining, e.g., segmenting a particular object by providing a bounding box, having interactive dialogues by asking questions about an image or video scene or manipulating the robot's behavior through language instructions.
arXiv Detail & Related papers (2023-07-25T17:59:18Z) - Interactive Natural Language Processing [67.87925315773924]
Interactive Natural Language Processing (iNLP) has emerged as a novel paradigm within the field of NLP.
This paper offers a comprehensive survey of iNLP, starting by proposing a unified definition and framework of the concept.
arXiv Detail & Related papers (2023-05-22T17:18:29Z) - Collective Relational Inference for learning heterogeneous interactions [8.215734914005845]
We propose a novel probabilistic method for relational inference, which possesses two distinctive characteristics compared to existing methods.
We evaluate the proposed methodology across several benchmark datasets and demonstrate that it outperforms existing methods in accurately inferring interaction types.
Overall the proposed model is data-efficient and generalizable to large systems when trained on smaller ones.
arXiv Detail & Related papers (2023-04-30T19:45:04Z) - RR-Net: Injecting Interactive Semantics in Human-Object Interaction
Detection [40.65483058890176]
Latest end-to-end HOI detectors are short of relation reasoning, which leads to inability to learn HOI-specific interactive semantics for predictions.
We first present a progressive Relation-aware Frame, which brings a new structure and parameter sharing pattern for interaction inference.
Based on modules above, we construct an end-to-end trainable framework named Relation Reasoning Network (abbr. RR-Net)
arXiv Detail & Related papers (2021-04-30T14:03:10Z) - Interactive Teaching for Conversational AI [2.5259192787433706]
Current conversational AI systems aim to understand a set of pre-designed requests and execute related actions.
Motivated by how children learn their first language interacting with adults, this paper describes a new Teachable AI system.
It is capable of learning new language nuggets called concepts, directly from end users using live interactive teaching sessions.
arXiv Detail & Related papers (2020-12-02T04:08:49Z) - Adaptive Interaction Modeling via Graph Operations Search [109.45125932109454]
We automate the process of structures design to learn adaptive structures for interaction modeling.
We experimentally demonstrate that our architecture search framework learns to construct adaptive interaction modeling structures.
Our method achieves competitive performance with state-of-the-arts.
arXiv Detail & Related papers (2020-05-05T13:01:09Z) - Cascaded Human-Object Interaction Recognition [175.60439054047043]
We introduce a cascade architecture for a multi-stage, coarse-to-fine HOI understanding.
At each stage, an instance localization network progressively refines HOI proposals and feeds them into an interaction recognition network.
With our carefully-designed human-centric relation features, these two modules work collaboratively towards effective interaction understanding.
arXiv Detail & Related papers (2020-03-09T17:05:04Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.