Learning Interactions and Relationships between Movie Characters
- URL: http://arxiv.org/abs/2003.13158v1
- Date: Sun, 29 Mar 2020 23:11:24 GMT
- Title: Learning Interactions and Relationships between Movie Characters
- Authors: Anna Kukleva and Makarand Tapaswi and Ivan Laptev
- Abstract summary: We propose neural models to learn and jointly predict interactions, relationships, and the pair of characters that are involved.
Localizing the pair of interacting characters in video is a time-consuming process, instead, we train our model to learn from clip-level weak labels.
We evaluate our models on the MovieGraphs dataset and show the impact of modalities, use of longer temporal context for predicting relationships, and achieve encouraging performance using weak labels.
- Score: 37.27773051465456
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Interactions between people are often governed by their relationships. On the
flip side, social relationships are built upon several interactions. Two
strangers are more likely to greet and introduce themselves while becoming
friends over time. We are fascinated by this interplay between interactions and
relationships, and believe that it is an important aspect of understanding
social situations. In this work, we propose neural models to learn and jointly
predict interactions, relationships, and the pair of characters that are
involved. We note that interactions are informed by a mixture of visual and
dialog cues, and present a multimodal architecture to extract meaningful
information from them. Localizing the pair of interacting characters in video
is a time-consuming process, instead, we train our model to learn from
clip-level weak labels. We evaluate our models on the MovieGraphs dataset and
show the impact of modalities, use of longer temporal context for predicting
relationships, and achieve encouraging performance using weak labels as
compared with ground-truth labels. Code is online.
Related papers
- Modeling Memory Imprints Induced by Interactions in Social Networks [0.0]
Despite the importance of relationships in social networks, there is little work exploring how interactions over extended periods correlate with people's memory imprints of relationship importance.
In this paper, we represent memory dynamics by adapting a well-known cognitive science model.
We find that this model, trained on one population, predicts not only on this population but also on a different one, suggesting the universality of memory imprints of social interactions among unrelated individuals.
arXiv Detail & Related papers (2022-10-06T20:35:07Z) - Triangular Character Animation Sampling with Motion, Emotion, and
Relation [78.80083186208712]
We present a novel framework to sample and synthesize animations by associating the characters' body motions, facial expressions, and social relations.
Our method can provide animators with an automatic way to generate 3D character animations, help synthesize interactions between Non-Player Characters (NPCs) and enhance machine emotion intelligence in virtual reality (VR)
arXiv Detail & Related papers (2022-03-09T18:19:03Z) - Distillation of Human-Object Interaction Contexts for Action Recognition [0.0]
We learn human-object relationships by exploiting the interaction of their local and global contexts.
We propose the Global-Local Interaction Distillation Network (GLIDN), learning human and object interactions through space and time.
GLIDN encodes humans and objects into graph nodes and learns local and global relations via graph attention network.
arXiv Detail & Related papers (2021-12-17T11:39:44Z) - SSAGCN: Social Soft Attention Graph Convolution Network for Pedestrian
Trajectory Prediction [59.064925464991056]
We propose one new prediction model named Social Soft Attention Graph Convolution Network (SSAGCN)
SSAGCN aims to simultaneously handle social interactions among pedestrians and scene interactions between pedestrians and environments.
Experiments on public available datasets prove the effectiveness of SSAGCN and have achieved state-of-the-art results.
arXiv Detail & Related papers (2021-12-05T01:49:18Z) - Spatio-Temporal Interaction Graph Parsing Networks for Human-Object
Interaction Recognition [55.7731053128204]
In given video-based Human-Object Interaction scene, modeling thetemporal relationship between humans and objects are the important cue to understand the contextual information presented in the video.
With the effective-temporal relationship modeling, it is possible not only to uncover contextual information in each frame but also directly capture inter-time dependencies.
The full use of appearance features, spatial location and the semantic information are also the key to improve the video-based Human-Object Interaction recognition performance.
arXiv Detail & Related papers (2021-08-19T11:57:27Z) - Learning Asynchronous and Sparse Human-Object Interaction in Videos [56.73059840294019]
Asynchronous-Sparse Interaction Graph Networks (ASSIGN) is able to automatically detect the structure of interaction events associated with entities in a video scene.
ASSIGN is tested on human-object interaction recognition and shows superior performance in segmenting and labeling of human sub-activities and object affordances from raw videos.
arXiv Detail & Related papers (2021-03-03T23:43:55Z) - A Machine Learning Approach to Predicting Continuous Tie Strengths [0.4014524824655105]
Relationships between people constantly evolve, altering interpersonal behavior and defining social groups.
Relationships between nodes in social networks can be represented by a tie strength, often empirically assessed using surveys.
We propose a system that allows for the continuous approximation of relationships as they evolve over time.
arXiv Detail & Related papers (2021-01-23T05:01:05Z) - "where is this relationship going?": Understanding Relationship
Trajectories in Narrative Text [28.14874371042193]
Given a narrative describing a social interaction, systems make inferences about the underlying relationship trajectory.
We construct a new dataset, Social Narrative Tree, which consists of 1250 stories documenting a variety of daily social interactions.
arXiv Detail & Related papers (2020-10-29T02:07:05Z) - RKT : Relation-Aware Self-Attention for Knowledge Tracing [2.9778695679660188]
We propose a novel Relation-aware self-attention model for Knowledge Tracing (RKT)
We introduce a relation-aware self-attention layer that incorporates the contextual information.
Our model outperforms state-of-the-art knowledge tracing methods.
arXiv Detail & Related papers (2020-08-28T16:47:03Z) - ConsNet: Learning Consistency Graph for Zero-Shot Human-Object
Interaction Detection [101.56529337489417]
We consider the problem of Human-Object Interaction (HOI) Detection, which aims to locate and recognize HOI instances in the form of human, action, object> in images.
We argue that multi-level consistencies among objects, actions and interactions are strong cues for generating semantic representations of rare or previously unseen HOIs.
Our model takes visual features of candidate human-object pairs and word embeddings of HOI labels as inputs, maps them into visual-semantic joint embedding space and obtains detection results by measuring their similarities.
arXiv Detail & Related papers (2020-08-14T09:11:18Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.