Mixed Multi-Model Semantic Interaction for Graph-based Narrative Visualizations

• URL: http://arxiv.org/abs/2302.06452v1
• Date: Mon, 13 Feb 2023 15:32:10 GMT
• Title: Mixed Multi-Model Semantic Interaction for Graph-based Narrative Visualizations
• Authors: Brian Keith Norambuena, Tanushree Mitra, Chris North
• Abstract summary: Narrative maps are a visual representation model that can assist analysts to understand narratives. We present a semantic interaction framework for narrative maps that can support analysts through their sensemaking process. We find that our SI system can model the analysts' intent and support incremental formalism for narrative maps.
• Score: 10.193264105560862
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Narrative sensemaking is an essential part of understanding sequential data. Narrative maps are a visual representation model that can assist analysts to understand narratives. In this work, we present a semantic interaction (SI) framework for narrative maps that can support analysts through their sensemaking process. In contrast to traditional SI systems which rely on dimensionality reduction and work on a projection space, our approach has an additional abstraction layer -- the structure space -- that builds upon the projection space and encodes the narrative in a discrete structure. This extra layer introduces additional challenges that must be addressed when integrating SI with the narrative extraction pipeline. We address these challenges by presenting the general concept of Mixed Multi-Model Semantic Interaction (3MSI) -- an SI pipeline, where the highest-level model corresponds to an abstract discrete structure and the lower-level models are continuous. To evaluate the performance of our 3MSI models for narrative maps, we present a quantitative simulation-based evaluation and a qualitative evaluation with case studies and expert feedback. We find that our SI system can model the analysts' intent and support incremental formalism for narrative maps.

Related papers

• Explore In-Context Segmentation via Latent Diffusion Models [132.26274147026854]
  latent diffusion model (LDM) is an effective minimalist for in-context segmentation. We build a new and fair in-context segmentation benchmark that includes both image and video datasets.
  arXiv  Detail & Related papers  (2024-03-14T17:52:31Z)
• Foundational Models Defining a New Era in Vision: A Survey and Outlook [151.49434496615427]
  Vision systems to see and reason about the compositional nature of visual scenes are fundamental to understanding our world. The models learned to bridge the gap between such modalities coupled with large-scale training data facilitate contextual reasoning, generalization, and prompt capabilities at test time. The output of such models can be modified through human-provided prompts without retraining, e.g., segmenting a particular object by providing a bounding box, having interactive dialogues by asking questions about an image or video scene or manipulating the robot's behavior through language instructions.
  arXiv  Detail & Related papers  (2023-07-25T17:59:18Z)
• M-SENSE: Modeling Narrative Structure in Short Personal Narratives Using Protagonist's Mental Representations [14.64546899992196]
  We propose the task of automatically detecting prominent elements of the narrative structure by analyzing the role of characters' inferred mental state. We introduce a STORIES dataset of short personal narratives containing manual annotations of key elements of narrative structure, specifically climax and resolution. Our model is able to achieve significant improvements in the task of identifying climax and resolution.
  arXiv  Detail & Related papers  (2023-02-18T20:48:02Z)
• Learning Interpretable Latent Dialogue Actions With Less Supervision [3.42658286826597]
  We present a novel architecture for explainable modeling of task-oriented dialogues with discrete latent variables. Our model is based on variational recurrent neural networks (VRNN) and requires no explicit annotation of semantic information.
  arXiv  Detail & Related papers  (2022-09-22T16:14:06Z)
• Support-set based Multi-modal Representation Enhancement for Video Captioning [121.70886789958799]
  We propose a Support-set based Multi-modal Representation Enhancement (SMRE) model to mine rich information in a semantic subspace shared between samples. Specifically, we propose a Support-set Construction (SC) module to construct a support-set to learn underlying connections between samples and obtain semantic-related visual elements. During this process, we design a Semantic Space Transformation (SST) module to constrain relative distance and administrate multi-modal interactions in a self-supervised way.
  arXiv  Detail & Related papers  (2022-05-19T03:40:29Z)
• Temporal Relevance Analysis for Video Action Models [70.39411261685963]
  We first propose a new approach to quantify the temporal relationships between frames captured by CNN-based action models. We then conduct comprehensive experiments and in-depth analysis to provide a better understanding of how temporal modeling is affected.
  arXiv  Detail & Related papers  (2022-04-25T19:06:48Z)
• Efficient Multi-Modal Embeddings from Structured Data [0.0]
  Multi-modal word semantics aims to enhance embeddings with perceptual input. Visual grounding can contribute to linguistic applications as well. New embedding conveys complementary information for text based embeddings.
  arXiv  Detail & Related papers  (2021-10-06T08:42:09Z)
• Unified Graph Structured Models for Video Understanding [93.72081456202672]
  We propose a message passing graph neural network that explicitly models relational-temporal relations. We show how our method is able to more effectively model relationships between relevant entities in the scene.
  arXiv  Detail & Related papers  (2021-03-29T14:37:35Z)
• Narrative Maps: An Algorithmic Approach to Represent and Extract Information Narratives [6.85316573653194]
  This article combines the theory of narrative representations with the data from modern online systems. A narrative map representation illustrates the events and stories in the narrative as a series of landmarks and routes on the map. Our findings have implications for intelligence analysts, computational journalists, and misinformation researchers.
  arXiv  Detail & Related papers  (2020-09-09T18:30:44Z)
• S2RMs: Spatially Structured Recurrent Modules [105.0377129434636]
  We take a step towards exploiting dynamic structure that are capable of simultaneously exploiting both modular andtemporal structures. We find our models to be robust to the number of available views and better capable of generalization to novel tasks without additional training.
  arXiv  Detail & Related papers  (2020-07-13T17:44:30Z)
• Tension Space Analysis for Emergent Narrative [0.1784936803975635]
  We present a novel approach to emergent narrative using the narratological theory of possible worlds. We demonstrate how the design of works in such a system can be understood through a formal means of analysis inspired by expressive range analysis. Finally, we propose a novel way through which content may be authored for the emergent narrative system using a sketch-based interface.
  arXiv  Detail & Related papers  (2020-04-22T19:26:09Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.