The Language of Infographics: Toward Understanding Conceptual Metaphor Use in Scientific Storytelling

• URL: http://arxiv.org/abs/2407.13416v1
• Date: Thu, 18 Jul 2024 11:39:50 GMT
• Title: The Language of Infographics: Toward Understanding Conceptual Metaphor Use in Scientific Storytelling
• Authors: Hana Pokojná, Tobias Isenberg, Stefan Bruckner, Barbora Kozlíková, Laura Garrison,
• Abstract summary: We map Conceptual Metaphor (CMT) to the visualization domain to address patterns of visual conceptual metaphors that are often used in science infographics. Our analysis shows that ontological and orientational conceptual metaphors are the most widely applied to translate complex scientific concepts.
• Score: 9.302187675469554
• License:
• Abstract: We apply an approach from cognitive linguistics by mapping Conceptual Metaphor Theory (CMT) to the visualization domain to address patterns of visual conceptual metaphors that are often used in science infographics. Metaphors play an essential part in visual communication and are frequently employed to explain complex concepts. However, their use is often based on intuition, rather than following a formal process. At present, we lack tools and language for understanding and describing metaphor use in visualization to the extent where taxonomy and grammar could guide the creation of visual components, e.g., infographics. Our classification of the visual conceptual mappings within scientific representations is based on the breakdown of visual components in existing scientific infographics. We demonstrate the development of this mapping through a detailed analysis of data collected from four domains (biomedicine, climate, space, and anthropology) that represent a diverse range of visual conceptual metaphors used in the visual communication of science. This work allows us to identify patterns of visual conceptual metaphor use within the domains, resolve ambiguities about why specific conceptual metaphors are used, and develop a better overall understanding of visual metaphor use in scientific infographics. Our analysis shows that ontological and orientational conceptual metaphors are the most widely applied to translate complex scientific concepts. To support our findings we developed a visual exploratory tool based on the collected database that places the individual infographics on a spatio-temporal scale and illustrates the breakdown of visual conceptual metaphors.

Related papers

• Science is Exploration: Computational Frontiers for Conceptual Metaphor Theory [0.0]
  We show that Large Language Models (LLMs) can accurately identify and explain the presence of conceptual metaphors in natural language data. Using a novel prompting technique based on metaphor annotation guidelines, we demonstrate that LLMs are a promising tool for large-scale computational research on conceptual metaphors.
  arXiv  Detail & Related papers  (2024-10-11T17:03:13Z)
• What Makes a Maze Look Like a Maze? [92.80800000328277]
  We introduce Deep Grounding (DSG), a framework that leverages explicit structured representations of visual abstractions for grounding and reasoning. At the core of DSG are schemas--dependency graph descriptions of abstract concepts that decompose them into more primitive-level symbols. We show that DSG significantly improves the abstract visual reasoning performance of vision-language models.
  arXiv  Detail & Related papers  (2024-09-12T16:41:47Z)
• For a semiotic AI: Bridging computer vision and visual semiotics for computational observation of large scale facial image archives [3.418398936676879]
  This work presents FRESCO, a framework designed to explore the socio-cultural implications of images on social media platforms at scale. FRESCO deconstructs images into numerical and categorical variables using state-of-the-art computer vision techniques. The framework analyzes images across three levels: the plastic level, encompassing fundamental visual features like lines and colors; the figurative level, representing specific entities or concepts; and the enunciation level, which focuses particularly on constructing the point of view of the spectator and observer.
  arXiv  Detail & Related papers  (2024-07-03T16:57:38Z)
• LMs stand their Ground: Investigating the Effect of Embodiment in Figurative Language Interpretation by Language Models [0.0]
  Figurative language is a challenge for language models since its interpretation deviates from their conventional order and meaning. Yet, humans can easily understand and interpret metaphors as they can be derived from embodied metaphors. This study shows how larger language models perform better at interpreting metaphoric sentences when the action of the metaphorical sentence is more embodied.
  arXiv  Detail & Related papers  (2023-05-05T11:44:12Z)
• MetaCLUE: Towards Comprehensive Visual Metaphors Research [43.604408485890275]
  We introduce MetaCLUE, a set of vision tasks on visual metaphor. We perform a comprehensive analysis of state-of-the-art models in vision and language based on our annotations. We hope this work provides a concrete step towards developing AI systems with human-like creative capabilities.
  arXiv  Detail & Related papers  (2022-12-19T22:41:46Z)
• Learnable Visual Words for Interpretable Image Recognition [70.85686267987744]
  We propose the Learnable Visual Words (LVW) to interpret the model prediction behaviors with two novel modules. The semantic visual words learning relaxes the category-specific constraint, enabling the general visual words shared across different categories. Our experiments on six visual benchmarks demonstrate the superior effectiveness of our proposed LVW in both accuracy and model interpretation.
  arXiv  Detail & Related papers  (2022-05-22T03:24:45Z)
• Emergent Graphical Conventions in a Visual Communication Game [80.79297387339614]
  Humans communicate with graphical sketches apart from symbolic languages. We take the very first step to model and simulate such an evolution process via two neural agents playing a visual communication game. We devise a novel reinforcement learning method such that agents are evolved jointly towards successful communication and abstract graphical conventions.
  arXiv  Detail & Related papers  (2021-11-28T18:59:57Z)
• Visual resemblance and communicative context constrain the emergence of graphical conventions [21.976382800327965]
  Drawing provides a versatile medium for communicating about the visual world. Do viewers understand drawings based solely on their ability to resemble the entities they refer to (i.e., as images)? Do they understand drawings based on shared but arbitrary associations with these entities (i.e. as symbols)?
  arXiv  Detail & Related papers  (2021-09-17T23:05:36Z)
• Metaphor Generation with Conceptual Mappings [58.61307123799594]
  We aim to generate a metaphoric sentence given a literal expression by replacing relevant verbs. We propose to control the generation process by encoding conceptual mappings between cognitive domains. We show that the unsupervised CM-Lex model is competitive with recent deep learning metaphor generation systems.
  arXiv  Detail & Related papers  (2021-06-02T15:27:05Z)
• Natural Language Rationales with Full-Stack Visual Reasoning: From Pixels to Semantic Frames to Commonsense Graphs [106.15931418425906]
  We present the first study focused on generating natural language rationales across several complex visual reasoning tasks. We present RationaleVT Transformer, an integrated model that learns to generate free-text rationales by combining pretrained language models with object recognition, grounded visual semantic frames, and visual commonsense graphs. Our experiments show that the base pretrained language model benefits from visual adaptation and that free-text rationalization is a promising research direction to complement model interpretability for complex visual-textual reasoning tasks.
  arXiv  Detail & Related papers  (2020-10-15T05:08:56Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.