Visual Theory of Mind Enables the Invention of Proto-Writing

• URL: http://arxiv.org/abs/2502.01568v4
• Date: Fri, 18 Apr 2025 19:18:17 GMT
• Title: Visual Theory of Mind Enables the Invention of Proto-Writing
• Authors: Benjamin A. Spiegel, Lucas Gelfond, George Konidaris,
• Abstract summary: Evidence suggests that the earliest forms of some writing systems originally consisted of iconic pictographs.<n>Our model sheds light on the cognitive and cultural processes underlying the emergence of proto-writing.
• Score: 10.013537728631038
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Symbolic writing systems are graphical semiotic codes that are ubiquitous in modern society but are otherwise absent in the animal kingdom. Anthropological evidence suggests that the earliest forms of some writing systems originally consisted of iconic pictographs, which signify their referent via visual resemblance. While previous studies have examined the emergence and, separately, the evolution of pictographic systems through a computational lens, most employ non-naturalistic methodologies that make it difficult to draw clear analogies to human and animal cognition. We develop a multi-agent reinforcement learning testbed for emergent communication called a Signification Game, and formulate a model of inferential communication that enables agents to leverage visual theory of mind to communicate actions using pictographs. Our model, which is situated within a broader formalism for animal communication, sheds light on the cognitive and cultural processes underlying the emergence of proto-writing.

Related papers

• Human-like conceptual representations emerge from language prediction [72.5875173689788]
  Large language models (LLMs) trained exclusively through next-token prediction over language data exhibit remarkably human-like behaviors. Are these models developing concepts akin to humans, and if so, how are such concepts represented and organized? Our results demonstrate that LLMs can flexibly derive concepts from linguistic descriptions in relation to contextual cues about other concepts. These findings establish that structured, human-like conceptual representations can naturally emerge from language prediction without real-world grounding.
  arXiv  Detail & Related papers  (2025-01-21T23:54:17Z)
• Science is Exploration: Computational Frontiers for Conceptual Metaphor Theory [0.0]
  We show that Large Language Models (LLMs) can accurately identify and explain the presence of conceptual metaphors in natural language data. Using a novel prompting technique based on metaphor annotation guidelines, we demonstrate that LLMs are a promising tool for large-scale computational research on conceptual metaphors.
  arXiv  Detail & Related papers  (2024-10-11T17:03:13Z)
• The Language of Infographics: Toward Understanding Conceptual Metaphor Use in Scientific Storytelling [9.302187675469554]
  We map Conceptual Metaphor (CMT) to the visualization domain to address patterns of visual conceptual metaphors that are often used in science infographics. Our analysis shows that ontological and orientational conceptual metaphors are the most widely applied to translate complex scientific concepts.
  arXiv  Detail & Related papers  (2024-07-18T11:39:50Z)
• Models of symbol emergence in communication: a conceptual review and a guide for avoiding local minima [0.0]
  Computational simulations are a popular method for testing hypotheses about the emergence of communication. We identify the assumptions and explanatory targets of several most representative models and summarise the known results. In line with this perspective, we sketch the road towards modelling the emergence of meaningful symbolic communication.
  arXiv  Detail & Related papers  (2023-03-08T12:53:03Z)
• CLAMP: Prompt-based Contrastive Learning for Connecting Language and Animal Pose [70.59906971581192]
  We introduce a novel prompt-based Contrastive learning scheme for connecting Language and AniMal Pose effectively. The CLAMP attempts to bridge the gap by adapting the text prompts to the animal keypoints during network training. Experimental results show that our method achieves state-of-the-art performance under the supervised, few-shot, and zero-shot settings.
  arXiv  Detail & Related papers  (2022-06-23T14:51:42Z)
• Things not Written in Text: Exploring Spatial Commonsense from Visual Signals [77.46233234061758]
  We investigate whether models with visual signals learn more spatial commonsense than text-based models. We propose a benchmark that focuses on the relative scales of objects, and the positional relationship between people and objects under different actions. We find that image synthesis models are more capable of learning accurate and consistent spatial knowledge than other models.
  arXiv  Detail & Related papers  (2022-03-15T17:02:30Z)
• Emergent Graphical Conventions in a Visual Communication Game [80.79297387339614]
  Humans communicate with graphical sketches apart from symbolic languages. We take the very first step to model and simulate such an evolution process via two neural agents playing a visual communication game. We devise a novel reinforcement learning method such that agents are evolved jointly towards successful communication and abstract graphical conventions.
  arXiv  Detail & Related papers  (2021-11-28T18:59:57Z)
• pix2rule: End-to-end Neuro-symbolic Rule Learning [84.76439511271711]
  This paper presents a complete neuro-symbolic method for processing images into objects, learning relations and logical rules. The main contribution is a differentiable layer in a deep learning architecture from which symbolic relations and rules can be extracted. We demonstrate that our model scales beyond state-of-the-art symbolic learners and outperforms deep relational neural network architectures.
  arXiv  Detail & Related papers  (2021-06-14T15:19:06Z)
• Metaphor Generation with Conceptual Mappings [58.61307123799594]
  We aim to generate a metaphoric sentence given a literal expression by replacing relevant verbs. We propose to control the generation process by encoding conceptual mappings between cognitive domains. We show that the unsupervised CM-Lex model is competitive with recent deep learning metaphor generation systems.
  arXiv  Detail & Related papers  (2021-06-02T15:27:05Z)
• Cetacean Translation Initiative: a roadmap to deciphering the communication of sperm whales [97.41394631426678]
  Recent research showed the promise of machine learning tools for analyzing acoustic communication in nonhuman species. We outline the key elements required for the collection and processing of massive bioacoustic data of sperm whales. The technological capabilities developed are likely to yield cross-applications and advancements in broader communities investigating non-human communication and animal behavioral research.
  arXiv  Detail & Related papers  (2021-04-17T18:39:22Z)
• Natural Language Rationales with Full-Stack Visual Reasoning: From Pixels to Semantic Frames to Commonsense Graphs [106.15931418425906]
  We present the first study focused on generating natural language rationales across several complex visual reasoning tasks. We present RationaleVT Transformer, an integrated model that learns to generate free-text rationales by combining pretrained language models with object recognition, grounded visual semantic frames, and visual commonsense graphs. Our experiments show that the base pretrained language model benefits from visual adaptation and that free-text rationalization is a promising research direction to complement model interpretability for complex visual-textual reasoning tasks.
  arXiv  Detail & Related papers  (2020-10-15T05:08:56Z)
• Structural Inductive Biases in Emergent Communication [36.26083882473554]
  We investigate the impact of representation learning in artificial agents by developing graph referential games. We show that agents parametrized by graph neural networks develop a more compositional language compared to bag-of-words and sequence models.
  arXiv  Detail & Related papers  (2020-02-04T14:59:08Z)
• Towards Graph Representation Learning in Emergent Communication [37.8523331078468]
  We use graph convolutional networks to support the evolution of language and cooperation in multi-agent systems. Motivated by an image-based referential game, we propose a graph referential game with varying degrees of complexity. We show that the emerged communication protocol is robust, that the agents uncover the true factors of variation in the game, and that they learn to generalize beyond the samples encountered during training.
  arXiv  Detail & Related papers  (2020-01-24T15:55:59Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.