Perceptual Structure in the Absence of Grounding for LLMs: The Impact of Abstractedness and Subjectivity in Color Language

• URL: http://arxiv.org/abs/2311.13105v1
• Date: Wed, 22 Nov 2023 02:12:36 GMT
• Title: Perceptual Structure in the Absence of Grounding for LLMs: The Impact of Abstractedness and Subjectivity in Color Language
• Authors: Pablo Loyola, Edison Marrese-Taylor, Andres Hoyos-Idobro
• Abstract summary: We show that there is considerable alignment between a defined color space and the feature space defined by a language model. Our results show that while color space alignment holds for monolexemic, highly pragmatic color descriptions, this alignment drops considerably in the presence of examples that exhibit elements of real linguistic usage.
• Score: 2.6094835036012864
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The need for grounding in language understanding is an active research topic. Previous work has suggested that color perception and color language appear as a suitable test bed to empirically study the problem, given its cognitive significance and showing that there is considerable alignment between a defined color space and the feature space defined by a language model. To further study this issue, we collect a large scale source of colors and their descriptions, containing almost a 1 million examples , and perform an empirical analysis to compare two kinds of alignments: (i) inter-space, by learning a mapping between embedding space and color space, and (ii) intra-space, by means of prompting comparatives between color descriptions. Our results show that while color space alignment holds for monolexemic, highly pragmatic color descriptions, this alignment drops considerably in the presence of examples that exhibit elements of real linguistic usage such as subjectivity and abstractedness, suggesting that grounding may be required in such cases.

Related papers

• L-CAD: Language-based Colorization with Any-level Descriptions using Diffusion Priors [62.80068955192816]
  We propose a unified model to perform language-based colorization with any-level descriptions. We leverage the pretrained cross-modality generative model for its robust language understanding and rich color priors. With the proposed novel sampling strategy, our model achieves instance-aware colorization in diverse and complex scenarios.
  arXiv  Detail & Related papers  (2023-05-24T14:57:42Z)
• Exploration of the Usage of Color Terms by Color-blind Participants in Online Discussion Platforms [4.445130093341008]
  We show that red-green color-blind speakers use the "red" and "green" color terms in less predictable contexts. These findings shed some new and interesting light on the role of sensory experience on our linguistic system.
  arXiv  Detail & Related papers  (2022-10-21T12:11:10Z)
• An Informational Space Based Semantic Analysis for Scientific Texts [62.997667081978825]
  This paper introduces computational methods for semantic analysis and the quantifying the meaning of short scientific texts. The representation of scientific-specific meaning is standardised by replacing the situation representations, rather than psychological properties. The research in this paper conducts the base for the geometric representation of the meaning of texts.
  arXiv  Detail & Related papers  (2022-05-31T11:19:32Z)
• Visual Superordinate Abstraction for Robust Concept Learning [80.15940996821541]
  Concept learning constructs visual representations that are connected to linguistic semantics. We ascribe the bottleneck to a failure of exploring the intrinsic semantic hierarchy of visual concepts. We propose a visual superordinate abstraction framework for explicitly modeling semantic-aware visual subspaces.
  arXiv  Detail & Related papers  (2022-05-28T14:27:38Z)
• Can Language Models Encode Perceptual Structure Without Grounding? A Case Study in Color [18.573415435334105]
  We employ a dataset of monolexemic color terms and color chips represented in CIELAB, a color space with a perceptually meaningful distance metric. Using two methods of evaluating the structural alignment of colors in this space with text-derived color term representations, we find significant correspondence. We find that warmer colors are, on average, better aligned to the perceptual color space than cooler ones.
  arXiv  Detail & Related papers  (2021-09-13T17:09:40Z)
• Learning Zero-Shot Multifaceted Visually Grounded Word Embeddingsvia Multi-Task Training [8.271859911016719]
  Language grounding aims at linking the symbolic representation of language (e.g., words) into the rich perceptual knowledge of the outside world. We argue that this approach sacrifices the abstract knowledge obtained from linguistic co-occurrence statistics in the process of acquiring perceptual information.
  arXiv  Detail & Related papers  (2021-04-15T14:49:11Z)
• From Spatial Relations to Spatial Configurations [64.21025426604274]
  spatial relation language is able to represent a large, comprehensive set of spatial concepts crucial for reasoning. We show how we extend the capabilities of existing spatial representation languages with the fine-grained decomposition of semantics.
  arXiv  Detail & Related papers  (2020-07-19T02:11:53Z)
• Understanding Spatial Relations through Multiple Modalities [78.07328342973611]
  spatial relations between objects can either be explicit -- expressed as spatial prepositions, or implicit -- expressed by spatial verbs such as moving, walking, shifting, etc. We introduce the task of inferring implicit and explicit spatial relations between two entities in an image. We design a model that uses both textual and visual information to predict the spatial relations, making use of both positional and size information of objects and image embeddings.
  arXiv  Detail & Related papers  (2020-07-19T01:35:08Z)
• Cultural Cartography with Word Embeddings [0.0]
  We show how word embeddings are commensurate with prevailing theories of meaning in sociology. First, one can hold terms constant and measure how the embedding space moves around them. Second, one can also hold the embedding space constant and see how documents or authors move relative to it.
  arXiv  Detail & Related papers  (2020-07-09T01:58:28Z)
• Probing Contextual Language Models for Common Ground with Visual Representations [76.05769268286038]
  We design a probing model that evaluates how effective are text-only representations in distinguishing between matching and non-matching visual representations. Our findings show that language representations alone provide a strong signal for retrieving image patches from the correct object categories. Visually grounded language models slightly outperform text-only language models in instance retrieval, but greatly under-perform humans.
  arXiv  Detail & Related papers  (2020-05-01T21:28:28Z)
• Incorporating Visual Semantics into Sentence Representations within a Grounded Space [20.784771968813747]
  We propose to transfer visual information to textual representations by learning an intermediate representation space: the grounded space. We show that this model outperforms the previous state-of-the-art on classification and semantic relatedness tasks.
  arXiv  Detail & Related papers  (2020-02-07T12:26:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.