Meaning Is Not A Metric: Using LLMs to make cultural context legible at scale

• URL: http://arxiv.org/abs/2505.23785v1
• Date: Fri, 23 May 2025 04:10:42 GMT
• Title: Meaning Is Not A Metric: Using LLMs to make cultural context legible at scale
• Authors: Cody Kommers, Drew Hemment, Maria Antoniak, Joel Z. Leibo, Hoyt Long, Emily Robinson, Adam Sobey,
• Abstract summary: We argue that large language models (LLMs) can make cultural context, and therefore human meaning, legible at an unprecedented scale in AI-based sociotechnical systems.<n>We frame this as a crucial direction for the application of generative AI and identify five key challenges.
• Score: 3.283323176831235
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: This position paper argues that large language models (LLMs) can make cultural context, and therefore human meaning, legible at an unprecedented scale in AI-based sociotechnical systems. We argue that such systems have previously been unable to represent human meaning because they rely on thin descriptions: numerical representations that enforce standardization and therefore strip human activity of the cultural context that gives it meaning. By contrast, scholars in the humanities and qualitative social sciences have developed frameworks for representing meaning through thick description: verbal representations that accommodate heterogeneity and retain contextual information needed to represent human meaning. While these methods can effectively codify meaning, they are difficult to deploy at scale. However, the verbal capabilities of LLMs now provide a means of (at least partially) automating the generation and processing of thick descriptions, potentially overcoming this bottleneck. We argue that the problem of rendering human meaning legible is not just about selecting better metrics, but about developing new representational formats (based on thick description). We frame this as a crucial direction for the application of generative AI and identify five key challenges: preserving context, maintaining interpretive pluralism, integrating perspectives based on lived experience and critical distance, distinguishing qualitative content from quantitative magnitude, and acknowledging meaning as dynamic rather than static. Furthermore, we suggest that thick description has the potential to serve as a unifying framework to address a number of emerging concerns about the difficulties of representing culture in (or using) LLMs.

Related papers

• Reasoning Beyond Labels: Measuring LLM Sentiment in Low-Resource, Culturally Nuanced Contexts [10.492471013369782]
  We present a framework that treats sentiment as a context-dependent, culturally embedded construct.<n>We evaluate how large language models (LLMs) reason about sentiment in WhatsApp messages from Nairobi youth health groups.
  arXiv  Detail & Related papers  (2025-08-06T08:27:55Z)
• Vocabulary-free Fine-grained Visual Recognition via Enriched Contextually Grounded Vision-Language Model [52.01031460230826]
  Traditional approaches rely heavily on fixed vocabularies and closed-set classification paradigms.<n>Recent research has demonstrated that combining large language models with vision-language models (VLMs) makes open-set recognition possible.<n>We propose our training-free method, Enriched-FineR, which demonstrates state-of-the-art results in fine-grained visual recognition.
  arXiv  Detail & Related papers  (2025-07-30T20:06:01Z)
• Cultural Bias in Large Language Models: Evaluating AI Agents through Moral Questionnaires [0.0]
  Large Language Models fail to represent diverse cultural moral frameworks despite their linguistic capabilities.<n>Surprisingly, increased model size doesn't consistently improve cultural representation fidelity.<n>Our results call for more grounded alignment objectives and evaluation metrics to ensure AI systems represent diverse human values.
  arXiv  Detail & Related papers  (2025-07-14T08:59:26Z)
• On The Role of Intentionality in Knowledge Representation: Analyzing Scene Context for Cognitive Agents with a Tiny Language Model [0.0]
  Promise Theory's model of Semantic Spacetime is used as an effective Tiny Language Model.<n>One can identify themes and concepts from a text on a low level.<n>Scale separation can be used to sort parts into intended' content and ambient context'
  arXiv  Detail & Related papers  (2025-07-14T07:34:58Z)
• Text Embeddings Should Capture Implicit Semantics, Not Just Surface Meaning [17.00358234728804]
  We argue that the text embedding research community should move beyond surface meaning and embrace implicit semantics as a central modeling goal.<n>Current embedding models are typically trained on data that lacks such depth and evaluated on benchmarks that reward the capture of surface meaning.<n>Our pilot study highlights this gap, showing that even state-of-the-art models perform only marginally better than simplistic baselines on implicit semantics tasks.
  arXiv  Detail & Related papers  (2025-06-10T02:11:42Z)
• From Tokens to Thoughts: How LLMs and Humans Trade Compression for Meaning [52.32745233116143]
  Humans organize knowledge into compact categories through semantic compression.<n>Large Language Models (LLMs) demonstrate remarkable linguistic abilities.<n>But whether their internal representations strike a human-like trade-off between compression and semantic fidelity is unclear.
  arXiv  Detail & Related papers  (2025-05-21T16:29:00Z)
• Evaluating LLMs for Targeted Concept Simplification for Domain-Specific Texts [53.421616210871704]
  Lack of context and unfamiliarity with difficult concepts is a major reason for adult readers' difficulty with domain-specific text. We introduce "targeted concept simplification," a simplification task for rewriting text to help readers comprehend text containing unfamiliar concepts. We benchmark the performance of open-source and commercial LLMs and a simple dictionary baseline on this task.
  arXiv  Detail & Related papers  (2024-10-28T05:56:51Z)
• Interpreting Pretrained Language Models via Concept Bottlenecks [55.47515772358389]
  Pretrained language models (PLMs) have made significant strides in various natural language processing tasks. The lack of interpretability due to their black-box'' nature poses challenges for responsible implementation. We propose a novel approach to interpreting PLMs by employing high-level, meaningful concepts that are easily understandable for humans.
  arXiv  Detail & Related papers  (2023-11-08T20:41:18Z)
• Visual Grounding Helps Learn Word Meanings in Low-Data Regimes [47.7950860342515]
  Modern neural language models (LMs) are powerful tools for modeling human sentence production and comprehension. But to achieve these results, LMs must be trained in distinctly un-human-like ways. Do models trained more naturalistically -- with grounded supervision -- exhibit more humanlike language learning? We investigate this question in the context of word learning, a key sub-task in language acquisition.
  arXiv  Detail & Related papers  (2023-10-20T03:33:36Z)
• Dealing with Semantic Underspecification in Multimodal NLP [3.5846770619764423]
  Intelligent systems that aim at mastering language as humans do must deal with its semantic underspecification. Standard NLP models have, in principle, no or limited access to such extra information. multimodal systems grounding language into other modalities, such as vision, are naturally equipped to account for this phenomenon.
  arXiv  Detail & Related papers  (2023-06-08T14:39:24Z)
• Natural Language Decompositions of Implicit Content Enable Better Text Representations [52.992875653864076]
  We introduce a method for the analysis of text that takes implicitly communicated content explicitly into account.<n>We use a large language model to produce sets of propositions that are inferentially related to the text that has been observed.<n>Our results suggest that modeling the meanings behind observed language, rather than the literal text alone, is a valuable direction for NLP.
  arXiv  Detail & Related papers  (2023-05-23T23:45:20Z)
• Meaning without reference in large language models [14.26628686684198]
  We argue that large language models (LLMs) likely capture important aspects of meaning. Because conceptual role is defined by the relationships between internal representational states, meaning cannot be determined from a model's architecture.
  arXiv  Detail & Related papers  (2022-08-05T02:48:26Z)
• Testing the Ability of Language Models to Interpret Figurative Language [69.59943454934799]
  Figurative and metaphorical language are commonplace in discourse. It remains an open question to what extent modern language models can interpret nonliteral phrases. We introduce Fig-QA, a Winograd-style nonliteral language understanding task.
  arXiv  Detail & Related papers  (2022-04-26T23:42:22Z)
• Towards Understanding and Mitigating Social Biases in Language Models [107.82654101403264]
  Large-scale pretrained language models (LMs) can be potentially dangerous in manifesting undesirable representational biases. We propose steps towards mitigating social biases during text generation. Our empirical results and human evaluation demonstrate effectiveness in mitigating bias while retaining crucial contextual information.
  arXiv  Detail & Related papers  (2021-06-24T17:52:43Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.