Computational Hermeneutics: Evaluating generative AI as a cultural technology

• URL: http://arxiv.org/abs/2604.16403v1
• Date: Tue, 31 Mar 2026 12:18:56 GMT
• Title: Computational Hermeneutics: Evaluating generative AI as a cultural technology
• Abstract summary: We argue that GenAI systems function as "context machines" that must inherently address three interpretive challenges.<n>We offer three principles for hermeneutic evaluation -- that benchmarks should be iterative, not one-off; include people, not just machines; and measure cultural context, not just model output.
• Score: 32.49036529181191
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Generative AI systems are increasingly recognized as cultural technologies, yet current evaluation frameworks often treat culture as a variable to be measured rather than fundamental to the system's operation. Drawing on hermeneutic theory from the humanities, we argue that GenAI systems function as "context machines" that must inherently address three interpretive challenges: situatedness (meaning only emerges in context), plurality (multiple valid interpretations coexist), and ambiguity (interpretations naturally conflict). We present computational hermeneutics as an emerging framework offering an interpretive account of what GenAI systems do, and how they might do it better. We offer three principles for hermeneutic evaluation -- that benchmarks should be iterative, not one-off; include people, not just machines; and measure cultural context, not just model output. This perspective offers a nascent paradigm for designing and evaluating contemporary AI systems: shifting from standardized questions about accuracy to contextual ones about meaning.

Related papers

• AI-Assisted Systematization for Evaluating GenAI Systems [45.6388922756291]
  We develop two AI-assisted systematizers to produce concept specs for two concepts.<n>We evaluate resulting concept specs on content validity and information recoverability.
  arXiv  Detail & Related papers  (2026-05-25T16:19:44Z)
• Measuring the Machine: Evaluating Generative AI as Pluralist Sociotechical Systems [0.0]
  The thesis argues that generative AI must be evaluated as a pluralist sociotechnical system.<n>MaSH Loops is a framework for tracing how models, users, and institutions co-construct meaning and values.<n>The World Values Benchmark introduces a distributional approach grounded in World Values Survey data.
  arXiv  Detail & Related papers  (2026-04-22T13:29:33Z)
• The Vibe-Automation of Automation: A Proactive Education Framework for Computer Science in the Age of Generative AI [0.7252027234425333]
  generative artificial intelligence (GenAI) represents a qualitative shift in computer science.<n>GenAI operates by navigating contextual, semantic, and coherence rather than optimizing predefined objective metrics.<n>The paper proposes a conceptual framework structured across three analytical levels and three domains of action.
  arXiv  Detail & Related papers  (2026-02-09T06:02:04Z)
• Position: Evaluating Generative AI Systems Is a Social Science Measurement Challenge [78.35388859345056]
  We argue that the ML community would benefit from learning from and drawing on the social sciences when developing measurement instruments for evaluating GenAI systems.<n>We present a four-level framework, grounded in measurement theory from the social sciences, for measuring concepts related to the capabilities, behaviors, and impacts of GenAI systems.
  arXiv  Detail & Related papers  (2025-02-01T21:09:51Z)
• Rethinking AI Cultural Alignment [1.8434042562191815]
  We show that humans' cultural values must be understood within the context of specific AI systems.<n>We argue that cultural alignment should be reframed as a bidirectional process.
  arXiv  Detail & Related papers  (2025-01-13T23:42:37Z)
• Towards Automatic Evaluation for Image Transcreation [52.71090829502756]
  We propose a suite of automatic evaluation metrics inspired by machine translation (MT) metrics.<n>We identify three critical dimensions of image transcreation: cultural relevance, semantic equivalence and visual similarity.<n>Our results show that proprietary VLMs best identify cultural relevance and semantic equivalence, while vision-encoder representations are adept at measuring visual similarity.
  arXiv  Detail & Related papers  (2024-12-18T10:55:58Z)
• Artificial intelligence in government: Concepts, standards, and a unified framework [0.0]
  Recent advances in artificial intelligence (AI) hold the promise of transforming government. It is critical that new AI systems behave in alignment with the normative expectations of society.
  arXiv  Detail & Related papers  (2022-10-31T10:57:20Z)
• An interdisciplinary conceptual study of Artificial Intelligence (AI) for helping benefit-risk assessment practices: Towards a comprehensive qualification matrix of AI programs and devices (pre-print 2020) [55.41644538483948]
  This paper proposes a comprehensive analysis of existing concepts coming from different disciplines tackling the notion of intelligence. The aim is to identify shared notions or discrepancies to consider for qualifying AI systems.
  arXiv  Detail & Related papers  (2021-05-07T12:01:31Z)
• LioNets: A Neural-Specific Local Interpretation Technique Exploiting Penultimate Layer Information [6.570220157893279]
  Interpretable machine learning (IML) is an urgent topic of research. This paper focuses on a local-based, neural-specific interpretation process applied to textual and time-series data.
  arXiv  Detail & Related papers  (2021-04-13T09:39:33Z)
• A Minimalist Dataset for Systematic Generalization of Perception, Syntax, and Semantics [131.93113552146195]
  We present a new dataset, Handwritten arithmetic with INTegers (HINT), to examine machines' capability of learning generalizable concepts. In HINT, machines are tasked with learning how concepts are perceived from raw signals such as images. We undertake extensive experiments with various sequence-to-sequence models, including RNNs, Transformers, and GPT-3.
  arXiv  Detail & Related papers  (2021-03-02T01:32:54Z)
• Bongard-LOGO: A New Benchmark for Human-Level Concept Learning and Reasoning [78.13740873213223]
  Bongard problems (BPs) were introduced as an inspirational challenge for visual cognition in intelligent systems. We propose a new benchmark Bongard-LOGO for human-level concept learning and reasoning.
  arXiv  Detail & Related papers  (2020-10-02T03:19:46Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.