Comprehension vs. Adoption: Evaluating a Language Workbench Through a Family of Experiments

• URL: http://arxiv.org/abs/2601.20394v1
• Date: Wed, 28 Jan 2026 09:00:59 GMT
• Title: Comprehension vs. Adoption: Evaluating a Language Workbench Through a Family of Experiments
• Authors: Giovanna Broccia, Maurice H. ter Beek, Walter Cazzola, Luca Favalli, Francesco Bertolotti, Alessio Ferrari,
• Abstract summary: This paper adopts a tailored version of the Method Evaluation Model (MEM) to evaluate the comprehensibility of Neverlang's meta-language.<n>It also investigates user acceptance in terms of perceived ease of use, perceived usefulness, and intention to use.<n>Surprisingly, no significant correlation is found between comprehensibility and user acceptance.
• Score: 12.601523755289051
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Language workbenches are tools that enable the definition, reuse, and composition of programming languages and their ecosystems, aiming to streamline language development. To facilitate their adoption by language designers, the comprehensibility of the language used to define other languages is an important aspect to evaluate. Moreover, considering that language workbenches are relatively new tools, user acceptance emerges as a crucial factor to be accounted for during their assessment. Current literature often neglects user-centred aspects like comprehensibility and acceptance in the assessment of this breed of tools. This paper addresses this gap through a family of experiments assessing Neverlang, a modular language workbench. The study adopts a tailored version of the Method Evaluation Model (MEM) to evaluate the comprehensibility of Neverlang's meta-language and programs, as well as user acceptance in terms of perceived ease of use, perceived usefulness, and intention to use. It also investigates the relationships among these dimensions. The experiments were conducted in three iterations involving participants from academia. The results reveal that users demonstrate sufficient comprehension of Neverlang's meta-language, particularly concerning its syntax, express a favourable perception of its usefulness, and indicate their intention to use it. However, the results also indicate that Neverlang's ease of use remains a challenge. Additionally, variations in the perceived ease of use and perceived usefulness, whether low or high, influence the users' intention to use the tool. Surprisingly, no significant correlation is found between comprehensibility and user acceptance. Notably, higher comprehensibility of the meta-language does not necessarily translate into greater acceptance, underscoring the complex interplay between comprehension and adoption.

Related papers

• On the Entity-Level Alignment in Crosslingual Consistency [62.33186691736433]
  SubSub and SubInj integrate English translations of subjects into prompts across languages, leading to substantial gains in factual recall accuracy and consistency.<n>These interventions reinforce the entity representation alignment in the conceptual space through model's internal pivot-language processing.
  arXiv  Detail & Related papers  (2025-10-11T16:26:50Z)
• ImpScore: A Learnable Metric For Quantifying The Implicitness Level of Sentence [40.4052848203136]
  Implicit language is essential for natural language processing systems to achieve precise text understanding and facilitate natural interactions with users.<n>This paper develops a scalar metric that quantifies the implicitness level of language without relying on external references.<n>We validate ImpScore through a user study that compares its assessments with human evaluations on out-of-distribution data.
  arXiv  Detail & Related papers  (2024-11-07T20:23:29Z)
• Scaffolded Language Models with Language Supervision for Mixed-Autonomy: A Survey [52.00674453604779]
  This survey organizes the literature on the design and optimization of emerging structures around post-trained LMs.<n>We refer to this overarching structure as scaffolded LMs and focus on LMs that are integrated into multi-step processes with tools.
  arXiv  Detail & Related papers  (2024-10-21T18:06:25Z)
• Holmes: A Benchmark to Assess the Linguistic Competence of Language Models [59.627729608055006]
  We introduce Holmes, a new benchmark designed to assess language models (LMs) linguistic competence. We use computation-based probing to examine LMs' internal representations regarding distinct linguistic phenomena. As a result, we meet recent calls to disentangle LMs' linguistic competence from other cognitive abilities.
  arXiv  Detail & Related papers  (2024-04-29T17:58:36Z)
• Interpretable User Satisfaction Estimation for Conversational Systems with Large Language Models [35.95405294377247]
  Existing approaches based on featurized ML models or text embeddings fall short in extracting generalizable patterns. We show that LLMs can extract interpretable signals of user satisfaction from their natural language utterances more effectively than embedding-based approaches.
  arXiv  Detail & Related papers  (2024-03-19T02:57:07Z)
• Syntax and Semantics Meet in the "Middle": Probing the Syntax-Semantics Interface of LMs Through Agentivity [68.8204255655161]
  We present the semantic notion of agentivity as a case study for probing such interactions. This suggests LMs may potentially serve as more useful tools for linguistic annotation, theory testing, and discovery.
  arXiv  Detail & Related papers  (2023-05-29T16:24:01Z)
• Knowledge Graph Guided Semantic Evaluation of Language Models For User Trust [7.063958622970576]
  This study evaluates the encoded semantics in the self-attention transformers by leveraging explicit knowledge graph structures. The opacity of language models has an immense bearing on societal issues of trust and explainable decision outcomes.
  arXiv  Detail & Related papers  (2023-05-08T18:53:14Z)
• Neural Abstructions: Abstractions that Support Construction for Grounded Language Learning [69.1137074774244]
  Leveraging language interactions effectively requires addressing limitations in the two most common approaches to language grounding. We introduce the idea of neural abstructions: a set of constraints on the inference procedure of a label-conditioned generative model. We show that with this method a user population is able to build a semantic modification for an open-ended house task in Minecraft.
  arXiv  Detail & Related papers  (2021-07-20T07:01:15Z)
• Pre-training Universal Language Representation [46.51685959045527]
  This work introduces universal language representation learning, i.e., embeddings of different levels of linguistic units or text with quite diverse lengths in a uniform vector space. We empirically verify that well designed pre-training scheme may effectively yield universal language representation.
  arXiv  Detail & Related papers  (2021-05-30T09:29:01Z)
• AM2iCo: Evaluating Word Meaning in Context across Low-ResourceLanguages with Adversarial Examples [51.048234591165155]
  We present AM2iCo, Adversarial and Multilingual Meaning in Context. It aims to faithfully assess the ability of state-of-the-art (SotA) representation models to understand the identity of word meaning in cross-lingual contexts. Results reveal that current SotA pretrained encoders substantially lag behind human performance.
  arXiv  Detail & Related papers  (2021-04-17T20:23:45Z)
• Curious Case of Language Generation Evaluation Metrics: A Cautionary Tale [52.663117551150954]
  A few popular metrics remain as the de facto metrics to evaluate tasks such as image captioning and machine translation. This is partly due to ease of use, and partly because researchers expect to see them and know how to interpret them. In this paper, we urge the community for more careful consideration of how they automatically evaluate their models.
  arXiv  Detail & Related papers  (2020-10-26T13:57:20Z)
• Natural language technology and query expansion: issues, state-of-the-art and perspectives [0.0]
  Linguistic characteristics that cause ambiguity and misinterpretation of queries as well as additional factors affect the users ability to accurately represent their information needs. We lay down the anatomy of a generic linguistic based query expansion framework and propose its module-based decomposition. For each of the modules we review the state-of-the-art solutions in the literature and categorized under the light of the techniques used.
  arXiv  Detail & Related papers  (2020-04-23T11:39:07Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.