DRIVE-T: A Methodology for Discriminative and Representative Data Viz Item Selection for Literacy Construct and Assessment

• URL: http://arxiv.org/abs/2508.04160v1
• Date: Wed, 06 Aug 2025 07:34:40 GMT
• Title: DRIVE-T: A Methodology for Discriminative and Representative Data Viz Item Selection for Literacy Construct and Assessment
• Abstract summary: DRIVE-T is a methodology designed to drive the construction and evaluation of assessment items.<n>It supports the identification of task-based items discriminability and representativeness for measuring levels of data visualization literacy.<n>We show and apply each step of the methodology to an item bank, which models the difficulty levels of a measurement construct.
• Score: 1.9389881806157316
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The underspecification of progressive levels of difficulty in measurement constructs design and assessment tests for data visualization literacy may hinder the expressivity of measurements in both test design and test reuse. To mitigate this problem, this paper proposes DRIVE-T (Discriminating and Representative Items for Validating Expressive Tests), a methodology designed to drive the construction and evaluation of assessment items. Given a data vizualization, DRIVE-T supports the identification of task-based items discriminability and representativeness for measuring levels of data visualization literacy. DRIVE-T consists of three steps: (1) tagging task-based items associated with a set of data vizualizations; (2) rating them by independent raters for their difficulty; (3) analysing raters' raw scores through a Many-Facet Rasch Measurement model. In this way, we can observe the emergence of difficulty levels of the measurement construct, derived from the discriminability and representativeness of task-based items for each data vizualization, ordered into Many-Facets construct levels. In this study, we show and apply each step of the methodology to an item bank, which models the difficulty levels of a measurement construct approximating a latent construct for data visualization literacy. This measurement construct is drawn from semiotics, i.e., based on the syntax, semantics and pragmatics knowledge that each data visualization may require to be mastered by people. The DRIVE-T methodology operationalises an inductive approach, observable in a post-design phase of the items preparation, for formative-style and practice-based measurement construct emergence. A pilot study with items selected through the application of DRIVE-T is also presented to test our approach.

Related papers

• Can LLMs Clean Up Your Mess? A Survey of Application-Ready Data Preparation with LLMs [66.63911043019294]
  Data preparation aims to denoise raw datasets, uncover cross-dataset relationships, and extract valuable insights from them.<n>This paper focuses on the use of LLM techniques to prepare data for diverse downstream tasks.<n>We introduce a task-centric taxonomy that organizes the field into three major tasks: data cleaning, standardization, error processing, imputation, data integration, and data enrichment.
  arXiv  Detail & Related papers  (2026-01-22T12:02:45Z)
• LeWiDi-2025 at NLPerspectives: The Third Edition of the Learning with Disagreements Shared Task [38.500623751317896]
  The LEWIDI series of shared tasks on Learning With Disagreements was established to promote this approach to training and evaluating AI models.<n>The third edition of the task builds on this goal by extending the LEWIDI benchmark to four datasets spanning paraphrase identification, irony detection, sarcasm detection, and natural language inference.
  arXiv  Detail & Related papers  (2025-10-09T17:04:28Z)
• Human-AI Collaboration or Academic Misconduct? Measuring AI Use in Student Writing Through Stylometric Evidence [0.0]
  This research investigates the use of authorship verification (AV) techniques to quantify AI assistance in academic writing.<n>We use three datasets - including a public dataset (PAN-14) and two from University of Melbourne students from various courses.<n>We develop an adapted Feature Vector Difference AV methodology to construct robust academic writing profiles for students.
  arXiv  Detail & Related papers  (2025-05-13T00:36:36Z)
• Enriching Social Science Research via Survey Item Linking [11.902701975866595]
  We model a task called Survey Item Linking (SIL) in two stages: mention detection and entity disambiguation.<n>To this end, we create a high-quality and richly annotated dataset consisting of 20,454 English and German sentences.<n>We demonstrate that the task is feasible, but observe that errors propagate from the first stage, leading to a lower overall task performance.
  arXiv  Detail & Related papers  (2024-12-20T12:14:33Z)
• How Hard is this Test Set? NLI Characterization by Exploiting Training Dynamics [49.9329723199239]
  We propose a method for the automated creation of a challenging test set without relying on the manual construction of artificial and unrealistic examples. We categorize the test set of popular NLI datasets into three difficulty levels by leveraging methods that exploit training dynamics. When our characterization method is applied to the training set, models trained with only a fraction of the data achieve comparable performance to those trained on the full dataset.
  arXiv  Detail & Related papers  (2024-10-04T13:39:21Z)
• DecompEval: Evaluating Generated Texts as Unsupervised Decomposed Question Answering [95.89707479748161]
  Existing evaluation metrics for natural language generation (NLG) tasks face the challenges on generalization ability and interpretability. We propose a metric called DecompEval that formulates NLG evaluation as an instruction-style question answering task. We decompose our devised instruction-style question about the quality of generated texts into the subquestions that measure the quality of each sentence. The subquestions with their answers generated by PLMs are then recomposed as evidence to obtain the evaluation result.
  arXiv  Detail & Related papers  (2023-07-13T16:16:51Z)
• Weakly Supervised Anomaly Detection: A Survey [75.26180038443462]
  Anomaly detection (AD) is a crucial task in machine learning with various applications. We present the first comprehensive survey of weakly supervised anomaly detection (WSAD) methods. For each setting, we provide formal definitions, key algorithms, and potential future directions.
  arXiv  Detail & Related papers  (2023-02-09T10:27:21Z)
• ROSCOE: A Suite of Metrics for Scoring Step-by-Step Reasoning [63.77667876176978]
  Large language models show improved downstream task interpretability when prompted to generate step-by-step reasoning to justify their final answers. These reasoning steps greatly improve model interpretability and verification, but objectively studying their correctness is difficult. We present ROS, a suite of interpretable, unsupervised automatic scores that improve and extend previous text generation evaluation metrics.
  arXiv  Detail & Related papers  (2022-12-15T15:52:39Z)
• Breakpoint Transformers for Modeling and Tracking Intermediate Beliefs [37.754787051387034]
  We propose a representation learning framework called breakpoint modeling. Our approach trains models in an efficient and end-to-end fashion to build intermediate representations. We show the benefit of our main breakpoint transformer, based on T5, over conventional representation learning approaches.
  arXiv  Detail & Related papers  (2022-11-15T07:28:14Z)
• Data-SUITE: Data-centric identification of in-distribution incongruous examples [81.21462458089142]
  Data-SUITE is a data-centric framework to identify incongruous regions of in-distribution (ID) data. We empirically validate Data-SUITE's performance and coverage guarantees.
  arXiv  Detail & Related papers  (2022-02-17T18:58:31Z)
• CINS: Comprehensive Instruction for Few-shot Learning in Task-oriented Dialog Systems [56.302581679816775]
  This paper proposes Comprehensive Instruction (CINS) that exploits PLMs with task-specific instructions. We design a schema (definition, constraint, prompt) of instructions and their customized realizations for three important downstream tasks in ToD. Experiments are conducted on these ToD tasks in realistic few-shot learning scenarios with small validation data.
  arXiv  Detail & Related papers  (2021-09-10T03:23:06Z)
• Domain-Guided Task Decomposition with Self-Training for Detecting Personal Events in Social Media [11.638298634523945]
  Mining social media for tasks such as detecting personal experiences or events, suffer from lexical sparsity, insufficient training data, and inventive lexicons. To reduce the burden of creating extensive labeled data, we propose to perform these tasks in two steps: 1. Decomposing the task into domain-specific sub-tasks by identifying key concepts, thus utilizing human domain understanding; 2. Combining the results of learners for each key concept using co-training to reduce the requirements for labeled training data.
  arXiv  Detail & Related papers  (2020-04-21T14:50:31Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.