Usability Evaluation and Improvement of a Tool for Self-Service Learning Analytics

• URL: http://arxiv.org/abs/2603.24321v1
• Date: Wed, 25 Mar 2026 14:01:53 GMT
• Title: Usability Evaluation and Improvement of a Tool for Self-Service Learning Analytics
• Abstract summary: Self-Service Learning Analytics (SSLA) tools aim to support educational stakeholders in creating learning analytics indicators without requiring technical expertise.<n>This paper presents a usability evaluation and improvement of the Indicator Editor, a no-code, exploratory SSLA tool.
• Score: 0.0
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Self-Service Learning Analytics (SSLA) tools aim to support educational stakeholders in creating learning analytics indicators without requiring technical expertise. While such tools promise user control and trans- parency, their effectiveness and adoption depend critically on usability aspects. This paper presents a compre- hensive usability evaluation and improvement of the Indicator Editor, a no-code, exploratory SSLA tool that enables non-technical users to implement custom learning analytics indicators through a structured workflow. Using an iterative evaluation approach, we conduct an exploratory qualitative user study, usability inspections of high-fidelity prototypes, and a workshop-based evaluation in an authentic educational setting with n = 46 students using standardized instruments, namely System Usability Scale (SUS), User Experience Question- naire (UEQ), and Net Promoter Score (NPS). Based on the evaluation findings, we derive concrete design implications that inform improvements in workflow guidance, feedback, and information presentation in the Indicator Editor. Furthermore, our evaluation provides practical insights for the design of usable SSLA tools.

Related papers

• Enabling Extensible Embodied Capabilities with Tools [29.084279305733336]
  Embodied intelligence methods formulate perception, reasoning, planning, and control within a unified parameterized policy.<n>We propose a capability externalization approach that decouples heterogeneous capabilities into independently optimized tools, dynamically invoked at inference time.
  arXiv  Detail & Related papers  (2026-05-26T07:18:36Z)
• RecThinker: An Agentic Framework for Tool-Augmented Reasoning in Recommendation [63.74915464611075]
  RecThinker is an agentic framework for tool-augmented reasoning in recommendation.<n>We develop a suite of specialized tools for RecThinker, enabling the model to acquire user item-side and collaborative information.
  arXiv  Detail & Related papers  (2026-03-10T16:07:17Z)
• AITutor-EvalKit: Exploring the Capabilities of AI Tutors [15.444428064780363]
  We present AITutor-EvalKit, an application that uses language technology to evaluate the pedagogical quality of AI tutors.<n>It provides software for demonstration and evaluation, as well as model inspection and data visualization.
  arXiv  Detail & Related papers  (2025-12-03T11:27:50Z)
• Human-Centered Development of Indicators for Self-Service Learning Analytics: A Transparency through Exploration Approach [0.0]
  The aim of learning analytics is to turn educational data into insights, decisions, and actions to improve learning and teaching.<n>The reasoning of the provided insights, decisions, and actions is often not transparent to the end-user.<n>In this paper, we shed light on achieving transparent learning analytics by following a transparency through exploration approach.
  arXiv  Detail & Related papers  (2025-10-09T16:15:54Z)
• Improving Large Language Models Function Calling and Interpretability via Guided-Structured Templates [56.73907811047611]
  Large language models (LLMs) have demonstrated strong reasoning and tool-use capabilities.<n>LLMs often fail in real-world tool-interactions due to incorrect parameterization, poor tool selection, or misinterpretation of user intent.<n>We introduce a curriculum-inspired framework that leverages structured reasoning templates to guide LLMs through more deliberate step-by-step instructions for generating function callings.
  arXiv  Detail & Related papers  (2025-09-22T17:55:14Z)
• Teaching at Scale: Leveraging AI to Evaluate and Elevate Engineering Education [3.557803321422781]
  This article presents a scalable, AI-supported framework for qualitative student feedback using large language models.<n>The system employs hierarchical summarization, anonymization, and exception handling to extract actionable themes from open-ended comments.<n>We report on its successful deployment across a large college of engineering.
  arXiv  Detail & Related papers  (2025-08-01T20:27:40Z)
• VisualToolAgent (VisTA): A Reinforcement Learning Framework for Visual Tool Selection [47.259066449806866]
  VisTA is a new reinforcement learning framework that empowers visual agents to dynamically explore, select, and combine tools from a diverse library based on empirical performance.<n>We show that VisTA achieves substantial performance gains over training-free baselines.<n>These results highlight VisTA's ability to enhance generalization, adaptively utilize diverse tools, and pave the way for flexible, experience-driven visual reasoning systems.
  arXiv  Detail & Related papers  (2025-05-26T17:59:17Z)
• ToolSpectrum : Towards Personalized Tool Utilization for Large Language Models [48.276461194773354]
  We introduce ToolSpectrum, a benchmark designed to evaluate large language models' capabilities in personalized tool utilization.<n>We formalize two key dimensions of personalization, user profile and environmental factors, and analyze their individual and synergistic impacts on tool utilization.<n>Our findings underscore the necessity of context-aware personalization in tool-augmented LLMs and reveal critical limitations for current models.
  arXiv  Detail & Related papers  (2025-05-19T14:30:46Z)
• Human-Centered Design for AI-based Automatically Generated Assessment Reports: A Systematic Review [4.974197456441281]
  This study emphasizes the importance of reducing teachers' cognitive demands through user-centered and intuitive designs.<n>It highlights the potential of diverse information presentation formats such as text, visual aids, and plots and advanced functionalities such as live and interactive features to enhance usability.<n>The framework aims to address challenges in engaging teachers with technology-enhanced assessment results, facilitating data-driven decision-making, and providing personalized feedback to improve the teaching and learning process.
  arXiv  Detail & Related papers  (2024-12-30T16:20:07Z)
• From Exploration to Mastery: Enabling LLMs to Master Tools via Self-Driven Interactions [60.733557487886635]
  This paper focuses on bridging the comprehension gap between Large Language Models and external tools.<n>We propose a novel framework, DRAFT, aimed at Dynamically Refining tool documentation.<n>This methodology pivots on an innovative trial-and-error approach, consisting of three distinct learning phases.
  arXiv  Detail & Related papers  (2024-10-10T17:58:44Z)
• T-Eval: Evaluating the Tool Utilization Capability of Large Language Models Step by Step [69.64348626180623]
  Large language models (LLM) have achieved remarkable performance on various NLP tasks. How to evaluate and analyze the tool-utilization capability of LLMs is still under-explored. We introduce T-Eval to evaluate the tool utilization capability step by step.
  arXiv  Detail & Related papers  (2023-12-21T17:02:06Z)
• Towards a Flexible User Interface for 'Quick and Dirty' Learning Analytics Indicator Design [0.0]
  There is a need for 'quick and dirty' methods to allow the low-cost design of LA indicators. We propose two approaches to support the flexible design of indicators, namely a task-driven approach and a data-driven approach.
  arXiv  Detail & Related papers  (2023-04-04T11:18:11Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.