Towards Recommending Usability Improvements with Multimodal Large Language Models

• URL: http://arxiv.org/abs/2508.16165v1
• Date: Fri, 22 Aug 2025 07:38:37 GMT
• Title: Towards Recommending Usability Improvements with Multimodal Large Language Models
• Authors: Sebastian Lubos, Alexander Felfernig, Gerhard Leitner, Julian Schwazer,
• Abstract summary: Common evaluation methods, such as usability testing and inspection, are effective but resource-intensive and require expert involvement.<n>Recent advances in multimodal LLMs offer promising opportunities to automate usability evaluation processes.<n>Our findings indicate the potential of LLMs to enable faster and more cost-effective usability evaluation.
• Score: 40.77787659104315
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Usability describes a set of essential quality attributes of user interfaces (UI) that influence human-computer interaction. Common evaluation methods, such as usability testing and inspection, are effective but resource-intensive and require expert involvement. This makes them less accessible for smaller organizations. Recent advances in multimodal LLMs offer promising opportunities to automate usability evaluation processes partly by analyzing textual, visual, and structural aspects of software interfaces. To investigate this possibility, we formulate usability evaluation as a recommendation task, where multimodal LLMs rank usability issues by severity. We conducted an initial proof-of-concept study to compare LLM-generated usability improvement recommendations with usability expert assessments. Our findings indicate the potential of LLMs to enable faster and more cost-effective usability evaluation, which makes it a practical alternative in contexts with limited expert resources.

Related papers

• SelfAI: Building a Self-Training AI System with LLM Agents [79.10991818561907]
  SelfAI is a general multi-agent platform that combines a User Agent for translating high-level research objectives into standardized experimental configurations.<n>An Experiment Manager orchestrates parallel, fault-tolerant training across heterogeneous hardware while maintaining a structured knowledge base for continuous feedback.<n>Across regression, computer vision, scientific computing, medical imaging, and drug discovery benchmarks, SelfAI consistently achieves strong performance and reduces redundant trials.
  arXiv  Detail & Related papers  (2025-11-29T09:18:39Z)
• Towards LLM-Based Usability Analysis for Recommender User Interfaces [41.966962052550656]
  We explore the potential of multimodal large language models to assess the usability of recommender system interfaces.<n>We take user interface screenshots from multiple recommender platforms to cover both preference elicitation and recommendation presentation scenarios.
  arXiv  Detail & Related papers  (2025-11-18T11:05:13Z)
• LLM-based Relevance Assessment for Web-Scale Search Evaluation at Pinterest [3.306725465028306]
  We present our approach at Pinterest Search to automate relevance evaluation for online experiments using fine-tuned LLMs.<n>We rigorously validate the alignment between LLM-generated judgments and human annotations.<n>This approach leads to higher-quality relevance metrics and significantly reduces the Minimum Detectable Effect (MDE) in online experiment measurements.
  arXiv  Detail & Related papers  (2025-09-03T23:07:49Z)
• PanguIR Technical Report for NTCIR-18 AEOLLM Task [12.061652026366591]
  Large language models (LLMs) are increasingly critical and challenging to evaluate.<n>Manual evaluation, while comprehensive, is often costly and resource-intensive.<n>automatic evaluation offers greater scalability but is constrained by the limitations of its evaluation criteria.
  arXiv  Detail & Related papers  (2025-03-04T07:40:02Z)
• Reference-Guided Verdict: LLMs-as-Judges in Automatic Evaluation of Free-Form Text [12.879551933541345]
  Large Language Models (LLMs) are capable of generating human-like conversations. Conventional metrics like BLEU and ROUGE are inadequate for capturing the subtle semantics and contextual richness of such generative outputs. We propose a reference-guided verdict method that automates the evaluation process by leveraging multiple LLMs-as-judges.
  arXiv  Detail & Related papers  (2024-08-17T16:01:45Z)
• Large Multimodal Agents: A Survey [78.81459893884737]
  Large language models (LLMs) have achieved superior performance in powering text-based AI agents. There is an emerging research trend focused on extending these LLM-powered AI agents into the multimodal domain. This review aims to provide valuable insights and guidelines for future research in this rapidly evolving field.
  arXiv  Detail & Related papers  (2024-02-23T06:04:23Z)
• T-Eval: Evaluating the Tool Utilization Capability of Large Language Models Step by Step [69.64348626180623]
  Large language models (LLM) have achieved remarkable performance on various NLP tasks. How to evaluate and analyze the tool-utilization capability of LLMs is still under-explored. We introduce T-Eval to evaluate the tool utilization capability step by step.
  arXiv  Detail & Related papers  (2023-12-21T17:02:06Z)
• MINT: Evaluating LLMs in Multi-turn Interaction with Tools and Language Feedback [78.60644407028022]
  We introduce MINT, a benchmark that evaluates large language models' ability to solve tasks with multi-turn interactions. LLMs generally benefit from tools and language feedback, with performance gains of 1-8% for each turn of tool use. LLMs evaluated, supervised instruction-finetuning (SIFT) and reinforcement learning from human feedback (RLHF) generally hurt multi-turn capabilities.
  arXiv  Detail & Related papers  (2023-09-19T15:25:42Z)
• ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debate [57.71597869337909]
  We build a multi-agent referee team called ChatEval to autonomously discuss and evaluate the quality of generated responses from different models. Our analysis shows that ChatEval transcends mere textual scoring, offering a human-mimicking evaluation process for reliable assessments.
  arXiv  Detail & Related papers  (2023-08-14T15:13:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.