Recommending Usability Improvements with Multimodal Large Language Models

• URL: http://arxiv.org/abs/2604.25420v1
• Date: Tue, 28 Apr 2026 09:29:13 GMT
• Title: Recommending Usability Improvements with Multimodal Large Language Models
• Abstract summary: We present a novel approach that uses limited application context and screen recordings of user interactions as input to an MLLM.<n>The model automatically identifies and describes usability issues based on Nielsens usabilitys, and provides corresponding explanations and improvement recommendations.
• Score: 40.82344810016507
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Usability describes quality attributes of application user interfaces that determine how effectively users can interact with them. Traditional usability evaluation methods require considerable expertise and resources, which can be challenging, especially for small teams and organizations. Automating usability evaluation could make it more accessible and help to improve the user experience. The recent emergence of powerful multimodal large language models (MLLMs) has opened new opportunities for automating usability evaluation and recommendation of improvements. These models can process visual inputs such as images and videos alongside textual context, which enables the identification of usability issues and the generation of actionable suggestions to resolve these issues. In this paper, we present a novel automated approach that uses limited application context and screen recordings of user interactions as input to an MLLM. The model automatically identifies and describes usability issues based on Nielsens usability heuristics, and provides corresponding explanations and improvement recommendations. To reduce the developer effort of manual prioritization, the recommendations are ranked by severity. The quality and practical usefulness of the generated recommendations were evaluated based on a user study that involved software engineers as participants. The evaluation focused on the highest-ranked suggestions provided by the model. The results demonstrate the potential of our approach to provide low-effort usability improvement recommendations. This makes it a promising complement to traditional evaluation methods, especially in settings with limited access to usability experts. In this sense, the approach serves as a basis for future integration into development tools to enable automated usability evaluation within software engineering workflows.

Related papers

• LLM-Based User Personas for Recommendations at Scale [16.740779038125524]
  Large Language Models (LLMs) offer unprecedented potential for enhancing recommendation systems through their world knowledge and reasoning capabilities.<n>Existing approaches often rely on structured IDs or offline processing, limiting semantic richness, real-time adaptability, and user-facing interpretability.<n>This work bridges the gap between high-level semantic understanding and industrial-scale recommendation, paving the way for more dynamic, explainable, and satisfying personalized experiences.
  arXiv  Detail & Related papers  (2026-06-10T15:18:32Z)
• RecThinker: An Agentic Framework for Tool-Augmented Reasoning in Recommendation [63.74915464611075]
  RecThinker is an agentic framework for tool-augmented reasoning in recommendation.<n>We develop a suite of specialized tools for RecThinker, enabling the model to acquire user item-side and collaborative information.
  arXiv  Detail & Related papers  (2026-03-10T16:07:17Z)
• Towards LLM-Based Usability Analysis for Recommender User Interfaces [41.966962052550656]
  We explore the potential of multimodal large language models to assess the usability of recommender system interfaces.<n>We take user interface screenshots from multiple recommender platforms to cover both preference elicitation and recommendation presentation scenarios.
  arXiv  Detail & Related papers  (2025-11-18T11:05:13Z)
• Methods for evaluating software accessibility [0.0]
  A more detailed and practically oriented accessibility assessment methodology has been proposed.<n>An analysis of the accessibility of the main pages of Vasyl Stefanyk Precarpathian National University's website was conducted.
  arXiv  Detail & Related papers  (2025-09-27T19:46:10Z)
• Towards Recommending Usability Improvements with Multimodal Large Language Models [40.77787659104315]
  Common evaluation methods, such as usability testing and inspection, are effective but resource-intensive and require expert involvement.<n>Recent advances in multimodal LLMs offer promising opportunities to automate usability evaluation processes.<n>Our findings indicate the potential of LLMs to enable faster and more cost-effective usability evaluation.
  arXiv  Detail & Related papers  (2025-08-22T07:38:37Z)
• Multi-agents based User Values Mining for Recommendation [52.26100802380767]
  We propose a zero-shot multi-LLM collaborative framework for effective and accurate user value extraction.<n>We apply text summarization techniques to condense item content while preserving essential meaning.<n>To mitigate hallucinations, we introduce two specialized agent roles: evaluators and supervisors.
  arXiv  Detail & Related papers  (2025-05-02T04:01:31Z)
• Explainable Recommendation with Simulated Human Feedback [8.532115411106068]
  We propose a novel human-like feedback-driven optimization framework for explainable recommendations.<n>This framework employs a dynamic interactive optimization mechanism for achieving human-centered explainable requirements without incurring high labor costs.<n>In particular, we propose to utilize large language models (LLMs) as human simulators to predict human-like feedback for guiding the learning process.
  arXiv  Detail & Related papers  (2025-04-19T02:46:10Z)
• Re-evaluating Open-ended Evaluation of Large Language Models [50.23008729038318]
  We show that the current Elo-based rating systems can be susceptible to and even reinforce biases in data, intentional or accidental.<n>We propose evaluation as a 3-player game, and introduce novel game-theoretic solution concepts to ensure robustness to redundancy.
  arXiv  Detail & Related papers  (2025-02-27T15:07:47Z)
• Interactive Agents to Overcome Ambiguity in Software Engineering [61.40183840499932]
  AI agents are increasingly being deployed to automate tasks, often based on ambiguous and underspecified user instructions.<n>Making unwarranted assumptions and failing to ask clarifying questions can lead to suboptimal outcomes.<n>We study the ability of LLM agents to handle ambiguous instructions in interactive code generation settings by evaluating proprietary and open-weight models on their performance.
  arXiv  Detail & Related papers  (2025-02-18T17:12:26Z)
• Uncertainty-Aware Explainable Recommendation with Large Language Models [15.229417987212631]
  We develop a model that utilizes the ID vectors of user and item inputs as prompts for GPT-2. We employ a joint training mechanism within a multi-task learning framework to optimize both the recommendation task and explanation task. Our method achieves 1.59 DIV, 0.57 USR and 0.41 FCR on the Yelp, TripAdvisor and Amazon dataset respectively.
  arXiv  Detail & Related papers  (2024-01-31T14:06:26Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.