From Technical Excellence to Practical Adoption: Lessons Learned Building an ML-Enhanced Trace Analysis Tool

• URL: http://arxiv.org/abs/2508.01430v1
• Date: Sat, 02 Aug 2025 16:32:46 GMT
• Title: From Technical Excellence to Practical Adoption: Lessons Learned Building an ML-Enhanced Trace Analysis Tool
• Authors: Kaveh Shahedi, Matthew Khouzam, Heng Li, Maxime Lamothe, Foutse Khomh,
• Abstract summary: We investigated barriers to trace analysis adoption in industrial settings.<n>We found that practitioners struggled with translating expert knowledge into actionable insights.<n> TMLL addresses this through adoption-focused design that embeds expert knowledge in interfaces.
• Score: 14.908341749591594
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: System tracing has become essential for understanding complex software behavior in modern systems, yet sophisticated trace analysis tools face significant adoption gaps in industrial settings. Through a year-long collaboration with Ericsson Montr\'eal, developing TMLL (Trace-Server Machine Learning Library, now in the Eclipse Foundation), we investigated barriers to trace analysis adoption. Contrary to assumptions about complexity or automation needs, practitioners struggled with translating expert knowledge into actionable insights, integrating analysis into their workflows, and trusting automated results they could not validate. We identified what we called the Excellence Paradox: technical excellence can actively impede adoption when conflicting with usability, transparency, and practitioner trust. TMLL addresses this through adoption-focused design that embeds expert knowledge in interfaces, provides transparent explanations, and enables incremental adoption. Validation through Ericsson's experts' feedback, Eclipse Foundation's integration, and a survey of 40 industry and academic professionals revealed consistent patterns: survey results showed that 77.5% prioritize quality and trust in results over technical sophistication, while 67.5% prefer semi-automated analysis with user control, findings supported by qualitative feedback from industrial collaboration and external peer review. Results validate three core principles: cognitive compatibility, embedded expertise, and transparency-based trust. This challenges conventional capability-focused tool development, demonstrating that sustainable adoption requires reorientation toward adoption-focused design with actionable implications for automated software engineering tools.

Related papers

• Software Fairness Testing in Practice [0.21427777919040417]
  This study investigates how software professionals test AI-powered systems for fairness through interviews with 22 practitioners working on AI and ML projects.<n>Our findings highlight a significant gap between theoretical fairness concepts and industry practice.<n>Key challenges include data quality and diversity, time constraints, defining effective metrics, and ensuring model interoperability.
  arXiv  Detail & Related papers  (2025-06-20T16:03:02Z)
• OpenUnlearning: Accelerating LLM Unlearning via Unified Benchmarking of Methods and Metrics [101.78963920333342]
  We introduce OpenUnlearning, a standardized framework for benchmarking large language models (LLMs) unlearning methods and metrics.<n>OpenUnlearning integrates 9 unlearning algorithms and 16 diverse evaluations across 3 leading benchmarks.<n>We also benchmark diverse unlearning methods and provide a comparative analysis against an extensive evaluation suite.
  arXiv  Detail & Related papers  (2025-06-14T20:16:37Z)
• Hybrid Reasoning for Perception, Explanation, and Autonomous Action in Manufacturing [0.0]
  CIPHER is a vision-language-action (VLA) model framework aiming to replicate human-like reasoning for industrial control.<n>It integrates a process expert, a regression model enabling quantitative characterization of system states.<n>It interprets visual or textual inputs from process monitoring, explains its decisions, and autonomously generates precise machine instructions.
  arXiv  Detail & Related papers  (2025-06-10T05:37:33Z)
• A Survey on (M)LLM-Based GUI Agents [62.57899977018417]
  Graphical User Interface (GUI) Agents have emerged as a transformative paradigm in human-computer interaction.<n>Recent advances in large language models and multimodal learning have revolutionized GUI automation across desktop, mobile, and web platforms.<n>This survey identifies key technical challenges, including accurate element localization, effective knowledge retrieval, long-horizon planning, and safety-aware execution control.
  arXiv  Detail & Related papers  (2025-03-27T17:58:31Z)
• AI-Driven Sentiment Analytics: Unlocking Business Value in the E-Commerce Landscape [0.0]
  This paper presents an AI-driven sentiment analysis system designed specifically for e-commerce applications.<n>Our approach integrates traditional machine learning techniques with modern deep learning models, allowing for a more nuanced understanding of customer sentiment.<n> Experimental results show that our system outperforms standard sentiment analysis methods, achieving an accuracy of 89.7% on diverse, large-scale datasets.
  arXiv  Detail & Related papers  (2025-03-20T18:56:22Z)
• DeepPerception: Advancing R1-like Cognitive Visual Perception in MLLMs for Knowledge-Intensive Visual Grounding [61.26026947423187]
  Human experts excel at fine-grained visual discrimination by leveraging domain knowledge to refine perceptual features.<n>Current Multimodal Large Language Models (MLLMs) struggle to integrate reasoning into visual perception.<n>We propose DeepPerception, an MLLM enhanced with cognitive visual perception capabilities.
  arXiv  Detail & Related papers  (2025-03-17T04:06:34Z)
• Enhancing LLM Reliability via Explicit Knowledge Boundary Modeling [48.15636223774418]
  Large language models (LLMs) are prone to hallucination stemming from misaligned self-awareness.<n>We propose the Explicit Knowledge Boundary Modeling framework to integrate fast and slow reasoning systems to harmonize reliability and usability.
  arXiv  Detail & Related papers  (2025-03-04T03:16:02Z)
• From Critique to Clarity: A Pathway to Faithful and Personalized Code Explanations with Large Language Models [3.5176327869934747]
  This paper presents an innovative approach to generate faithful and personalized code explanations.<n>Our methodology integrates prompt enhancement, self-consuming mechanisms, personalized content customization, and interaction with external tools.<n>Our findings suggest that this approach significantly improves the quality and relevance of code explanations.
  arXiv  Detail & Related papers  (2024-12-08T09:02:04Z)
• The Role of Accuracy and Validation Effectiveness in Conversational Business Analytics [0.0]
  This study examines conversational business analytics, an approach that utilizes AI to address the technical competency gaps that hinder end users effectively using traditional self-service analytics. By facilitating natural language interactions, conversational business analytics aims to empower users to independently retrieve data and generate insights.
  arXiv  Detail & Related papers  (2024-11-18T23:58:24Z)
• Exploring Federated Unlearning: Review, Comparison, and Insights [101.64910079905566]
  federated unlearning enables the selective removal of data from models trained in federated systems.<n>This paper examines existing federated unlearning approaches, examining their algorithmic efficiency, impact on model accuracy, and effectiveness in preserving privacy.<n>We propose the OpenFederatedUnlearning framework, a unified benchmark for evaluating federated unlearning methods.
  arXiv  Detail & Related papers  (2023-10-30T01:34:33Z)
• Distributed intelligence on the Edge-to-Cloud Continuum: A systematic literature review [62.997667081978825]
  This review aims at providing a comprehensive vision of the main state-of-the-art libraries and frameworks for machine learning and data analytics available today. The main simulation, emulation, deployment systems, and testbeds for experimental research on the Edge-to-Cloud Continuum available today are also surveyed.
  arXiv  Detail & Related papers  (2022-04-29T08:06:05Z)
• Empowered and Embedded: Ethics and Agile Processes [60.63670249088117]
  We argue that ethical considerations need to be embedded into the (agile) software development process. We put emphasis on the possibility to implement ethical deliberations in already existing and well established agile software development processes.
  arXiv  Detail & Related papers  (2021-07-15T11:14:03Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.