Towards Understanding Machine Learning Testing in Practise

• URL: http://arxiv.org/abs/2305.04988v2
• Date: Mon, 22 May 2023 11:43:42 GMT
• Title: Towards Understanding Machine Learning Testing in Practise
• Authors: Arumoy Shome, Luis Cruz, Arie van Deursen
• Abstract summary: We propose to study visualisations of Machine Learning pipelines by mining Jupyter notebooks. First, gather general insights and trends using a qualitative study of a smaller sample of notebooks. And then use the knowledge gained from the qualitative study to design an empirical study using a larger sample of notebooks.
• Score: 23.535630175567146
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Visualisations drive all aspects of the Machine Learning (ML) Development Cycle but remain a vastly untapped resource by the research community. ML testing is a highly interactive and cognitive process which demands a human-in-the-loop approach. Besides writing tests for the code base, bulk of the evaluation requires application of domain expertise to generate and interpret visualisations. To gain a deeper insight into the process of testing ML systems, we propose to study visualisations of ML pipelines by mining Jupyter notebooks. We propose a two prong approach in conducting the analysis. First, gather general insights and trends using a qualitative study of a smaller sample of notebooks. And then use the knowledge gained from the qualitative study to design an empirical study using a larger sample of notebooks. Computational notebooks provide a rich source of information in three formats -- text, code and images. We hope to utilise existing work in image analysis and Natural Language Processing for text and code, to analyse the information present in notebooks. We hope to gain a new perspective into program comprehension and debugging in the context of ML testing.

Related papers

• Thematic Analysis with Open-Source Generative AI and Machine Learning: A New Method for Inductive Qualitative Codebook Development [0.0]
  We present the Generative AI-enabled Theme Organization and Structuring (GATOS) workflow. It uses open-source machine learning techniques, natural language processing tools, and generative text models to facilitate thematic analysis. We show that the GATOS workflow is able to identify themes in the text that were used to generate the original synthetic datasets.
  arXiv  Detail & Related papers  (2024-09-28T18:52:16Z)
• Rethinking Visual Prompting for Multimodal Large Language Models with External Knowledge [76.45868419402265]
  multimodal large language models (MLLMs) have made significant strides by training on vast high-quality image-text datasets. However, the inherent difficulty in explicitly conveying fine-grained or spatially dense information in text, such as masks, poses a challenge for MLLMs. This paper proposes a new visual prompt approach to integrate fine-grained external knowledge, gleaned from specialized vision models, into MLLMs.
  arXiv  Detail & Related papers  (2024-07-05T17:43:30Z)
• Assistive Image Annotation Systems with Deep Learning and Natural Language Capabilities: A Review [0.0]
  This paper explores AI-assistive deep learning image annotation systems that provide textual suggestions, captions, or descriptions of the input image to the annotator. We review various datasets and how they contribute to the training and evaluation of AI-assistive annotation systems. Despite the promising potential, there is limited publicly available work on AI-assistive image annotation with textual output capabilities.
  arXiv  Detail & Related papers  (2024-06-28T22:56:17Z)
• Towards Automatic Translation of Machine Learning Visual Insights to Analytical Assertions [23.535630175567146]
  We present our vision for developing an automated tool capable of translating visual properties observed in Machine Learning (ML) visualisations into Python assertions. The tool aims to streamline the process of manually verifying these visualisations in the ML development cycle, which is critical as real-world data and assumptions often change post-deployment.
  arXiv  Detail & Related papers  (2024-01-15T14:11:59Z)
• Large Language Models for Generative Information Extraction: A Survey [89.71273968283616]
  Large Language Models (LLMs) have demonstrated remarkable capabilities in text understanding and generation. We present an extensive overview by categorizing these works in terms of various IE subtasks and techniques. We empirically analyze the most advanced methods and discover the emerging trend of IE tasks with LLMs.
  arXiv  Detail & Related papers  (2023-12-29T14:25:22Z)
• A Machine Learning-oriented Survey on Tiny Machine Learning [9.690117347832722]
  The emergence of Tiny Machine Learning (TinyML) has positively revolutionized the field of Artificial Intelligence. TinyML carries an essential role within the fourth and fifth industrial revolutions in helping societies, economies, and individuals employ effective AI-infused computing technologies.
  arXiv  Detail & Related papers  (2023-09-21T09:47:12Z)
• Understanding ME? Multimodal Evaluation for Fine-grained Visual Commonsense [98.70218717851665]
  It is unclear whether the models really understand the visual scene and underlying commonsense knowledge due to limited evaluation data resources. We present a Multimodal Evaluation (ME) pipeline to automatically generate question-answer pairs to test models' understanding of the visual scene, text, and related knowledge. We then take a step further to show that training with the ME data boosts the model's performance in standard VCR evaluation.
  arXiv  Detail & Related papers  (2022-11-10T21:44:33Z)
• Align, Reason and Learn: Enhancing Medical Vision-and-Language Pre-training with Knowledge [68.90835997085557]
  We propose a systematic and effective approach to enhance structured medical knowledge from three perspectives. First, we align the representations of the vision encoder and the language encoder through knowledge. Second, we inject knowledge into the multi-modal fusion model to enable the model to perform reasoning using knowledge as the supplementation of the input image and text. Third, we guide the model to put emphasis on the most critical information in images and texts by designing knowledge-induced pretext tasks.
  arXiv  Detail & Related papers  (2022-09-15T08:00:01Z)
• A Transfer Learning Pipeline for Educational Resource Discovery with Application in Leading Paragraph Generation [71.92338855383238]
  We propose a pipeline that automates web resource discovery for novel domains. The pipeline achieves F1 scores of 0.94 and 0.82 when evaluated on two similar but novel target domains. This is the first study that considers various web resources for survey generation.
  arXiv  Detail & Related papers  (2022-01-07T03:35:40Z)
• Knowledge-Aware Meta-learning for Low-Resource Text Classification [87.89624590579903]
  This paper studies a low-resource text classification problem and bridges the gap between meta-training and meta-testing tasks. We propose KGML to introduce additional representation for each sentence learned from the extracted sentence-specific knowledge graph.
  arXiv  Detail & Related papers  (2021-09-10T07:20:43Z)
• Code2Image: Intelligent Code Analysis by Computer Vision Techniques and Application to Vulnerability Prediction [0.6091702876917281]
  We present a novel method to represent source code as image while preserving semantic and syntactic properties. The method makes it possible to directly enter the resulting image representation of source codes into deep learning (DL) algorithms as input. We demonstrate feasibility and effectiveness of our method by realizing a vulnerability prediction use case over a public dataset.
  arXiv  Detail & Related papers  (2021-05-07T09:10:20Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.