Analysing Python Machine Learning Notebooks with Moose

• URL: http://arxiv.org/abs/2509.11748v1
• Date: Mon, 15 Sep 2025 09:59:49 GMT
• Title: Analysing Python Machine Learning Notebooks with Moose
• Abstract summary: Machine Learning (ML) code, particularly within notebooks, often exhibits lower quality compared to traditional software.<n>Bad practices arise at three distinct levels: general Python coding conventions, the organizational structure of the notebook itself, and ML-specific aspects such as API usage.<n>This paper introduces Vespucci Linter, a static analysis tool with multi-level capabilities, built on Moose.
• Score: 0.12249546377051435
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Machine Learning (ML) code, particularly within notebooks, often exhibits lower quality compared to traditional software. Bad practices arise at three distinct levels: general Python coding conventions, the organizational structure of the notebook itself, and ML-specific aspects such as reproducibility and correct API usage. However, existing analysis tools typically focus on only one of these levels and struggle to capture ML-specific semantics, limiting their ability to detect issues. This paper introduces Vespucci Linter, a static analysis tool with multi-level capabilities, built on Moose and designed to address this challenge. Leveraging a metamodeling approach that unifies the notebook's structural elements with Python code entities, our linter enables a more contextualized analysis to identify issues across all three levels. We implemented 22 linting rules derived from the literature and applied our tool to a corpus of 5,000 notebooks from the Kaggle platform. The results reveal violations at all levels, validating the relevance of our multi-level approach and demonstrating Vespucci Linter's potential to improve the quality and reliability of ML development in notebook environments.

Related papers

• On the Relation between Code Quality and Machine Learning Performance: A Large-scale Empirical Study [0.13381749415517016]
  We investigated the relationship between code quality and machine learning (ML) performance in notebooks.<n>General Python code quality is decoupled from ML performance, showing negligible or non-significant correlations across all observations.<n>Code expertise provides no information on quality or performance, but competition expertise correlates with better performance.
  arXiv  Detail & Related papers  (2026-09-08T12:52:54Z)
• HoWToBench: Holistic Evaluation for LLM's Capability in Human-level Writing using Tree of Writing [84.84959282480641]
  We propose Tree-of-Writing (ToW) to resolve the implicit inconsistency often found when LLM-as-a-judge aggregates all sub-features in text evaluation.<n>We also present HowToBench, a large-scale Chinese writing benchmark encompassing 12 genres and 1302 instructions across three task categories.
  arXiv  Detail & Related papers  (2026-04-21T04:26:39Z)
• VP-Bench: A Comprehensive Benchmark for Visual Prompting in Multimodal Large Language Models [62.486978131415924]
  We introduce VP-Bench, a benchmark for assessing MLLMs' capability in visual prompting and utilization.<n>VP-Bench employs a two-stage evaluation framework: Stage 1 examines models' ability to perceive VPs in natural scenes, using 30k attribute prompts spanning eight shapes and 355 combinations.<n> Stage 2 investigates the impact of VPs on downstream tasks, measuring their effectiveness in real-world problem-solving scenarios.
  arXiv  Detail & Related papers  (2025-11-14T16:06:25Z)
• From Code Changes to Quality Gains: An Empirical Study in Python ML Systems with PyQu [1.5259434915412646]
  We introduce PyQu, a novel tool to identify quality-enhancing commits with an average accuracy, precision, and recall of 0.84 and 0.85 of average F1 score.<n> 41% of the changes are newly discovered by our study and have not been identified by state-of-the-art Python changes detection tools.
  arXiv  Detail & Related papers  (2025-11-04T18:55:19Z)
• Towards a Holistic and Automated Evaluation Framework for Multi-Level Comprehension of LLMs in Book-Length Contexts [19.640586886024952]
  HAMLET is a framework for evaluating the long-context comprehension of large language models.<n>It structures texts into a three-level key-fact hierarchy at root, branch, and leaf-levels.<n>It employs query-focused summarization to evaluate how well models recall and faithfully represent information at each level.
  arXiv  Detail & Related papers  (2025-08-27T05:23:22Z)
• Evaluating Large Language Model with Knowledge Oriented Language Specific Simple Question Answering [73.73820209993515]
  We introduce KoLasSimpleQA, the first benchmark evaluating the multilingual factual ability of Large Language Models (LLMs)<n>Inspired by existing research, we created the question set with features such as single knowledge point coverage, absolute objectivity, unique answers, and temporal stability.<n>Results show significant performance differences between the two domains.
  arXiv  Detail & Related papers  (2025-05-22T12:27:02Z)
• AutoDetect: Towards a Unified Framework for Automated Weakness Detection in Large Language Models [95.09157454599605]
  Large Language Models (LLMs) are becoming increasingly powerful, but they still exhibit significant but subtle weaknesses.<n>Traditional benchmarking approaches cannot thoroughly pinpoint specific model deficiencies.<n>We introduce a unified framework, AutoDetect, to automatically expose weaknesses in LLMs across various tasks.
  arXiv  Detail & Related papers  (2024-06-24T15:16:45Z)
• Benchmarking Uncertainty Quantification Methods for Large Language Models with LM-Polygraph [83.90988015005934]
  Uncertainty quantification is a key element of machine learning applications.<n>We introduce a novel benchmark that implements a collection of state-of-the-art UQ baselines.<n>We conduct a large-scale empirical investigation of UQ and normalization techniques across eleven tasks, identifying the most effective approaches.
  arXiv  Detail & Related papers  (2024-06-21T20:06:31Z)
• Bug In the Code Stack: Can LLMs Find Bugs in Large Python Code Stacks [1.3586572110652484]
  This study explores the capabilities of Large Language Models (LLMs) in retrieving contextual information from large text documents. Our benchmark, Bug In The Code Stack (BICS), is designed to assess the ability of LLMs to identify simple syntax bugs within large source code. Our findings reveal three key insights: (1) code-based environments pose significantly more challenge compared to text-based environments for retrieval tasks, (2) there is a substantial performance disparity among different models, and (3) there is a notable correlation between longer context lengths and performance degradation.
  arXiv  Detail & Related papers  (2024-06-21T17:37:10Z)
• PythonSaga: Redefining the Benchmark to Evaluate Code Generating LLMs [1.9207412600219353]
  We evaluate two popular benchmarks for Python code generation, analyzing their diversity and difficulty. Our findings unveil a critical bias towards a limited set of programming concepts, neglecting most of the other concepts entirely. We propose a novel benchmark, PythonSaga, featuring 185 hand-crafted prompts on a balanced representation of 38 programming concepts.
  arXiv  Detail & Related papers  (2024-01-08T12:36:43Z)
• SEED-Bench-2: Benchmarking Multimodal Large Language Models [67.28089415198338]
  Multimodal large language models (MLLMs) have recently demonstrated exceptional capabilities in generating not only texts but also images given interleaved multimodal inputs. SEED-Bench-2 comprises 24K multiple-choice questions with accurate human annotations, which spans 27 dimensions. We evaluate the performance of 23 prominent open-source MLLMs and summarize valuable observations.
  arXiv  Detail & Related papers  (2023-11-28T05:53:55Z)
• ML-Bench: Evaluating Large Language Models and Agents for Machine Learning Tasks on Repository-Level Code [76.84199699772903]
  ML-Bench is a benchmark rooted in real-world programming applications that leverage existing code repositories to perform tasks. To evaluate both Large Language Models (LLMs) and AI agents, two setups are employed: ML-LLM-Bench for assessing LLMs' text-to-code conversion within a predefined deployment environment, and ML-Agent-Bench for testing autonomous agents in an end-to-end task execution within a Linux sandbox environment.
  arXiv  Detail & Related papers  (2023-11-16T12:03:21Z)
• MAgIC: Investigation of Large Language Model Powered Multi-Agent in Cognition, Adaptability, Rationality and Collaboration [98.18244218156492]
  Large Language Models (LLMs) have significantly advanced natural language processing.<n>As their applications expand into multi-agent environments, there arises a need for a comprehensive evaluation framework.<n>This work introduces a novel competition-based benchmark framework to assess LLMs within multi-agent settings.
  arXiv  Detail & Related papers  (2023-11-14T21:46:27Z)
• L2CEval: Evaluating Language-to-Code Generation Capabilities of Large Language Models [102.00201523306986]
  We present L2CEval, a systematic evaluation of the language-to-code generation capabilities of large language models (LLMs) We analyze the factors that potentially affect their performance, such as model size, pretraining data, instruction tuning, and different prompting methods. In addition to assessing model performance, we measure confidence calibration for the models and conduct human evaluations of the output programs.
  arXiv  Detail & Related papers  (2023-09-29T17:57:00Z)
• Towards Understanding Machine Learning Testing in Practise [23.535630175567146]
  We propose to study visualisations of Machine Learning pipelines by mining Jupyter notebooks. First, gather general insights and trends using a qualitative study of a smaller sample of notebooks. And then use the knowledge gained from the qualitative study to design an empirical study using a larger sample of notebooks.
  arXiv  Detail & Related papers  (2023-05-08T18:52:26Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.