Understanding and Supporting Debugging Workflows in Multiverse Analysis

• URL: http://arxiv.org/abs/2210.03804v3
• Date: Sun, 4 Jun 2023 07:23:56 GMT
• Title: Understanding and Supporting Debugging Workflows in Multiverse Analysis
• Authors: Ken Gu, Eunice Jun, and Tim Althoff
• Abstract summary: Multiverse analysis is a paradigm for statistical analysis that considers all combinations of reasonable analysis choices in parallel. Recent tools help analysts specify multiverse analyses, but they remain difficult to use in practice. We develop a command-line interface tool, Multiverse Debugger, which helps diagnose bugs in the multiverse and propagate.
• Score: 12.23386451120784
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Multiverse analysis, a paradigm for statistical analysis that considers all combinations of reasonable analysis choices in parallel, promises to improve transparency and reproducibility. Although recent tools help analysts specify multiverse analyses, they remain difficult to use in practice. In this work, we identify debugging as a key barrier due to the latency from running analyses to detecting bugs and the scale of metadata processing needed to diagnose a bug. To address these challenges, we prototype a command-line interface tool, Multiverse Debugger, which helps diagnose bugs in the multiverse and propagate fixes. In a qualitative lab study (n=13), we use Multiverse Debugger as a probe to develop a model of debugging workflows and identify specific challenges, including difficulty in understanding the multiverse's composition. We conclude with design implications for future multiverse analysis authoring systems.

Related papers

• Leveraging Slither and Interval Analysis to build a Static Analysis Tool [0.0]
  This paper presents our progress toward finding defects that are sometimes not detected or completely detected by state-of-the-art analysis tools. We developed a working solution built on top of Slither that uses interval analysis to evaluate the contract state during the execution of each instruction.
  arXiv  Detail & Related papers  (2024-10-31T09:28:09Z)
• Data Analysis in the Era of Generative AI [56.44807642944589]
  This paper explores the potential of AI-powered tools to reshape data analysis, focusing on design considerations and challenges. We explore how the emergence of large language and multimodal models offers new opportunities to enhance various stages of data analysis workflow. We then examine human-centered design principles that facilitate intuitive interactions, build user trust, and streamline the AI-assisted analysis workflow across multiple apps.
  arXiv  Detail & Related papers  (2024-09-27T06:31:03Z)
• UCB-driven Utility Function Search for Multi-objective Reinforcement Learning [75.11267478778295]
  In Multi-objective Reinforcement Learning (MORL) agents are tasked with optimising decision-making behaviours. We focus on the case of linear utility functions parameterised by weight vectors w. We introduce a method based on Upper Confidence Bound to efficiently search for the most promising weight vectors during different stages of the learning process.
  arXiv  Detail & Related papers  (2024-05-01T09:34:42Z)
• MultiDimEr: a multi-dimensional bug analyzEr [5.318531077716712]
  We categorize and visualize dimensions of bug reports to identify accruing technical debt. This evidence can serve practitioners and decision makers not only as an argumentative basis for steering improvement efforts, but also as a starting point for root cause analysis.
  arXiv  Detail & Related papers  (2024-02-16T16:00:42Z)
• When Dataflow Analysis Meets Large Language Models [9.458251511218817]
  This paper introduces LLMDFA, an LLM-powered dataflow analysis framework that analyzes arbitrary code snippets without requiring a compilation infrastructure. Inspired by summary-based dataflow analysis, LLMDFA decomposes the problem into three sub-problems, which are effectively resolved by several essential strategies. Our evaluation has shown that the design can mitigate the hallucination and improve the reasoning ability, obtaining high precision and recall in detecting dataflow-related bugs.
  arXiv  Detail & Related papers  (2024-02-16T15:21:35Z)
• Enhancing Large Language Models in Coding Through Multi-Perspective Self-Consistency [127.97467912117652]
  Large language models (LLMs) have exhibited remarkable ability in code generation. However, generating the correct solution in a single attempt still remains a challenge. We propose the Multi-Perspective Self-Consistency (MPSC) framework incorporating both inter- and intra-consistency.
  arXiv  Detail & Related papers  (2023-09-29T14:23:26Z)
• The Hitchhiker's Guide to Program Analysis: A Journey with Large Language Models [18.026567399243]
  Large Language Models (LLMs) offer a promising alternative to static analysis. In this paper, we take a deep dive into the open space of LLM-assisted static analysis. We develop LLift, a fully automated framework that interfaces with both a static analysis tool and an LLM.
  arXiv  Detail & Related papers  (2023-08-01T02:57:43Z)
• Improved Compositional Generalization by Generating Demonstrations for Meta-Learning [53.818234285773165]
  We show substantially improved performance on a previously unsolved compositional behaviour split without a loss of performance on other splits. In this case, searching for relevant demonstrations even with an oracle function is not sufficient to attain good performance when using meta-learning.
  arXiv  Detail & Related papers  (2023-05-22T14:58:54Z)
• Transformer-based Multi-Aspect Modeling for Multi-Aspect Multi-Sentiment Analysis [56.893393134328996]
  We propose a novel Transformer-based Multi-aspect Modeling scheme (TMM), which can capture potential relations between multiple aspects and simultaneously detect the sentiment of all aspects in a sentence. Our method achieves noticeable improvements compared with strong baselines such as BERT and RoBERTa.
  arXiv  Detail & Related papers  (2020-11-01T11:06:31Z)
• Total Deep Variation: A Stable Regularizer for Inverse Problems [71.90933869570914]
  We introduce the data-driven general-purpose total deep variation regularizer. In its core, a convolutional neural network extracts local features on multiple scales and in successive blocks. We achieve state-of-the-art results for numerous imaging tasks.
  arXiv  Detail & Related papers  (2020-06-15T21:54:15Z)
• Multi-view Alignment and Generation in CCA via Consistent Latent Encoding [34.57297855115903]
  Multi-view alignment is critical in many real-world multi-view applications. This paper studies multi-view alignment from the Bayesian perspective. We present Adversarial CCA (ACCA) which achieves consistent latent encodings.
  arXiv  Detail & Related papers  (2020-05-24T10:50:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.