Scaling Inter-procedural Dataflow Analysis on the Cloud

• URL: http://arxiv.org/abs/2412.12579v1
• Date: Tue, 17 Dec 2024 06:18:56 GMT
• Title: Scaling Inter-procedural Dataflow Analysis on the Cloud
• Authors: Zewen Sun, Yujin Zhang, Duanchen Xu, Yiyu Zhang, Yun Qi, Yueyang Wang, Yi Li, Zhaokang Wang, Yue Li, Xuandong Li, Zhiqiang Zuo, Qingda Lu, Wenwen Peng, Shengjian Guo,
• Abstract summary: We develop a distributed framework called BigDataflow running on a large-scale cluster.<n>BigDataflow can finish analyzing the program of millions lines of code in minutes.
• Score: 19.562864760293955
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Apart from forming the backbone of compiler optimization, static dataflow analysis has been widely applied in a vast variety of applications, such as bug detection, privacy analysis, program comprehension, etc. Despite its importance, performing interprocedural dataflow analysis on large-scale programs is well known to be challenging. In this paper, we propose a novel distributed analysis framework supporting the general interprocedural dataflow analysis. Inspired by large-scale graph processing, we devise dedicated distributed worklist algorithms for both whole-program analysis and incremental analysis. We implement these algorithms and develop a distributed framework called BigDataflow running on a large-scale cluster. The experimental results validate the promising performance of BigDataflow -- BigDataflow can finish analyzing the program of millions lines of code in minutes. Compared with the state-of-the-art, BigDataflow achieves much more analysis efficiency.

Related papers

• Data Analysis Prediction over Multiple Unseen Datasets: A Vector Embedding Approach [0.3683202928838613]
  We propose a novel methodology that infers the outcome of analytics operators by creating a model from datasets similar to the queried one. Our model can project different real-world scenarios to a lower vector embedding representation and distinguish between them.
  arXiv  Detail & Related papers  (2025-02-24T11:21:08Z)
• Research on the Application of Spark Streaming Real-Time Data Analysis System and large language model Intelligent Agents [1.4582633500696451]
  This study explores the integration of Agent AI with LangGraph to enhance real-time data analysis systems in big data environments. The proposed framework overcomes limitations of static, inefficient stateful computations, and lack of human intervention. System architecture incorporates Apache Spark Streaming, Kafka, and LangGraph to create a high-performance sentiment analysis system.
  arXiv  Detail & Related papers  (2024-12-10T05:51:11Z)
• LLMDFA: Analyzing Dataflow in Code with Large Language Models [8.92611389987991]
  This paper presents LLMDFA, a compilation-free and customizable dataflow analysis framework. We decompose the problem into several subtasks and introduce a series of novel strategies. On average, LLMDFA achieves 87.10% precision and 80.77% recall, surpassing existing techniques with F1 score improvements of up to 0.35.
  arXiv  Detail & Related papers  (2024-02-16T15:21:35Z)
• Scalable Batch Acquisition for Deep Bayesian Active Learning [70.68403899432198]
  In deep active learning, it is important to choose multiple examples to markup at each step. Existing solutions to this problem, such as BatchBALD, have significant limitations in selecting a large number of examples. We present the Large BatchBALD algorithm, which aims to achieve comparable quality while being more computationally efficient.
  arXiv  Detail & Related papers  (2023-01-13T11:45:17Z)
• Where Is My Training Bottleneck? Hidden Trade-Offs in Deep Learning Preprocessing Pipelines [77.45213180689952]
  Preprocessing pipelines in deep learning aim to provide sufficient data throughput to keep the training processes busy. We introduce a new perspective on efficiently preparing datasets for end-to-end deep learning pipelines. We obtain an increased throughput of 3x to 13x compared to an untuned system.
  arXiv  Detail & Related papers  (2022-02-17T14:31:58Z)
• Enel: Context-Aware Dynamic Scaling of Distributed Dataflow Jobs using Graph Propagation [52.9168275057997]
  This paper presents Enel, a novel dynamic scaling approach that uses message propagation on an attributed graph to model dataflow jobs. We show that Enel is able to identify effective rescaling actions, reacting for instance to node failures, and can be reused across different execution contexts.
  arXiv  Detail & Related papers  (2021-08-27T10:21:08Z)
• Providing Meaningful Data Summarizations Using Examplar-based Clustering in Industry 4.0 [67.80123919697971]
  We show, that our GPU implementation provides speedups of up to 72x using single-precision and up to 452x using half-precision compared to conventional CPU algorithms. We apply our algorithm to real-world data from injection molding manufacturing processes and discuss how found summaries help with steering this specific process to cut costs and reduce the manufacturing of bad parts.
  arXiv  Detail & Related papers  (2021-05-25T15:55:14Z)
• Divide-and-conquer methods for big data analysis [0.0]
  Divide-and-conquer methodology refers to a multiple-step process. This article reviews some recently developments of divide-and-conquer methods in a variety of settings.
  arXiv  Detail & Related papers  (2021-02-22T04:40:55Z)
• A Forward Backward Greedy approach for Sparse Multiscale Learning [0.0]
  We propose a feature driven Reproducing Kernel Hilbert space (RKHS) for which the associated kernel has a weighted multiscale structure. For generating approximations in this space, we provide a practical forward-backward algorithm that is shown to greedily construct a set of basis functions having a multiscale structure. We analyze the performance of the approach on a variety of simulation and real data sets.
  arXiv  Detail & Related papers  (2021-02-14T04:22:52Z)
• Stochastic Optimization with Laggard Data Pipelines [65.20044914532221]
  We show that "dataechoed" extensions of common optimization methods exhibit provable improvements over their synchronous counterparts. Specifically, we show that in convex optimization with minibatches, data echoing affords speedups on the curvature-dominated part of the convergence rate, while maintaining the optimal statistical rate.
  arXiv  Detail & Related papers  (2020-10-26T14:55:31Z)
• Real-Time Regression with Dividing Local Gaussian Processes [62.01822866877782]
  Local Gaussian processes are a novel, computationally efficient modeling approach based on Gaussian process regression. Due to an iterative, data-driven division of the input space, they achieve a sublinear computational complexity in the total number of training points in practice. A numerical evaluation on real-world data sets shows their advantages over other state-of-the-art methods in terms of accuracy as well as prediction and update speed.
  arXiv  Detail & Related papers  (2020-06-16T18:43:31Z)
• Interpretable Feature Learning in Multivariate Big Data Analysis for Network Monitoring [0.4342241136871849]
  We present an extension of the Multivariate Big Data Analysis (MBDA) methodology, a recently proposed interpretable data analysis tool. We propose a solution to the automatic derivation of features, a cornerstone step for the application of MBDA when the amount of data is massive. We apply the extended MBDA to two case studies: UGR'16, a benchmark flow-based real-traffic dataset for anomaly detection, and Dartmouth'18, the longest and largest Wi-Fi trace known to date.
  arXiv  Detail & Related papers  (2019-07-05T04:51:49Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.