QUIS: Question-guided Insights Generation for Automated Exploratory Data Analysis

• URL: http://arxiv.org/abs/2410.10270v3
• Date: Mon, 21 Oct 2024 08:13:45 GMT
• Title: QUIS: Question-guided Insights Generation for Automated Exploratory Data Analysis
• Authors: Abhijit Manatkar, Ashlesha Akella, Parthivi Gupta, Krishnasuri Narayanam,
• Abstract summary: We introduce QUIS, a fully automated EDA system that operates in two stages: insight generation (ISGen) driven by question generation (QUGen) The ISGen module analyzes data to produce multiple relevant insights in response to each question, requiring no prior training and enabling QUIS to adapt to new datasets.
• Score: 1.9521598508325781
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Discovering meaningful insights from a large dataset, known as Exploratory Data Analysis (EDA), is a challenging task that requires thorough exploration and analysis of the data. Automated Data Exploration (ADE) systems use goal-oriented methods with Large Language Models and Reinforcement Learning towards full automation. However, these methods require human involvement to anticipate goals that may limit insight extraction, while fully automated systems demand significant computational resources and retraining for new datasets. We introduce QUIS, a fully automated EDA system that operates in two stages: insight generation (ISGen) driven by question generation (QUGen). The QUGen module generates questions in iterations, refining them from previous iterations to enhance coverage without human intervention or manually curated examples. The ISGen module analyzes data to produce multiple relevant insights in response to each question, requiring no prior training and enabling QUIS to adapt to new datasets.

Related papers

• ILAEDA: An Imitation Learning Based Approach for Automatic Exploratory Data Analysis [5.012314384895538]
  We argue that not all of the essential features of what makes an operation important can be accurately captured mathematically using rewards. We propose an AutoEDA model trained through imitation learning from expert EDA sessions, bypassing the need for manually defined interestingness measures. Our method outperforms the existing state-of-the-art end-to-end EDA approach on benchmarks by upto 3x, showing strong performance and generalization.
  arXiv  Detail & Related papers  (2024-10-15T04:56:13Z)
• AIDE: An Automatic Data Engine for Object Detection in Autonomous Driving [68.73885845181242]
  We propose an Automatic Data Engine (AIDE) that automatically identifies issues, efficiently curates data, improves the model through auto-labeling, and verifies the model through generation of diverse scenarios. We further establish a benchmark for open-world detection on AV datasets to comprehensively evaluate various learning paradigms, demonstrating our method's superior performance at a reduced cost.
  arXiv  Detail & Related papers  (2024-03-26T04:27:56Z)
• Automated data processing and feature engineering for deep learning and big data applications: a survey [0.0]
  Modern approach to artificial intelligence (AI) aims to design algorithms that learn directly from data. Not all data processing tasks in conventional deep learning pipelines have been automated.
  arXiv  Detail & Related papers  (2024-03-18T01:07:48Z)
• DACO: Towards Application-Driven and Comprehensive Data Analysis via Code Generation [83.30006900263744]
  Data analysis is a crucial analytical process to generate in-depth studies and conclusive insights. We propose to automatically generate high-quality answer annotations leveraging the code-generation capabilities of LLMs. Our DACO-RL algorithm is evaluated by human annotators to produce more helpful answers than SFT model in 57.72% cases.
  arXiv  Detail & Related papers  (2024-03-04T22:47:58Z)
• Automatic Question-Answer Generation for Long-Tail Knowledge [65.11554185687258]
  We propose an automatic approach to generate specialized QA datasets for tail entities. We conduct extensive experiments by employing pretrained LLMs on our newly generated long-tail QA datasets.
  arXiv  Detail & Related papers  (2024-03-03T03:06:31Z)
• Design & Implementation of Automatic Machine Condition Monitoring and Maintenance System in Limited Resource Situations [0.0]
  In the era of the fourth industrial revolution, it is essential to automate fault detection and diagnosis of machineries. Some machines health monitoring systems are used globally but they are expensive and need trained personnel to operate and analyse. Predictive maintenance and occupational health and safety culture are not available due to inadequate infrastructure, lack of skilled manpower, financial crisis, and others in developing countries.
  arXiv  Detail & Related papers  (2024-01-22T08:06:04Z)
• AutoAct: Automatic Agent Learning from Scratch for QA via Self-Planning [54.47116888545878]
  AutoAct is an automatic agent learning framework for QA. It does not rely on large-scale annotated data and synthetic planning trajectories from closed-source models.
  arXiv  Detail & Related papers  (2024-01-10T16:57:24Z)
• Open-sourced Data Ecosystem in Autonomous Driving: the Present and Future [130.87142103774752]
  This review systematically assesses over seventy open-source autonomous driving datasets. It offers insights into various aspects, such as the principles underlying the creation of high-quality datasets. It also delves into the scientific and technical challenges that warrant resolution.
  arXiv  Detail & Related papers  (2023-12-06T10:46:53Z)
• STAR: Boosting Low-Resource Information Extraction by Structure-to-Text Data Generation with Large Language Models [56.27786433792638]
  STAR is a data generation method that leverages Large Language Models (LLMs) to synthesize data instances. We design fine-grained step-by-step instructions to obtain the initial data instances. Our experiments show that the data generated by STAR significantly improve the performance of low-resource event extraction and relation extraction tasks.
  arXiv  Detail & Related papers  (2023-05-24T12:15:19Z)
• Deep Transfer Learning for Automatic Speech Recognition: Towards Better Generalization [3.6393183544320236]
  Speech recognition has become an important challenge when using deep learning (DL) It requires large-scale training datasets and high computational and storage resources. Deep transfer learning (DTL) has been introduced to overcome these issues.
  arXiv  Detail & Related papers  (2023-04-27T21:08:05Z)
• How Can Subgroup Discovery Help AIOps? [0.0]
  We study how Subgroup Discovery can help AIOps. This project involves both data mining researchers and practitioners from Infologic, a French software editor.
  arXiv  Detail & Related papers  (2021-09-10T14:41:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.