Towards an Integrated Platform for Big Data Analysis
- URL: http://arxiv.org/abs/2004.13021v1
- Date: Mon, 27 Apr 2020 03:15:23 GMT
- Title: Towards an Integrated Platform for Big Data Analysis
- Authors: Mahdi Bohlouli, Frank Schulz, Lefteris Angelis, David Pahor, Ivona
Brandic, David Atlan, Rosemary Tate
- Abstract summary: This paper presents the vision of an integrated plat-form for big data analysis that combines all these aspects.
Main benefits of this approach are an enhanced scalability of the whole platform, a better parameterization of algorithms, and an improved usability during the end-to-end data analysis process.
- Score: 4.5257812998381315
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: The amount of data in the world is expanding rapidly. Every day, huge amounts
of data are created by scientific experiments, companies, and end users'
activities. These large data sets have been labeled as "Big Data", and their
storage, processing and analysis presents a plethora of new challenges to
computer science researchers and IT professionals. In addition to efficient
data management, additional complexity arises from dealing with semi-structured
or unstructured data, and from time critical processing requirements. In order
to understand these massive amounts of data, advanced visualization and data
exploration techniques are required. Innovative approaches to these challenges
have been developed during recent years, and continue to be a hot topic for
re-search and industry in the future. An investigation of current approaches
reveals that usually only one or two aspects are ad-dressed, either in the data
management, processing, analysis or visualization. This paper presents the
vision of an integrated plat-form for big data analysis that combines all these
aspects. Main benefits of this approach are an enhanced scalability of the
whole platform, a better parameterization of algorithms, a more efficient usage
of system resources, and an improved usability during the end-to-end data
analysis process.
Related papers
- A Survey on Data Synthesis and Augmentation for Large Language Models [35.59526251210408]
This paper reviews and summarizes data generation techniques throughout the lifecycle of Large Language Models.
We discuss the current constraints faced by these methods and investigate potential pathways for future development and research.
arXiv Detail & Related papers (2024-10-16T16:12:39Z) - Data Proportion Detection for Optimized Data Management for Large Language Models [32.62631669919273]
We introduce a new topic, textitdata proportion detection, which enables the automatic estimation of pre-training data proportions.
We provide rigorous theoretical proofs, practical algorithms, and preliminary experimental results for data proportion detection.
arXiv Detail & Related papers (2024-09-26T04:30:32Z) - Data Issues in Industrial AI System: A Meta-Review and Research Strategy [10.540603300770885]
Artificial intelligence (AI) is assuming an increasingly pivotal role within industrial systems.
Despite the recent trend within various industries to adopt AI, the actual adoption of AI is not as developed as perceived.
How to address these data issues stands as a significant concern confronting both industry and academia.
arXiv Detail & Related papers (2024-06-22T08:36:59Z) - Benchmarking Data Science Agents [11.582116078653968]
Large Language Models (LLMs) have emerged as promising aids as data science agents, assisting humans in data analysis and processing.
Yet their practical efficacy remains constrained by the varied demands of real-world applications and complicated analytical process.
We introduce DSEval -- a novel evaluation paradigm, as well as a series of innovative benchmarks tailored for assessing the performance of these agents.
arXiv Detail & Related papers (2024-02-27T03:03:06Z) - Privacy-Preserving Graph Machine Learning from Data to Computation: A
Survey [67.7834898542701]
We focus on reviewing privacy-preserving techniques of graph machine learning.
We first review methods for generating privacy-preserving graph data.
Then we describe methods for transmitting privacy-preserved information.
arXiv Detail & Related papers (2023-07-10T04:30:23Z) - Demonstration of InsightPilot: An LLM-Empowered Automated Data
Exploration System [48.62158108517576]
We introduce InsightPilot, an automated data exploration system designed to simplify the data exploration process.
InsightPilot automatically selects appropriate analysis intents, such as understanding, summarizing, and explaining.
In brief, an IQuery is an abstraction and automation of data analysis operations, which mimics the approach of data analysts.
arXiv Detail & Related papers (2023-04-02T07:27:49Z) - A Comprehensive Survey of Dataset Distillation [73.15482472726555]
It has become challenging to handle the unlimited growth of data with limited computing power.
Deep learning technology has developed unprecedentedly in the last decade.
This paper provides a holistic understanding of dataset distillation from multiple aspects.
arXiv Detail & Related papers (2023-01-13T15:11:38Z) - Advanced Data Augmentation Approaches: A Comprehensive Survey and Future
directions [57.30984060215482]
We provide a background of data augmentation, a novel and comprehensive taxonomy of reviewed data augmentation techniques, and the strengths and weaknesses (wherever possible) of each technique.
We also provide comprehensive results of the data augmentation effect on three popular computer vision tasks, such as image classification, object detection and semantic segmentation.
arXiv Detail & Related papers (2023-01-07T11:37:32Z) - Research Trends and Applications of Data Augmentation Algorithms [77.34726150561087]
We identify the main areas of application of data augmentation algorithms, the types of algorithms used, significant research trends, their progression over time and research gaps in data augmentation literature.
We expect readers to understand the potential of data augmentation, as well as identify future research directions and open questions within data augmentation research.
arXiv Detail & Related papers (2022-07-18T11:38:32Z) - Occams Razor for Big Data? On Detecting Quality in Large Unstructured
Datasets [0.0]
New trend towards analytic complexity represents a severe challenge for the principle of parsimony or Occams Razor in science.
Computational building block approaches for data clustering can help to deal with large unstructured datasets in minimized computation time.
The review concludes on how cultural differences between East and West are likely to affect the course of big data analytics.
arXiv Detail & Related papers (2020-11-12T16:06:01Z) - Data Mining with Big Data in Intrusion Detection Systems: A Systematic
Literature Review [68.15472610671748]
Cloud computing has become a powerful and indispensable technology for complex, high performance and scalable computation.
The rapid rate and volume of data creation has begun to pose significant challenges for data management and security.
The design and deployment of intrusion detection systems (IDS) in the big data setting has, therefore, become a topic of importance.
arXiv Detail & Related papers (2020-05-23T20:57:12Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.