OpenDataLab: Empowering General Artificial Intelligence with Open Datasets

• URL: http://arxiv.org/abs/2407.13773v1
• Date: Tue, 4 Jun 2024 10:42:01 GMT
• Title: OpenDataLab: Empowering General Artificial Intelligence with Open Datasets
• Authors: Conghui He, Wei Li, Zhenjiang Jin, Chao Xu, Bin Wang, Dahua Lin,
• Abstract summary: This paper introduces OpenDataLab, a platform designed to bridge the gap between diverse data sources and the need for unified data processing. OpenDataLab integrates a wide range of open-source AI datasets and enhances data acquisition efficiency through intelligent querying and high-speed downloading services. We anticipate that OpenDataLab will significantly boost artificial general intelligence (AGI) research and facilitate advancements in related AI fields.
• Score: 53.22840149601411
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The advancement of artificial intelligence (AI) hinges on the quality and accessibility of data, yet the current fragmentation and variability of data sources hinder efficient data utilization. The dispersion of data sources and diversity of data formats often lead to inefficiencies in data retrieval and processing, significantly impeding the progress of AI research and applications. To address these challenges, this paper introduces OpenDataLab, a platform designed to bridge the gap between diverse data sources and the need for unified data processing. OpenDataLab integrates a wide range of open-source AI datasets and enhances data acquisition efficiency through intelligent querying and high-speed downloading services. The platform employs a next-generation AI Data Set Description Language (DSDL), which standardizes the representation of multimodal and multi-format data, improving interoperability and reusability. Additionally, OpenDataLab optimizes data processing through tools that complement DSDL. By integrating data with unified data descriptions and smart data toolchains, OpenDataLab can improve data preparation efficiency by 30\%. We anticipate that OpenDataLab will significantly boost artificial general intelligence (AGI) research and facilitate advancements in related AI fields. For more detailed information, please visit the platform's official website: https://opendatalab.com.

Related papers

• LAMBDA: A Large Model Based Data Agent [7.240586338370509]
  We introduce LArge Model Based Data Agent (LAMBDA), a novel open-source, code-free multi-agent data analysis system. LAMBDA is designed to address data analysis challenges in complex data-driven applications. It has the potential to enhance data analysis paradigms by seamlessly integrating human and artificial intelligence.
  arXiv  Detail & Related papers  (2024-07-24T06:26:36Z)
• DSDL: Data Set Description Language for Bridging Modalities and Tasks in AI Data [50.88106211204689]
  In the era of artificial intelligence, the diversity of data modalities and annotation formats often renders data unusable directly. This article introduces a framework that aims to simplify dataset processing by providing a unified standard for AI datasets. The standardized specifications of DSDL reduce the workload for users in data dissemination, processing, and usage.
  arXiv  Detail & Related papers  (2024-05-28T16:07:45Z)
• LESS: Selecting Influential Data for Targeted Instruction Tuning [64.78894228923619]
  We propose LESS, an efficient algorithm to estimate data influences and perform Low-rank gradiEnt Similarity Search for instruction data selection. We show that training on a LESS-selected 5% of the data can often outperform training on the full dataset across diverse downstream tasks. Our method goes beyond surface form cues to identify data that the necessary reasoning skills for the intended downstream application.
  arXiv  Detail & Related papers  (2024-02-06T19:18:04Z)
• Data Race Detection Using Large Language Models [1.0013600887991827]
  Large language models (LLMs) are an alternative strategy to facilitate analyses and optimizations of high-performance computing programs. In this paper, we explore a novel LLM-based data race detection approach combining prompting engineering and fine-tuning techniques.
  arXiv  Detail & Related papers  (2023-08-15T00:08:43Z)
• METAM: Goal-Oriented Data Discovery [9.73435089036831]
  METAM is a goal-oriented framework that queries the downstream task with a candidate dataset, forming a feedback loop that automatically steers the discovery and augmentation process. We show METAM's theoretical guarantees and demonstrate those empirically on a broad set of tasks.
  arXiv  Detail & Related papers  (2023-04-18T15:42:25Z)
• Outsourcing Training without Uploading Data via Efficient Collaborative Open-Source Sampling [49.87637449243698]
  Traditional outsourcing requires uploading device data to the cloud server. We propose to leverage widely available open-source data, which is a massive dataset collected from public and heterogeneous sources. We develop a novel strategy called Efficient Collaborative Open-source Sampling (ECOS) to construct a proximal proxy dataset from open-source data for cloud training.
  arXiv  Detail & Related papers  (2022-10-23T00:12:18Z)
• DataPerf: Benchmarks for Data-Centric AI Development [81.03754002516862]
  DataPerf is a community-led benchmark suite for evaluating ML datasets and data-centric algorithms. We provide an open, online platform with multiple rounds of challenges to support this iterative development. The benchmarks, online evaluation platform, and baseline implementations are open source.
  arXiv  Detail & Related papers  (2022-07-20T17:47:54Z)
• DataLab: A Platform for Data Analysis and Intervention [96.75253335629534]
  DataLab is a unified data-oriented platform that allows users to interactively analyze the characteristics of data. toolname has features for dataset recommendation and global vision analysis. So far, DataLab covers 1,715 datasets and 3,583 of its transformed version.
  arXiv  Detail & Related papers  (2022-02-25T18:32:19Z)
• LADA: Look-Ahead Data Acquisition via Augmentation for Active Learning [24.464022706979886]
  This paper proposes Look-Ahead Data Acquisition via augmentation, or LADA, to integrate data acquisition and data augmentation. LADA considers both 1) unlabeled data instance to be selected and 2) virtual data instance to be generated by data augmentation. The performance of LADA shows a significant improvement over the recent augmentation and acquisition baselines.
  arXiv  Detail & Related papers  (2020-11-09T05:21:14Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.