DatAasee -- A Metadata-Lake as Metadata Catalog for a Virtual Data-Lake

• URL: http://arxiv.org/abs/2409.05512v2
• Date: Tue, 06 May 2025 13:51:59 GMT
• Title: DatAasee -- A Metadata-Lake as Metadata Catalog for a Virtual Data-Lake
• Abstract summary: This work constructs a data architecture, derived from the data-lake: the metadata-lake.<n>A proof-of-concept implementation of this proposed metadata aggregator is presented and also evaluated.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Metadata management for distributed data sources is a long-standing but ever-growing problem. To counter this challenge in a research-data and library-oriented setting, this work constructs a data architecture, derived from the data-lake: the metadata-lake. A proof-of-concept implementation of this proposed metadata aggregator is presented and also evaluated.

Related papers

• LAND: A Longitudinal Analysis of Neuromorphic Datasets [47.84658158364453]
  This review captures the existing neuromorphic datasets, covering over 423 datasets.<n>Analysing these datasets shows the difficulties arising from their size, the lack of standardisation, and difficulties in accessing the actual data.<n>A more important concern is the rise of synthetic datasets, created by either simulation or video-to-events methods.
  arXiv  Detail & Related papers  (2026-02-17T19:46:33Z)
• Can LLMs Clean Up Your Mess? A Survey of Application-Ready Data Preparation with LLMs [66.63911043019294]
  Data preparation aims to denoise raw datasets, uncover cross-dataset relationships, and extract valuable insights from them.<n>This paper focuses on the use of LLM techniques to prepare data for diverse downstream tasks.<n>We introduce a task-centric taxonomy that organizes the field into three major tasks: data cleaning, standardization, error processing, imputation, data integration, and data enrichment.
  arXiv  Detail & Related papers  (2026-01-22T12:02:45Z)
• Smart Metadata in Action: The Social Impact Data Commons [0.5545791216381869]
  This article describes the use of metadata and standards in the Social Impact Data Commons to expose official statisticians to an innovative project built on actionable and evaluable metadata.<n>We begin by introducing the concept of the Data Commons, focusing on its features, and presenting an overview of current implementations of the Data Commons.<n>We then present the core metadata case study, demonstrating how smart metadata support the Data Commons.
  arXiv  Detail & Related papers  (2025-11-21T16:18:47Z)
• LLM/Agent-as-Data-Analyst: A Survey [54.08761322298559]
  Large language models (LLMs) and agent techniques have brought a fundamental shift in the functionality and development paradigm of data analysis tasks.<n>LLMs enable complex data understanding, natural language, semantic analysis functions, and autonomous pipeline orchestration.
  arXiv  Detail & Related papers  (2025-09-28T17:31:38Z)
• Flexible metadata harvesting for ecology using large language models [3.4117490081172774]
  We develop a large language model (LLM)-based metadata harvester.<n>It flexibly extracts metadata from any dataset's landing page.<n>It converts these to a user-defined, unified format using existing metadata standards.
  arXiv  Detail & Related papers  (2025-08-21T10:10:29Z)
• Metadata Enrichment of Long Text Documents using Large Language Models [3.536523762475449]
  In this project, we semantically enriched and enhanced the metadata of long text documents, theses and dissertations, retrieved from the HathiTrust Digital Library in English published from 1920 to 2020.<n>This dataset provides a valuable resource for advancing research in areas such as computational social science, digital humanities, and information science.
  arXiv  Detail & Related papers  (2025-06-26T00:55:47Z)
• MOLE: Metadata Extraction and Validation in Scientific Papers Using LLMs [54.5729817345543]
  MOLE is a framework that automatically extracts metadata attributes from scientific papers covering datasets of languages other than Arabic.<n>Our methodology processes entire documents across multiple input formats and incorporates robust validation mechanisms for consistent output.
  arXiv  Detail & Related papers  (2025-05-26T10:31:26Z)
• Data Advisor: Dynamic Data Curation for Safety Alignment of Large Language Models [79.65071553905021]
  We propose Data Advisor, a method for generating data that takes into account the characteristics of the desired dataset. Data Advisor monitors the status of the generated data, identifies weaknesses in the current dataset, and advises the next iteration of data generation.
  arXiv  Detail & Related papers  (2024-10-07T17:59:58Z)
• Putting Data at the Centre of Offline Multi-Agent Reinforcement Learning [3.623224034411137]
  offline multi-agent reinforcement learning (MARL) is an exciting direction of research that uses static datasets to find optimal control policies for multi-agent systems. Though the field is by definition data-driven, efforts have thus far neglected data in their drive to achieve state-of-the-art results. We show how the majority of works generate their own datasets without consistent methodology and provide sparse information about the characteristics of these datasets.
  arXiv  Detail & Related papers  (2024-09-18T14:13:24Z)
• An Integrated Data Processing Framework for Pretraining Foundation Models [57.47845148721817]
  Researchers and practitioners often have to manually curate datasets from difference sources. We propose a data processing framework that integrates a Processing Module and an Analyzing Module. The proposed framework is easy to use and highly flexible.
  arXiv  Detail & Related papers  (2024-02-26T07:22:51Z)
• Semantic Data Management in Data Lakes [0.0]
  In recent years, data lakes emerged as away to manage large amounts of heterogeneous data for modern data analytics. One way to prevent data lakes from turning into inoperable data swamps is semantic data management. We classify the approaches into (i) basic semantic data management, (ii) semantic modeling approaches for enriching metadata in data lakes, and (iii) methods for ontologybased data access.
  arXiv  Detail & Related papers  (2023-10-23T21:16:50Z)
• Dataset Factory: A Toolchain For Generative Computer Vision Datasets [0.9013233848500058]
  We propose a "dataset factory" that separates the storage and processing of samples from metadata. This enables data-centric operations at scale for machine learning teams and individual researchers.
  arXiv  Detail & Related papers  (2023-09-20T19:43:37Z)
• Modeling Data Analytics Architecture for Smart Cities Data-Driven Applications using DAT [1.8945921149936187]
  This article shares our experiences in developing a Data Analytics Architecture (DAA) using model-driven engineering for Data-Driven Smart Cities applications utilizing DAT. DAA uses model-driven engineering for Data-Driven Smart Cities applications utilizing DAT.
  arXiv  Detail & Related papers  (2023-07-17T21:52:57Z)
• infoVerse: A Universal Framework for Dataset Characterization with Multidimensional Meta-information [68.76707843019886]
  infoVerse is a universal framework for dataset characterization. infoVerse captures multidimensional characteristics of datasets by incorporating various model-driven meta-information. In three real-world applications (data pruning, active learning, and data annotation), the samples chosen on infoVerse space consistently outperform strong baselines.
  arXiv  Detail & Related papers  (2023-05-30T18:12:48Z)
• TRoVE: Transforming Road Scene Datasets into Photorealistic Virtual Environments [84.6017003787244]
  This work proposes a synthetic data generation pipeline to address the difficulties and domain-gaps present in simulated datasets. We show that using annotations and visual cues from existing datasets, we can facilitate automated multi-modal data generation.
  arXiv  Detail & Related papers  (2022-08-16T20:46:08Z)
• Metadata Representations for Queryable ML Model Zoos [73.24799582702326]
  Machine learning (ML) practitioners and organizations are building model zoos of pre-trained models, containing metadata describing properties of the models. The metatada is currently not standardised; its expressivity is limited; and there is no way to store and query it. In this paper, we advocate for standardized ML model meta-data representation and management, proposing a toolkit supported to help practitioners manage and query that metadata.
  arXiv  Detail & Related papers  (2022-07-19T15:04:14Z)
• Datasets: A Community Library for Natural Language Processing [55.48866401721244]
  datasets is a community library for contemporary NLP. The library includes more than 650 unique datasets, has more than 250 contributors, and has helped support a variety of novel cross-dataset research projects.
  arXiv  Detail & Related papers  (2021-09-07T03:59:22Z)
• A Big Data Lake for Multilevel Streaming Analytics [0.4640835690336652]
  This paper focuses on storing high volume, velocity and variety data in the raw formats in a data storage architecture called a data lake. We discuss and compare different open source and commercial platforms that can be used to develop a data lake. We present a real-world data lake development use case for data stream ingestion, staging, and multilevel streaming analytics.
  arXiv  Detail & Related papers  (2020-09-25T19:57:21Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.