A Datalake for Data-driven Social Science Research

• URL: http://arxiv.org/abs/2512.02463v1
• Date: Tue, 02 Dec 2025 06:40:47 GMT
• Title: A Datalake for Data-driven Social Science Research
• Authors: Puneet Arya, Ojas Sahasrabudhe, Adwaiya Srivastav, Partha Pratim Das, Maya Ramanath,
• Abstract summary: We present a Datalake infrastructure tailored to the needs of interdisciplinary social science research.<n>Our system supports ingestion and integration of diverse data types, automatic provenance and version tracking, role-based access control, and built-in tools for visualization and analysis.<n>We argue that such infrastructure can democratize access to advanced data science practices, especially for NGOs, students, and grassroots organizations.
• Score: 2.285735909183272
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Social science research increasingly demands data-driven insights, yet researchers often face barriers such as lack of technical expertise, inconsistent data formats, and limited access to reliable datasets.Social science research increasingly demands data-driven insights, yet researchers often face barriers such as lack of technical expertise, inconsistent data formats, and limited access to reliable datasets. In this paper, we present a Datalake infrastructure tailored to the needs of interdisciplinary social science research. Our system supports ingestion and integration of diverse data types, automatic provenance and version tracking, role-based access control, and built-in tools for visualization and analysis. We demonstrate the utility of our Datalake using real-world use cases spanning governance, health, and education. A detailed walkthrough of one such use case -- analyzing the relationship between income, education, and infant mortality -- shows how our platform streamlines the research process while maintaining transparency and reproducibility. We argue that such infrastructure can democratize access to advanced data science practices, especially for NGOs, students, and grassroots organizations. The Datalake continues to evolve with plans to support ML pipelines, mobile access, and citizen data feedback mechanisms.

Related papers

• Understanding Usage and Engagement in AI-Powered Scientific Research Tools: The Asta Interaction Dataset [47.98539809308384]
  We analyze the Asta Interaction dataset, a large-scale resource comprising over 200,000 user queries and interaction logs.<n>We characterize query patterns, engagement behaviors, and how usage evolves with experience.<n>We release the anonymized dataset and analysis with a new query taxonomy to inform future designs of real-world AI research assistants.
  arXiv  Detail & Related papers  (2026-02-26T18:40:28Z)
• A GenAI System for Improved FAIR Independent Biological Database Integration [0.0]
  We introduce an experimental natural language-based query processing system designed to empower scientists to discover, access, and query biological databases.<n> FAIRBridge harnesses the capabilities of AI to interpret query intents, map them to relevant databases, and generate executable queries.<n>The system also includes robust tools for mitigating low-quality query processing, ensuring high fidelity and responsiveness in the information delivered.
  arXiv  Detail & Related papers  (2025-06-22T08:04:24Z)
• Data-Driven Breakthroughs and Future Directions in AI Infrastructure: A Comprehensive Review [0.0]
  This paper presents a comprehensive synthesis of major breakthroughs in artificial intelligence (AI) over the past fifteen years.<n>It identifies key inflection points in AI' s evolution by tracing the convergence of computational resources, data access, and algorithmic innovation.
  arXiv  Detail & Related papers  (2025-05-22T15:12:48Z)
• Network Intrusion Datasets: A Survey, Limitations, and Recommendations [0.0]
  Data-driven cyberthreat detection has become a crucial defense technique in modern cybersecurity.<n>Despite the importance of data, its scarcity has long been recognized as a major obstacle in NIDS research.
  arXiv  Detail & Related papers  (2025-02-10T17:14:37Z)
• DISCOVER: A Data-driven Interactive System for Comprehensive Observation, Visualization, and ExploRation of Human Behaviour [6.716560115378451]
  We introduce a modular, flexible, yet user-friendly software framework specifically developed to streamline computational-driven data exploration for human behavior analysis. Our primary objective is to democratize access to advanced computational methodologies, thereby enabling researchers across disciplines to engage in detailed behavioral analysis without the need for extensive technical proficiency.
  arXiv  Detail & Related papers  (2024-07-18T11:28:52Z)
• Social Intelligence Data Infrastructure: Structuring the Present and Navigating the Future [59.78608958395464]
  We build a Social AI Data Infrastructure, which consists of a comprehensive social AI taxonomy and a data library of 480 NLP datasets. Our infrastructure allows us to analyze existing dataset efforts, and also evaluate language models' performance in different social intelligence aspects. We show there is a need for multifaceted datasets, increased diversity in language and culture, more long-tailed social situations, and more interactive data in future social intelligence data efforts.
  arXiv  Detail & Related papers  (2024-02-28T00:22:42Z)
• On Responsible Machine Learning Datasets with Fairness, Privacy, and Regulatory Norms [56.119374302685934]
  There have been severe concerns over the trustworthiness of AI technologies. Machine and deep learning algorithms depend heavily on the data used during their development. We propose a framework to evaluate the datasets through a responsible rubric.
  arXiv  Detail & Related papers  (2023-10-24T14:01:53Z)
• Privacy-Preserving Graph Machine Learning from Data to Computation: A Survey [67.7834898542701]
  We focus on reviewing privacy-preserving techniques of graph machine learning. We first review methods for generating privacy-preserving graph data. Then we describe methods for transmitting privacy-preserved information.
  arXiv  Detail & Related papers  (2023-07-10T04:30:23Z)
• Research Trends and Applications of Data Augmentation Algorithms [77.34726150561087]
  We identify the main areas of application of data augmentation algorithms, the types of algorithms used, significant research trends, their progression over time and research gaps in data augmentation literature. We expect readers to understand the potential of data augmentation, as well as identify future research directions and open questions within data augmentation research.
  arXiv  Detail & Related papers  (2022-07-18T11:38:32Z)
• INODE: Building an End-to-End Data Exploration System in Practice [Extended Vision] [30.411996388471817]
  INODE is an end-to-end data exploration system. We demonstrate it in three significant use cases in the fields of Cancer Biomarker Reearch, Research and Innovation Policy Making, and Astrophysics.
  arXiv  Detail & Related papers  (2021-04-09T05:04:04Z)
• Explainable Patterns: Going from Findings to Insights to Support Data Analytics Democratization [60.18814584837969]
  We present Explainable Patterns (ExPatt), a new framework to support lay users in exploring and creating data storytellings. ExPatt automatically generates plausible explanations for observed or selected findings using an external (textual) source of information.
  arXiv  Detail & Related papers  (2021-01-19T16:13:44Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.