Web Scraping for Research: Legal, Ethical, Institutional, and Scientific Considerations

• URL: http://arxiv.org/abs/2410.23432v1
• Date: Wed, 30 Oct 2024 20:20:44 GMT
• Title: Web Scraping for Research: Legal, Ethical, Institutional, and Scientific Considerations
• Authors: Megan A. Brown, Andrew Gruen, Gabe Maldoff, Solomon Messing, Zeve Sanderson, Michael Zimmer,
• Abstract summary: This paper proposes a comprehensive framework for web scraping in social science research for U.S.-based researchers. We present an overview of the current regulatory environment impacting when and how researchers can access, collect, store, and share data via scraping. We then provide researchers with recommendations to conduct scraping in a scientifically legitimate and ethical manner.
• Score: 11.851771490297693
• License:
• Abstract: Scientists across disciplines often use data from the internet to conduct research, generating valuable insights about human behavior. However, as generative AI relying on massive text corpora becomes increasingly valuable, platforms have greatly restricted access to data through official channels. As a result, researchers will likely engage in more web scraping to collect data, introducing new challenges and concerns for researchers. This paper proposes a comprehensive framework for web scraping in social science research for U.S.-based researchers, examining the legal, ethical, institutional, and scientific factors that researchers should consider when scraping the web. We present an overview of the current regulatory environment impacting when and how researchers can access, collect, store, and share data via scraping. We then provide researchers with recommendations to conduct scraping in a scientifically legitimate and ethical manner. We aim to equip researchers with the relevant information to mitigate risks and maximize the impact of their research amidst this evolving data access landscape.

Related papers

• A Survey of Privacy-Preserving Model Explanations: Privacy Risks, Attacks, and Countermeasures [50.987594546912725]
  Despite a growing corpus of research in AI privacy and explainability, there is little attention on privacy-preserving model explanations. This article presents the first thorough survey about privacy attacks on model explanations and their countermeasures.
  arXiv  Detail & Related papers  (2024-03-31T12:44:48Z)
• Data Science for Social Good [2.8621556092850065]
  We present a framework for "data science for social good" (DSSG) research. We perform an analysis of the literature to empirically demonstrate the paucity of work on DSSG in information systems. We hope that this article and the special issue will spur future DSSG research.
  arXiv  Detail & Related papers  (2023-11-02T15:40:20Z)
• A Responsive Framework for Research Portals Data using Semantic Web Technology [0.6798775532273751]
  The research aims to address this issue by designing a framework for the semantic organization of research portal data. The framework focuses on the extraction of information from two specific research portals, namely Microsoft Academic and IEEE Xplore.
  arXiv  Detail & Related papers  (2023-06-20T16:12:33Z)
• The ethical ambiguity of AI data enrichment: Measuring gaps in research ethics norms and practices [2.28438857884398]
  This study explores how, and to what extent, comparable research ethics requirements and norms have developed for AI research and data enrichment. Leading AI venues have begun to establish protocols for human data collection, but these are are inconsistently followed by authors.
  arXiv  Detail & Related papers  (2023-06-01T16:12:55Z)
• Assessing Scientific Contributions in Data Sharing Spaces [64.16762375635842]
  This paper introduces the SCIENCE-index, a blockchain-based metric measuring a researcher's scientific contributions. To incentivize researchers to share their data, the SCIENCE-index is augmented to include a data-sharing parameter. Our model is evaluated by comparing the distribution of its output for geographically diverse researchers to that of the h-index.
  arXiv  Detail & Related papers  (2023-03-18T19:17:47Z)
• Human-Centered Responsible Artificial Intelligence: Current & Future Trends [76.94037394832931]
  In recent years, the CHI community has seen significant growth in research on Human-Centered Responsible Artificial Intelligence. All of this work is aimed at developing AI that benefits humanity while being grounded in human rights and ethics, and reducing the potential harms of AI. In this special interest group, we aim to bring together researchers from academia and industry interested in these topics to map current and future research trends.
  arXiv  Detail & Related papers  (2023-02-16T08:59:42Z)
• How Data Scientists Review the Scholarly Literature [4.406926847270567]
  We examine the literature review practices of data scientists. Data science represents a field seeing an exponential rise in papers. No prior work has examined the specific practices and challenges faced by these scientists.
  arXiv  Detail & Related papers  (2023-01-10T03:53:05Z)
• DeepShovel: An Online Collaborative Platform for Data Extraction in Geoscience Literature with AI Assistance [48.55345030503826]
  Geoscientists need to read a huge amount of literature to locate, extract, and aggregate relevant results and data. DeepShovel is a publicly-available AI-assisted data extraction system to support their needs. A follow-up user evaluation with 14 researchers suggested DeepShovel improved users' efficiency of data extraction for building scientific databases.
  arXiv  Detail & Related papers  (2022-02-21T12:18:08Z)
• Yes-Yes-Yes: Donation-based Peer Reviewing Data Collection for ACL Rolling Review and Beyond [58.71736531356398]
  We present an in-depth discussion of peer reviewing data, outline the ethical and legal desiderata for peer reviewing data collection, and propose the first continuous, donation-based data collection workflow. We report on the ongoing implementation of this workflow at the ACL Rolling Review and deliver the first insights obtained with the newly collected data.
  arXiv  Detail & Related papers  (2022-01-27T11:02:43Z)
• Learnings from Frontier Development Lab and SpaceML -- AI Accelerators for NASA and ESA [57.06643156253045]
  Research with AI and ML technologies lives in a variety of settings with often asynchronous goals and timelines. We perform a case study of the Frontier Development Lab (FDL), an AI accelerator under a public-private partnership from NASA and ESA. FDL research follows principled practices that are grounded in responsible development, conduct, and dissemination of AI research.
  arXiv  Detail & Related papers  (2020-11-09T21:23:03Z)
• Ethical issues with using Internet of Things devices in citizen science research: A scoping review [1.933681537640272]
  This chapter presents a scoping review of published scientific studies that utilise both citizen scientists and Internet of Things devices. We selected studies where the authors had included at least a short discussion of the ethical issues encountered during the research process. Following this analysis, our discussion provides recommendations for researchers who wish to integrate citizen scientists and Internet of Things devices into their research.
  arXiv  Detail & Related papers  (2020-07-18T12:22:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.