You Get What You Sample: Evaluating Sampling Strategies for Web Security Measurements

• URL: http://arxiv.org/abs/2609.11218v1
• Date: Thu, 10 Sep 2026 08:21:18 GMT
• Title: You Get What You Sample: Evaluating Sampling Strategies for Web Security Measurements
• Abstract summary: This work is the first comprehensive investigation into how sampling methodologies affect the measurements and the conclusions.<n>We show that while Top $N$ sampling may be a rational strategy, the researchers have to bear in mind that Top $N$ does not reflect the overall distribution of the web.<n>We propose to use an adaptive probability-based sampling strategy that remains effective even when the prevalence of the target issue is unknown.
• Score: 15.393354110563237
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Web measurement studies rely on domain datasets such as Tranco to quantify the prevalence and impact of security issues at scale, but exhaustively analyzing these datasets is often infeasible because of the cost of advanced analysis techniques, requiring the use of sampling. Despite its widespread use, sampling remains largely guided by convention---most commonly \emph{Top $N$} domain selection---rather than evidence, and its influence on the validity and generalizability of security findings has received little systematic evaluation. Consequently, it remains unclear whether common sampling strategies introduce systematic bias, distort observed vulnerability rates, or limit comparability across studies. In this work, we undertake, to the best of our knowledge, the first comprehensive investigation into how sampling methodologies affect the measurements and the conclusions. Through a comprehensive literature review and large-scale measurements of 500k Tranco and 24.8M Common Crawl hosts, we perform a comparative evaluation of datasets and sampling strategies. We show that, while Top $N$ sampling may be a rational strategy, the researchers have to bear in mind that Top $N$ does not reflect the overall distribution of the web. Instead, probability-based strategies yield stable, unbiased estimates for prevalence and many impact objectives. Hybrid sampling provides no advantages over pure probability sampling, as its deterministic prefix consistently contributes negatively to accuracy. Building on these results, we provide data-backed guidance for future studies, proposing to use an adaptive probability-based sampling strategy that remains effective even when the prevalence of the target issue is unknown.

Related papers

• Robust Sampling for Active Statistical Inference [11.929391566298841]
  Active statistical inference is a new method for inference with AI-assisted data collection.<n>We present robust sampling strategies for active statistical inference.<n>We demonstrate the utility of the method on a series of real datasets.
  arXiv  Detail & Related papers  (2025-11-12T05:18:36Z)
• On the Reliability of Sampling Strategies in Offline Recommender Evaluation [3.4956406636452626]
  offline evaluation plays a central role in benchmarking recommender systems when online testing is impractical or risky.<n>It is susceptible to two key sources of bias: exposure bias, where users only interact with items they are shown, and sampling bias, introduced when evaluation is performed on a subset of logged items rather than the full catalog.
  arXiv  Detail & Related papers  (2025-08-07T13:50:05Z)
• Optimal Sampling for Generalized Linear Model under Measurement Constraint with Surrogate Variables [3.5903555216741405]
  In some cases, surrogate variables are accessible across the entire dataset and can serve as approximations to the true response variable.<n>We propose an optimal sampling strategy that effectively harnesses the available information from surrogate variables.
  arXiv  Detail & Related papers  (2025-01-01T22:41:52Z)
• Stratified Prediction-Powered Inference for Hybrid Language Model Evaluation [62.2436697657307]
  Prediction-powered inference (PPI) is a method that improves statistical estimates based on limited human-labeled data.<n>We propose a method called Stratified Prediction-Powered Inference (StratPPI)<n>We show that the basic PPI estimates can be considerably improved by employing simple data stratification strategies.
  arXiv  Detail & Related papers  (2024-06-06T17:37:39Z)
• Uncertainty for Active Learning on Graphs [70.44714133412592]
  Uncertainty Sampling is an Active Learning strategy that aims to improve the data efficiency of machine learning models.<n>We benchmark Uncertainty Sampling beyond predictive uncertainty and highlight a significant performance gap to other Active Learning strategies.<n>We develop ground-truth Bayesian uncertainty estimates in terms of the data generating process and prove their effectiveness in guiding Uncertainty Sampling toward optimal queries.
  arXiv  Detail & Related papers  (2024-05-02T16:50:47Z)
• On the Universal Adversarial Perturbations for Efficient Data-free Adversarial Detection [55.73320979733527]
  We propose a data-agnostic adversarial detection framework, which induces different responses between normal and adversarial samples to UAPs. Experimental results show that our method achieves competitive detection performance on various text classification tasks.
  arXiv  Detail & Related papers  (2023-06-27T02:54:07Z)
• Uncertainty-Aware Instance Reweighting for Off-Policy Learning [63.31923483172859]
  We propose a Uncertainty-aware Inverse Propensity Score estimator (UIPS) for improved off-policy learning. Experiment results on synthetic and three real-world recommendation datasets demonstrate the advantageous sample efficiency of the proposed UIPS estimator.
  arXiv  Detail & Related papers  (2023-03-11T11:42:26Z)
• ZigZag: Universal Sampling-free Uncertainty Estimation Through Two-Step Inference [54.17205151960878]
  We introduce a sampling-free approach that is generic and easy to deploy. We produce reliable uncertainty estimates on par with state-of-the-art methods at a significantly lower computational cost.
  arXiv  Detail & Related papers  (2022-11-21T13:23:09Z)
• MEET: A Monte Carlo Exploration-Exploitation Trade-off for Buffer Sampling [2.501153467354696]
  State-of-the-art sampling strategies for the experience replay buffer improve the performance of the Reinforcement Learning agent. They do not incorporate uncertainty in the Q-Value estimation. This paper proposes a new sampling strategy that leverages the exploration-exploitation trade-off.
  arXiv  Detail & Related papers  (2022-10-24T18:55:41Z)
• Rethinking Collaborative Metric Learning: Toward an Efficient Alternative without Negative Sampling [156.7248383178991]
  Collaborative Metric Learning (CML) paradigm has aroused wide interest in the area of recommendation systems (RS) We find that negative sampling would lead to a biased estimation of the generalization error. Motivated by this, we propose an efficient alternative without negative sampling for CML named textitSampling-Free Collaborative Metric Learning (SFCML)
  arXiv  Detail & Related papers  (2022-06-23T08:50:22Z)
• Holistic Approach to Measure Sample-level Adversarial Vulnerability and its Utility in Building Trustworthy Systems [17.707594255626216]
  Adversarial attack perturbs an image with an imperceptible noise, leading to incorrect model prediction. We propose a holistic approach for quantifying adversarial vulnerability of a sample by combining different perspectives. We demonstrate that by reliably estimating adversarial vulnerability at the sample level, it is possible to develop a trustworthy system.
  arXiv  Detail & Related papers  (2022-05-05T12:36:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.