Investigating the Impact of Isolation on Synchronized Benchmarks

• URL: http://arxiv.org/abs/2511.03533v1
• Date: Wed, 05 Nov 2025 15:05:17 GMT
• Title: Investigating the Impact of Isolation on Synchronized Benchmarks
• Abstract summary: This work evaluates three such strategies: cgroups and CPU pinning, Docker containers, and Firecracker MicroVMs.<n>We compare all strategies with an unisolated baseline experiment, by running benchmarks with a duet setup alongside a noise generator.<n>Results show that process isolation generally lowered false positives, except for our experiments with Docker containers.
• Score: 0.0
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Benchmarking in cloud environments suffers from performance variability from multi-tenant resource contention. Duet benchmarking mitigates this by running two workload versions concurrently on the same VM, exposing them to identical external interference. However, intra-VM contention between synchronized workloads necessitates additional isolation mechanisms. This work evaluates three such strategies: cgroups and CPU pinning, Docker containers, and Firecracker MicroVMs. We compare all strategies with an unisolated baseline experiment, by running benchmarks with a duet setup alongside a noise generator. This noise generator "steals" compute resources to degrade performance measurements. All experiments showed different latency distributions while under the effects of noise generation, but results show that process isolation generally lowered false positives, except for our experiments with Docker containers. Even though Docker containers rely internally on cgroups and CPU pinning, they were more susceptible to performance degradation due to noise influence. Therefore, we recommend to use process isolation for synchronized workloads, with the exception of Docker containers.

Related papers

• Docker Containers vs. Virtual Machines: A Comparative Study of Architecture, Performance, Configuration, and Security [0.0]
  Modern application platforms must isolate workloads while preserving deployment speed, portability, resource efficiency, and security.<n>VMs and Docker containers address this requirement at different abstraction layers.<n>This paper presents a comparative, literature-based analysis of the two approaches across architecture, configuration and lifecycle management, performance, scalability, and security.
  arXiv  Detail & Related papers  (2026-09-14T18:00:30Z)
• Benchmarking Composable Compression Techniques in Mixture-of-Experts LLMs [50.677556944160216]
  MoEXBench is a systematic benchmark for evaluating composable MoE compression as an end-to-end deployment workflow.<n>It measures composable-compression quality, workload and architecture, pruning/quantization/KV cache sensitivity, and deployment efficiency on commodity hardware.
  arXiv  Detail & Related papers  (2026-08-22T00:01:18Z)
• SCOUT: Symmetric Consensus Outlier Detection for Failure Localization in LLM Pre-Training [7.018753691717986]
  In pre-training, synchronization propagates rank-local stalls, slowdowns, and numerical errors into job-wide symptoms, obscuring their origin.<n>We present SCOUT, a unified runtime failure-localization framework built on one design principle: identify outliers through strict-majority consensus among equivalent replicas.<n>SCOUT integrates with PyTorch, TorchTitan, Megatron-Core, and DeepSpeed without training-loop or framework-source modifications.
  arXiv  Detail & Related papers  (2026-08-11T15:12:14Z)
• Cross-Agent Campaign Attribution: Linking Asynchronous Attacks Across LLM Agents [8.011258884089948]
  We formalize cross-agent asynchronous campaign attribution, linking sessions from the same latent adversarial campaign without shared runtime state.<n>Asynchronous Fingerprint Vectors ($A2FV$) is a proxy-side reference protocol for scoring pairwise campaign similarity.<n>On SCD-v1, $A2FV$ achieves 0.82 pairwise AUC for campaign linking, while score-only adaptations of per-session detectors and chunked LLM judges remain near chance.
  arXiv  Detail & Related papers  (2026-07-21T08:01:51Z)
• Isolation Failure From Shared Storage: Characterizing and Exploiting Page-Cache SCA Leakage Across Containers and VMs [30.297643848902776]
  We investigate whether unprivileged timing measurements can reveal page-cache residency across isolation boundaries.<n>Results place page-cache attacks within the broader class of OS-mediated microarchitectural timing channels.
  arXiv  Detail & Related papers  (2026-07-20T03:47:34Z)
• BlazeFL: Fast and Deterministic Federated Learning Simulation [1.3218606066831913]
  Federated learning (FL) research increasingly relies on single-node simulations with hundreds or thousands of virtual clients.<n>Yet parallel client training often introduces nondeterminism through shared random state and scheduling variability.<n>We present BlazeFL, a lightweight framework for single-node FL simulation that alleviates this trade-off through free-threaded shared-memory execution.
  arXiv  Detail & Related papers  (2026-04-04T06:35:15Z)
• EVM-QuestBench: An Execution-Grounded Benchmark for Natural-Language Transaction Code Generation [9.472124187479915]
  In on-chain transaction scenarios, even a minor error can cause irreversible loss for users.<n>We introduce EVM-QuestBench, an execution-grounded benchmark for natural-language transaction-script generation.<n>We evaluate 20 models and find large performance gaps, with split scores revealing persistent asymmetry between single-action precision and multi-step workflow completion.
  arXiv  Detail & Related papers  (2026-01-10T13:25:27Z)
• RAC-DMVC: Reliability-Aware Contrastive Deep Multi-View Clustering under Multi-Source Noise [32.96137519578033]
  Multi-view clustering (MVC) aims to separate the multi-view data into distinct clusters in an unsupervised manner.<n>This paper addresses a more challenging task: MVC under multi-source noises, including missing noise and observation noise.<n>We propose a novel framework, Reliability-Aware Contrastive Deep Multi-View Clustering (RAC-DMVC), which constructs a reliability graph to guide robust representation learning under noisy environments.
  arXiv  Detail & Related papers  (2025-11-17T16:28:12Z)
• Are We Using the Right Benchmark: An Evaluation Framework for Visual Token Compression Methods [54.4711434793961]
  We show that simple image downsampling consistently outperforms many advanced compression methods across multiple widely used benchmarks.<n>Motivated by these findings, we introduce VTC-Bench, an evaluation framework that incorporates a data filtering mechanism to denoise existing benchmarks.
  arXiv  Detail & Related papers  (2025-10-08T15:44:28Z)
• Echo: Decoupling Inference and Training for Large-Scale RL Alignment on Heterogeneous Swarms [4.127488674019288]
  Post-training for large language models co-locates trajectory sampling and policy optimisation on the same GPU cluster.<n>We present Echo, the RL system that cleanly decouples these two phases across heterogeneous "inference" and "training" swarms.
  arXiv  Detail & Related papers  (2025-08-07T13:37:04Z)
• Automatically Identify and Rectify: Robust Deep Contrastive Multi-view Clustering in Noisy Scenarios [76.02688769599686]
  We propose a novel multi-view clustering framework for the automatic identification and rectification of noisy data, termed AIRMVC.<n>Specifically, we reformulate noisy identification as an anomaly identification problem using GMM.<n>We then design a hybrid rectification strategy to mitigate the adverse effects of noisy data based on the identification results.
  arXiv  Detail & Related papers  (2025-05-27T16:16:54Z)
• Bencher: Simple and Reproducible Benchmarking for Black-Box Optimization [5.703483582960509]
  Bencher is a modular benchmarking framework for black-box optimization.<n>Each benchmark is isolated in its own virtual Python environment and accessed via a unified, version-agnostic remote procedure call (RPC) interface.<n>Bencher can be deployed locally or remotely via Docker or on high-performance computing clusters via Singularity.
  arXiv  Detail & Related papers  (2025-05-27T15:18:58Z)
• AsyncDiff: Parallelizing Diffusion Models by Asynchronous Denoising [49.785626309848276]
  AsyncDiff is a universal and plug-and-play acceleration scheme that enables model parallelism across multiple devices. For the Stable Diffusion v2.1, AsyncDiff achieves a 2.7x speedup with negligible degradation and a 4.0x speedup with only a slight reduction of 0.38 in CLIP Score. Our experiments also demonstrate that AsyncDiff can be readily applied to video diffusion models with encouraging performances.
  arXiv  Detail & Related papers  (2024-06-11T03:09:37Z)
• BurstAttention: An Efficient Distributed Attention Framework for Extremely Long Sequences [96.74779792715819]
  We propose a distributed attention framework named BurstAttention'' to optimize memory access and communication operations. The experimental results under different length settings demonstrate that BurstAttention offers significant advantages for processing long sequences.
  arXiv  Detail & Related papers  (2024-03-14T12:51:58Z)
• Towards Robust Few-shot Point Cloud Semantic Segmentation [57.075074484313]
  Few-shot point cloud semantic segmentation aims to train a model to quickly adapt to new unseen classes with only a handful of support set samples. We propose a Component-level Clean Noise Separation (CCNS) representation learning to learn discriminative feature representations. We also propose a Multi-scale Degree-based Noise Suppression (MDNS) scheme to remove the noisy shots from the support set.
  arXiv  Detail & Related papers  (2023-09-20T11:40:10Z)
• Class Prototype-based Cleaner for Label Noise Learning [73.007001454085]
  Semi-supervised learning methods are current SOTA solutions to the noisy-label learning problem. We propose a simple yet effective solution, named textbfClass textbfPrototype-based label noise textbfCleaner.
  arXiv  Detail & Related papers  (2022-12-21T04:56:41Z)
• Noise2Same: Optimizing A Self-Supervised Bound for Image Denoising [54.730707387866076]
  We introduce Noise2Same, a novel self-supervised denoising framework. In particular, Noise2Same requires neither J-invariance nor extra information about the noise model. Our results show that our Noise2Same remarkably outperforms previous self-supervised denoising methods.
  arXiv  Detail & Related papers  (2020-10-22T18:12:26Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.