Operating Multi-Node Full Fine-Tuning on NVIDIA B300: A Field Report on Telemetry-Based Triage, Negative Results, and Operational Hardening

• URL: http://arxiv.org/abs/2608.05944v1
• Date: Thu, 06 Aug 2026 12:11:25 GMT
• Title: Operating Multi-Node Full Fine-Tuning on NVIDIA B300: A Field Report on Telemetry-Based Triage, Negative Results, and Operational Hardening
• Abstract summary: We report operational experience full-fine-tuning a 32.76B- parameter dense model (Qwen3-32B) on 16 x NVIDIA B300 (two nodes, FSDP / ZeRO-3)<n>Our contribution is the integrated field experience and a set of calibrated measurements on new hardware.<n>We position our instantiation against PyTorch's documented Join / equalize-to-minimum practice.
• Score: 6.384763560610076
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We report operational experience full-fine-tuning a 32.76B-parameter dense model (Qwen3-32B) on 16 x NVIDIA B300 (two nodes, FSDP / ZeRO-3) -- among the first published field accounts on this accelerator. We claim no new algorithm. The individual mechanisms we use are established practice; our contribution is the integrated field experience and a set of calibrated measurements on new hardware. Concretely we offer four practitioner artifacts. (1) A B300-calibrated power-draw triage table that distinguishes compute / communication / data-starvation / checkpoint-or-deadlock / idle by board wattage (utilization% reads 100% during an NCCL hang). (2) A set of honest negative results that dispel common optimization folklore at this scale: a controlled A/B in which per-step NFS reading matches a pretokenized local cache (~53k tok/s) because the corpus fits in page cache and the job is compute-bound; and a reconstruction of an earlier "throughput collapse" as NFS/CPU contention rather than a storage-medium limit. (3) Calibrated 4/8/16-GPU strong-scaling and GPU-hour numbers on B300 (near-linear, as expected in this regime; we report absolute values as reference data). (4) A worked failure case -- an epoch-end NCCL deadlock from per-rank token-packing imbalance -- together with a 2.7-second pre-run invariant gate and an external watcher that turn multi-hour silent failures into instant rejections. This deadlock and its remedy correspond to PyTorch's documented Join / equalize-to-minimum practice; we position our instantiation against that prior art and report the GPU-hours the failure cost and the gate saves. The transferable takeaway is operational, not algorithmic: for data-dependent data-parallel jobs, watch power rather than utilization, and verify invariants before launch -- a passing smoke test is not evidence of a safe full run.

Related papers

• Carry-Through Checksum: A Lightweight Fault-Detection for CNN Inference at the Edge [0.21748200848556343]
  We propose carry-through, a fundamentally new scheme for soft-error detection in CNN inference on embedded GPU.<n>The proposed method detects 95.86% and 86.56% of critical faults for FP32 and FP16, respectively, at almost no additional per-image overhead.
  arXiv  Detail & Related papers  (2026-09-15T07:15:30Z)
• VC-Attention: Value Smoothing and Softmax Casting for Low-bit Attention [47.30761996669105]
  We propose VC-Attention, a training-free low-bit attention framework.<n>It addresses both by smoothing value tokens by a probability Cast. V-Smooth reorders value tokens by lightweight clustering online, so the tokens in a fused hardware block quantize well together.<n>It maps log-domain scores directly to E4M3 probability codes with one fused multiply-add, eliminating the FP32 exponential and the format conversion.
  arXiv  Detail & Related papers  (2026-09-14T16:17:35Z)
• TriCalRAG: A Three-Strategy, Retrieval-Augmented Benchmark for On-Premise LLM-Based Root Cause Analysis in AIOps [1.8136615181861766]
  Cloud-hosted large language models (LLMs) are increasingly used for root cause analysis (RCA) in AIOps pipelines.<n>They introduce data privacy risk, network latency, and per-query cost that scale poorly with production log volumes.<n>We presentRAG, a benchmark evaluating open-weights served locally via vLLM against a classical LSTM-based log anomaly detector.
  arXiv  Detail & Related papers  (2026-09-13T19:52:46Z)
• Phase-Decoupled, Model-Calibrated Power Control for Disaggregated LLM Serving [1.249668212599889]
  We deploy NVIDIA's Max-Q inference profile on a disaggregated B200 system.<n>On an 8x B200 node serving Qwen3-Coder-480B (FP8) under agentic load, our balanced mode delivers +20.4% tokens/J at +3.5% mean e2e.
  arXiv  Detail & Related papers  (2026-09-10T06:23:37Z)
• SPORK: Self-Speculative Forking to Accelerate Agentic LLM Inference [6.978795233982147]
  SPORK dispatches the speculated tool call early, overlapping its execution with the remaining chain-of-thought decode.<n>SPORK is a training-free controller that dispatches the speculated tool call early, overlapping its execution with the remaining chain-of-thought decode.
  arXiv  Detail & Related papers  (2026-07-03T13:51:32Z)
• Benchmarking Confidential GPU Inference on NVIDIA H100 under Intel TDX [2.6853734738584047]
  This paper presents a benchmark study comparing standard non-confidential execution with confidential computing mode on a single NVIDIA H100 80GB GPU.<n>The results suggest that confidential GPU inference can retain usable throughput under load, but capacity planning must account for both the steady throughput penalty and the earlier saturation behavior observed for larger models.
  arXiv  Detail & Related papers  (2026-05-20T22:52:12Z)
• From Detection to Recovery: Operational Analysis on LLM Pre-training with 504 GPUs [1.2875754718643733]
  This technical report presents an empirical analysis of a 63-node NVIDIA B200 production cluster.<n>Five parties (SKT, Upstage, Lablup, NVIDIA Korea, and VAST Data) share a unified monitoring pipeline.<n>This arrangement enabled joint diagnosis of a 60-node-scale storage I/O bottleneck that did not appear at 2-4-node scale.
  arXiv  Detail & Related papers  (2026-05-10T06:46:06Z)
• Nondeterminism-Aware Optimistic Verification for Floating-Point Neural Networks [11.159101812934503]
  Nondeterministic Aware Optimistic verification protocol for floating-point execution on heterogeneous accelerators.<n>We implement NAO as a PyTorch-compatible runtime and a contract layer currently deployed on Holesky testnet.
  arXiv  Detail & Related papers  (2025-10-15T21:10:39Z)
• Sequential Kernelized Independence Testing [77.237958592189]
  We design sequential kernelized independence tests inspired by kernelized dependence measures.<n>We demonstrate the power of our approaches on both simulated and real data.
  arXiv  Detail & Related papers  (2022-12-14T18:08:42Z)
• Discriminative-Generative Dual Memory Video Anomaly Detection [81.09977516403411]
  Recently, people tried to use a few anomalies for video anomaly detection (VAD) instead of only normal data during the training process. We propose a DiscRiminative-gEnerative duAl Memory (DREAM) anomaly detection model to take advantage of a few anomalies and solve data imbalance.
  arXiv  Detail & Related papers  (2021-04-29T15:49:01Z)
• Beta-CROWN: Efficient Bound Propagation with Per-neuron Split Constraints for Complete and Incomplete Neural Network Verification [151.62491805851107]
  We develop $beta$-CROWN, a bound propagation based verifier that can fully encode per-neuron splits. $beta$-CROWN is close to three orders of magnitude faster than LP-based BaB methods for robustness verification. By terminating BaB early, our method can also be used for incomplete verification.
  arXiv  Detail & Related papers  (2021-03-11T11:56:54Z)
• I-BERT: Integer-only BERT Quantization [78.43819756382103]
  We propose I-BERT, a novel quantization scheme for Transformer based models. I-BERT performs an end-to-end integer-only BERT inference without any floating point calculation. We show that for both cases, I-BERT achieves similar (and slightly higher) accuracy as compared to the full-precision baseline.
  arXiv  Detail & Related papers  (2021-01-05T02:42:58Z)
• AQD: Towards Accurate Fully-Quantized Object Detection [94.06347866374927]
  We propose an Accurate Quantized object Detection solution, termed AQD, to get rid of floating-point computation. Our AQD achieves comparable or even better performance compared with the full-precision counterpart under extremely low-bit schemes.
  arXiv  Detail & Related papers  (2020-07-14T09:07:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.