Benchmarking LLM-Guided Control-Plane Policies for Backend Fault Isolation in HAProxy

• URL: http://arxiv.org/abs/2608.10532v1
• Date: Tue, 11 Aug 2026 06:15:20 GMT
• Title: Benchmarking LLM-Guided Control-Plane Policies for Backend Fault Isolation in HAProxy
• Abstract summary: We ask whether a Large Language Model can replace the static routing policy itself.<n>We find a capability threshold near 3B active parameters.<n>Above it, every model, regardless of architecture, saturates near an 88% reduction in client-perceived 5xx errors.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Static load balancers cannot mitigate a backend that is degraded rather than down: round-robin and least-connections keep routing traffic to a server returning HTTP 500s until an operator intervenes. We ask whether a Large Language Model can replace the static routing policy itself, reading HAProxy and Prometheus telemetry every 10 seconds and isolating faulty servers through guardrailed calls to the HAProxy Data Plane API. On a reproducible benchmark with a persistent structural fault built into roughly one-third of a heterogeneous fleet, we sweep 15 open-weight models across five families (0.35B to 35B total parameters; dense, mixture-of-experts, and efficient-sparse architectures), reasoning modes, fleet scales of 3 to 9 backends, and two routing algorithms, totaling 240 runs. We find a capability threshold near 3B active parameters. Below it, LLM policies are typically unreliable and sometimes worse than no policy; above it, every model, regardless of architecture, saturates near an 88% reduction in client-perceived 5xx errors over the static baseline. The threshold is approximate: Gemma 4 E2B clears it with 2B active parameters, while the dense 3B Granite 4.0 Micro does not. The availability gain has costs. Draining concentrates load onto surviving servers, inflating tail latency 2.6 to 2.8 times, and enabling reasoning multiplies token spend roughly tenfold, overrunning the control interval and degrading effectiveness. The efficient operating point is a supra-threshold model in its cheapest non-reasoning mode, wrapped inside deterministic guardrails.

Related papers

• Phase-Decoupled, Model-Calibrated Power Control for Disaggregated LLM Serving [1.249668212599889]
  We deploy NVIDIA's Max-Q inference profile on a disaggregated B200 system.<n>On an 8x B200 node serving Qwen3-Coder-480B (FP8) under agentic load, our balanced mode delivers +20.4% tokens/J at +3.5% mean e2e.
  arXiv  Detail & Related papers  (2026-09-10T06:23:37Z)
• From Production Traffic to Post-Training: Building a Self-Hosted LLM That Covers the Corporate Request Mix [55.08143827481755]
  We consolidate traffic from over 200 internal applications onto a single model.<n>We train a separate GRPO expert per axis and merge them via two-stage SLERP.<n>The model absorbs 50% of platform traffic, 116M requests per month, at a fraction of the serving cost.
  arXiv  Detail & Related papers  (2026-09-01T17:39:26Z)
• LMSM: LLM Security Framework Inspired by Linux Security Modules [56.93644694627783]
  Large language models (LLMs) are increasingly deployed with layered defenses, yet malicious prompts can still bypass them.<n>We present Language Model Security Modules (LMSM), a security framework that adapts the separation behind Linux Security Modules (LSM) to LLM serving.
  arXiv  Detail & Related papers  (2026-08-26T12:13:05Z)
• Cacheable by Design? Training Mixture-of-Experts Routers for Locality Against the Edge Memory-Bandwidth Wall: A Pre-Registered Negative Result with a Systems Measurement Study [0.0]
  We quantify bandwidth wall on Qwen3-235B (Q4_K_M, 134 GB)<n>We build llamamoe-trace, a zero-surgery router-telemetry tool, and measure routing on Q3-30B.<n>We pre-register training of 137M MoE language models with auxiliary locality and domain router losses.
  arXiv  Detail & Related papers  (2026-08-18T19:23:32Z)
• Nanbeige4.2-3B on Apple Silicon: Fixing Deployment Bugs and Decreasing Looped Transformer Memory Overhead [0.5076419064097732]
  Nanbeige4.2-3B is built around a Looped Transformer (LT) that reuses one stack layer for a second forward pass.<n>We show that fixing these bugs is still not sufficient for agentic tasks.<n>We introduce a chunked-prefill strategy which alleviates the incurred memory penalty.
  arXiv  Detail & Related papers  (2026-08-14T06:05:26Z)
• DREAM Technical Report [53.37525197047151]
  We present DREAM, an autonomous optimization control architecture that adds a perception-aware, orchestrable, and auditable policy layer atop existing pipelines without replacing them.<n> DREAM has two core components. First, a three-tier Intent Engine fuses on-device signals into structured L0/L1/L2 intent representations; its edge-cloud trigger chain reduces reporting volume to approximately 8.7%.<n>Second, a Meta Engine uses a MetaModel for layered M1-to-M2-to-M3 reasoning: intent summarization, strategy planning informed by Strategy Memory, and parameter translation. It dispatches the resulting parameters through a unified outlet with
  arXiv  Detail & Related papers  (2026-08-10T10:35:32Z)
• Real-Time Detection and Repair of LLM Agent Failures [0.0]
  LLM agents fail mid-episode loop, cascade tool errors, drift off goal, fabricate results, or silently absorb corrupted content.<n>We ask how much detection is achievable from observable step alone, using monitors costing microseconds per step and trained only on healthy runs.<n>System runs at 200 microseconds per step, three orders of magnitude below a judge call.
  arXiv  Detail & Related papers  (2026-08-03T16:34:46Z)
• Autonomous Topology Mutation: Safe Runtime Restructuring for Multi-Agent LLM Systems with Capability, State, and Shadow Invariants [0.11470070927586014]
  We introduce Autonomous Topology Mutation (ATM), a runtime team-mutation mechanism for multi-agent LLM frameworks.<n>ATM combines telemetry-driven overload detection with three safety invariants that gate each structural change.<n>On 720 DeepSeek-V3-driven task runs with deterministic tool stubs across four ablation conditions and three workloads, the ATM factoriser split lifts code-task success from 3.3% to 61.7%.
  arXiv  Detail & Related papers  (2026-06-04T11:38:06Z)
• LayerRoute: Input-Conditioned Adaptive Layer Skipping via LoRA Fine-Tuning for Agentic Language Models [0.0]
  We introduce LayerRoute, a lightweight adapter that learns to selectively skip transformer blocks on a per-input basis.<n> LayerRoute augments each of the 24 transformer blocks in Qwen2.5-0.5B-Instruct with: (1) a per-layer router that outputs a hard binary gate via the straight-through estimator, and (2) LoRA adapters (rank 8, 1.08M parameters) on the Q/K/V/O attention projections.<n>A single end-to-end training pass on agentic data (Hermes, Glaive, GSM8K, Turing) with a gate regularisation term
  arXiv  Detail & Related papers  (2026-06-01T07:50:05Z)
• HyDRA: Hybrid Dynamic Routing Architecture for Heterogeneous LLM Pools [2.0320563270126493]
  We present HyDRA, a framework that predicts fine-grained, multi-dimensional capability requirements per query.<n>A shortfall-matching algorithm then selects the cheapest model whose capabilities meet the predicted requirements.<n>The framework is deployed to all users in GitHub Copilot's VS Code Chat auto-mode.
  arXiv  Detail & Related papers  (2026-05-16T18:19:30Z)
• MEME: Multi-entity & Evolving Memory Evaluation [76.57263966646404]
  MEME defines six tasks spanning the full space defined by the multi-entity and evolving axes.<n> evaluating six memory systems spanning three memory paradigms on 100 controlled episodes.<n>We find that all systems collapse on dependency reasoning under the default configuration.
  arXiv  Detail & Related papers  (2026-05-12T17:55:10Z)
• Enforcing Benign Trajectories: A Behavioral Firewall for Structured-Workflow AI Agents [0.0]
  codename is a telemetry-driven behavioral anomaly detection firewall.<n>codename compiles verified benign tool-call telemetry into a parameterized deterministic finite automaton.<n>codename achieves a 5.6% macro-averaged attack success rate (ASR) across five scenarios.
  arXiv  Detail & Related papers  (2026-04-29T04:02:59Z)
• Full-Duplex-Bench-v3: Benchmarking Tool Use for Full-Duplex Voice Agents Under Real-World Disfluency [61.68376148916503]
  We introduce Full-Duplex-Bench-v3 (FDB-v3), a benchmark for evaluating spoken language models under naturalistic speech conditions and multi-step tool use.<n>Unlike prior work, our dataset consists entirely of real human audio annotated for five disfluency categories, paired with scenarios requiring chained API calls across four task domains.<n>We evaluate six model configurations -- GPT-Realtime, Gemini Live 2.5, Gemini Live 3.1, Grok, Ultravox v0.7, and a traditional Cascaded pipeline (Whisper$rightarrow$rightarrow$TTS) -- across accuracy, latency, and turn-
  arXiv  Detail & Related papers  (2026-04-06T16:46:52Z)
• LLM Readiness Harness: Evaluation, Observability, and CI Gates for LLM/RAG Applications [51.56484100374058]
  We present a readiness harness for LLM and RAG applications that turns evaluation into a deployment decision workflow.<n>The system combines automated benchmarks, OpenTelemetry observability, and CI quality gates under a minimal API contract.<n>We evaluate the harness on ticket-routing and BEIR tasks with full Azure matrix coverage.
  arXiv  Detail & Related papers  (2026-03-28T18:03:32Z)
• Step 3.5 Flash: Open Frontier-Level Intelligence with 11B Active Parameters [169.7981969517903]
  Step 3.5 Flash bridges frontier-level agentic intelligence and computational efficiency.<n>We focus on what matters most when building agents: sharp reasoning and fast, reliable execution.
  arXiv  Detail & Related papers  (2026-02-11T07:53:51Z)
• HiRE: High Recall Approximate Top-$k$ Estimation for Efficient LLM Inference [68.59839755875252]
  HiRE comprises of two novel components: (i) a compression scheme to cheaply predict top-$k$ rows/columns with high recall, followed by full computation restricted to the predicted subset, and (ii) DA-TOP-$k$: an efficient multi-device approximate top-$k$ operator. We demonstrate that on a one billion parameter model, HiRE applied to both the softmax as well as feedforward layers, achieves almost matching pretraining and downstream accuracy, and speeds up inference latency by $1.47times$ on a single TPUv5e device.
  arXiv  Detail & Related papers  (2024-02-14T18:04:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.