A Universal Cliff and a Design Fingerprint: Cross-Section Defect Detection Under LLM Orchestration

• URL: http://arxiv.org/abs/2605.26174v1
• Date: Mon, 25 May 2026 05:09:48 GMT
• Title: A Universal Cliff and a Design Fingerprint: Cross-Section Defect Detection Under LLM Orchestration
• Abstract summary: Production language-model systems answer a request byspanning it across an invisible orchestration of worker agents.<n>We ask what this does to a class of defect no single worker can see.<n>We vary only the model -- ten systems across five generations from one developer and five providers from distinct alignment paradigms.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Production language-model systems answer a request by partitioning it across an invisible orchestration of worker agents that recompose one integrated report. We ask what this does to a class of defect no single worker can see: a contradiction in the relation between two distant sections of a document. Holding the documents, defects, mechanism, scoring, and seed fixed, we vary only the model -- ten systems across five generations from one developer and five providers from distinct alignment paradigms. Two layers separate. First, a universal detection cliff: every model that finds these cross-section defects under a single agent loses that ability under orchestration, detection falling two-thirds or more across every paradigm tested. The cliff is mechanism-derived and not closed by scale or extended reasoning. Second, how models behave once fallen. A signal-detection decomposition shows that, among the six models discriminating above chance, only one developer's generations move along the reporting-criterion axis: as alignment is strengthened, the model misses fewer defects yet raises more false alarms on clean documents -- two faces of one criterion shift, scaling with generation within that developer (p < 0.001) and near-absent elsewhere. At the floor the missed defect is often not out of view: the model's private record reconstructs the structural fault accurately, while the integrated report signs off on its soundness, its concern spent on the artifact and an absent collaborator. This resists quantification -- an automated judge is unstable (precision 17-50%) and keywords cannot separate it from ordinary agreement -- a resistance we report as a finding. We release all runs, probes, defect keys, scorer prompts, and scripts. An integrated report's confidence is uninformative about partition-spanning defects, the most aligned systems are not the safest, and the cliff is structural.

Related papers

• Calibrating Interpretability Instruments Before Trusting Their Verdicts [1.0829694003408499]
  Causal claims about large language model (LLM) internals rest on measurements.<n>These measurements fail in specific, diagnosable ways that return a plausible number instead of an error.<n>This note documents six such failures from a causal interpretability program on refusal and moral representation.
  arXiv  Detail & Related papers  (2026-09-13T19:37:02Z)
• Calibration is the Bottleneck: An Action-Class Diagnostic of Multi-Turn Tool-Calling [65.03665015319457]
  We propose an action-class-oriented diagnostic framework that decomposes multi-turn failures into two modes: action-class miscalibration and action-execution failure.<n>We validate it on a panel of tool-calling models across multiple multi-turn benchmarks.
  arXiv  Detail & Related papers  (2026-09-01T09:08:15Z)
• PhenoIntel: A Lifecycle-Aligned Multi-Agent Web Application for Verified, Accessible Plant Phenotype Analysis [0.0]
  PhenoIntel is a web platform that turns the full machine-learning workflow into a reliable phenotyping system.<n>Nine specialised agents divide the analysis into stages, from image collection through model selection, inference, and reporting.<n>Independent checks separate these stages, and every agent reads from and writes to one shared, fixed-structure record.
  arXiv  Detail & Related papers  (2026-08-28T07:06:50Z)
• Runtime Observability for Heterogeneous Attention Memory [20.158719055939383]
  We give a runtime observability contract that covers all four memory classes with three operators.<n>We compose the per-stage stack into an executable request-level risk ledger.
  arXiv  Detail & Related papers  (2026-08-06T10:41:50Z)
• The LLM Proposes, the Executive Disposes: A Self-Verifying Agent Instrument that Dissociates Commitment Drift from Binding Drift in Long-Horizon Agents [0.0]
  We present an agent instrument built so that verification is structural rather than post-hoc.<n>Every run invalidates itself when per-organ write-error, render-size, or salted-canary-echo floors are breached.<n>We report a clean, single-variable result on a failure every long-horizon agent suffers.
  arXiv  Detail & Related papers  (2026-08-04T15:10:37Z)
• Fantastic Adaptive Taxonomies and How to Use Them [100.20614173157708]
  AdaMAST converts execution traces into a compact, evidence-grounded failure taxonomy.<n>No code is hand-authored, no trace human-annotated.<n>In agent-system search, taxonomy-coded diagnoses of failed candidates outperform free-form.
  arXiv  Detail & Related papers  (2026-07-17T17:41:16Z)
• DriftScope: Measuring The Hidden Effects of Diffusion Model Adaptation [50.584834483403675]
  Adapting pre-trained text-to-image diffusion models, whether to learn new visual concepts or erase unwanted ones, is routinely evaluated on its intended effects alone.<n>We demonstrate that adaptation systematically damages semantically unrelated concepts in ways that aggregate metrics structurally cannot surface.<n>We introduce DriftScope, a prompt-level diagnostic tool that takes any two model checkpoints and returns a ranked list of tokens whose visual concepts have shifted most between them.
  arXiv  Detail & Related papers  (2026-06-30T20:57:51Z)
• AuthTrace: Diagnosing Evidence Construction in Thematically Dense Single-Author Corpora [6.956097396264084]
  We introduce AuthTrace, a diagnostic benchmark built on thematically dense single-author corpora.<n>AuthTrace provides explicit quoted evidence, exact fan-in annotation, and a unified pack-level protocol measuring evidence recall, evidence precision, and answer correctness.
  arXiv  Detail & Related papers  (2026-05-25T03:10:52Z)
• Coordination as an Architectural Layer for LLM-Based Multi-Agent Systems [0.0]
  Multi-agent LLM systems fail in production at rates between 41% and 87%, mostly due to coordination defects.<n>We argue that coordination should be treated as a architectural layer, separable from agent logic and from information access.<n>We instantiate this with an information-controlled design on prediction markets.
  arXiv  Detail & Related papers  (2026-05-05T02:56:44Z)
• Taming Actor-Observer Asymmetry in Agents via Dialectical Alignment [59.536125286960186]
  Multi-agent frameworks assigning specialized roles are increasingly adopted to enable self-reflection and mutual auditing.<n>We find it simultaneously induces a human-like cognitive bias known as Actor-Observer Asymmetry.<n>ReTAS guides agents to synthesize conflicting viewpoints into an objective consensus.
  arXiv  Detail & Related papers  (2026-04-21T15:05:58Z)
• Silent Commitment Failure in Instruction-Tuned Language Models: Evidence of Governability Divergence Across Architectures [0.0]
  We introduce governability -- the degree to which a model's errors are detectable before output commitment and correctable once detected.<n>We show benchmark accuracy does not predict governability, correction capacity varies independently of detection, and identical governance scaffolds produce opposite effects across models.<n>We propose a Detection and Correction Matrix classifying model-task combinations into four regimes: Governable, Monitor Only, Steer Blind, and Ungovernable.
  arXiv  Detail & Related papers  (2026-03-22T21:50:28Z)
• Retrieval-Infused Reasoning Sandbox: A Benchmark for Decoupling Retrieval and Reasoning Capabilities [32.76303717104482]
  We introduce DeR2, a controlled deep-research sandbox that isolates document-grounded reasoning.<n>DeR2 decouples evidence access from reasoning via four regimes--Instruction-only, Concepts, Related-only, and Full-set.<n> Experiments across a diverse set of state-of-the-art foundation models reveal substantial variation and significant headroom.
  arXiv  Detail & Related papers  (2026-01-29T16:26:19Z)
• From Feature Interaction to Feature Generation: A Generative Paradigm of CTR Prediction Models [81.43473418572567]
  Click-Through Rate (CTR) prediction is a core task in recommendation systems.<n>We propose a novel generative framework to address embedding dimensional collapse and information redundancy.<n>We show that SFG consistently mitigates embedding collapse and reduces information redundancy, while yielding substantial performance gains.
  arXiv  Detail & Related papers  (2025-12-16T03:17:18Z)
• Noise & pattern: identity-anchored Tikhonov regularization for robust structural anomaly detection [58.535473924035365]
  Anomaly detection plays a pivotal role in automated industrial inspection, aiming to identify subtle or rare defects in otherwise uniform visual patterns.<n>We tackle structural anomaly detection using a self-supervised autoencoder that learns to repair corrupted inputs.<n>We introduce a corruption model that injects artificial disruptions into training images to mimic structural defects.
  arXiv  Detail & Related papers  (2025-11-10T15:48:50Z)
• Metacognitive Self-Correction for Multi-Agent System via Prototype-Guided Next-Execution Reconstruction [58.51530390018909]
  Large Language Model based multi-agent systems excel at collaborative problem solving but remain brittle to cascading errors.<n>We present MASC, a metacognitive framework that endows MAS with real-time, unsupervised, step-level error detection and self-correction.
  arXiv  Detail & Related papers  (2025-10-16T05:35:37Z)
• Unsupervised Model Diagnosis [49.36194740479798]
  This paper proposes Unsupervised Model Diagnosis (UMO) to produce semantic counterfactual explanations without any user guidance. Our approach identifies and visualizes changes in semantics, and then matches these changes to attributes from wide-ranging text sources.
  arXiv  Detail & Related papers  (2024-10-08T17:59:03Z)
• Understanding Factual Errors in Summarization: Errors, Summarizers, Datasets, Error Detectors [105.12462629663757]
  In this work, we aggregate factuality error annotations from nine existing datasets and stratify them according to the underlying summarization model. We compare performance of state-of-the-art factuality metrics, including recent ChatGPT-based metrics, on this stratified benchmark and show that their performance varies significantly across different types of summarization models.
  arXiv  Detail & Related papers  (2022-05-25T15:26:48Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.