Similarity Gates Approve Reversals: A Validity Audit of Embedding-Cosine Thresholds in Agent Systems

• URL: http://arxiv.org/abs/2608.10216v1
• Date: Mon, 10 Aug 2026 20:36:40 GMT
• Title: Similarity Gates Approve Reversals: A Validity Audit of Embedding-Cosine Thresholds in Agent Systems
• Abstract summary: Agent frameworks ship quality gates that compare text blocks by embedding-cosine similarity and decide at a fixed cutoff.<n>Deduplication filters, semantic caches, drift guards, and answer grader gates deploy to answer the question: "Does this text still mean the same thing?"<n>We audit this gate class as a measurement instrument.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Agent frameworks ship quality gates that compare text blocks by embedding-cosine similarity and decide at a fixed cutoff. Deduplication filters, semantic caches, drift guards, and answer grader gates deploy to answer the question: "Does this text still mean the same thing?" But the score answers a different question: "How much did the wording change?" We audit this gate class as a measurement instrument. In the cases these gates exist to catch, the two can run in opposite ways. Many times, reversing an instruction is a single word edit, while agreement often rephrases a sentence. The consequence is a safety check that fires backwards. The production drift guard we audited caught 0 of 56 meaning-breaking mutations, and one approved item, "withhold the study drug" -> "administer the study drug", came in at cosine 0.9608. We observed five shipped operating points, and balanced accuracy across 90 configuration-threshold-task cells never exceeded 0.700 (median 0.525). The same confounder also corrupted evaluations. A naively built corpus inherits this confounder and can return an inverted verdict, with a decision AUROC exactly 0.000 in 13 of 18 configuration-task cells (at most 0.040 in all 18) against 0.440-0.815 for the same nine configurations under a balanced 2x2 design. Twice in the effort it captured our own headline claims. Obvious repairs fail: an encoder swap and an overlap-conditioned gate (0.750 in-sample, 0.533 held-out) land at chance on separately authored held-out data, and an NLI drop-in did no better. Embeddings do still bear hope here, as the strongest two of nine configurations separated reversal from paraphrase at matched overlap (AUROC 0.79-0.90), but only a matched-pair audit reveals the deployment regime. We release the corpus method, harness, and frozen results, and contend that scores gated this way measure the wrong thing. We believe a valid instrument is buildable.

Related papers

• What Output-Equivalence Oracles Miss: An Empirical Study of Equivalence-Invisible Bug Fixes in Quantum Transpilers (Qiskit, tket, Cirq) [0.25489046505746704]
  Quantum compilers are judged correct by an output-equivalence.<n>We measure how often this happens in real merged compiler fixes.<n>A conservative floor remains even restricted to the one unconditionally equivalence-invisible channel.
  arXiv  Detail & Related papers  (2026-09-12T09:49:43Z)
• IBIB: A Protocol for Measuring Enterprise AI Systems by Serving Route, Not Model Identifier [0.0]
  All 18 audited benchmarks score advertised model identifiers.<n>A gold-blind capability-binding preflight verifies that a route can execute the evaluation contract before any task reaches it.<n>A reliability-inclusive first-pass scoring rule keeps failure in the score while keeping unsupported capability out.
  arXiv  Detail & Related papers  (2026-09-09T17:31:29Z)
• Bilevel Coordinated Reflection: A Game-Theoretic Approach to Multi-Agent LLM Systems [40.47928908205563]
  Multi-agent LLM systems commonly use an orchestrator to decompose a task for a team of workers and then improve through textual reflection.<n>Despite strong empirical results, these systems lack a unified account of coordination, memory improvement, and the role of external verification.<n>We provide confidence gating for evaluation and re-anchoring guarantees for piecewise-stationary environments.<n>Experiments instantiate these objects with environment-grounded metrics and test the predicted coordination and drift laws.
  arXiv  Detail & Related papers  (2026-09-02T15:50:10Z)
• Measuring the Wrong Thing: Internal Harmfulness Scores Anti-Rank Successful Jailbreaks [8.249133191294703]
  Internal safety scores judge a prompt before any text is generated.<n>A filter tuned on a score that measures the wrong quantity spends its false positive budget on attacks that would have failed anyway.<n>We introduce Active Attention Probing, which supplies a fixed content independent measurement coordinate.
  arXiv  Detail & Related papers  (2026-08-10T14:05:32Z)
• Trajectories That Segment Themselves: Agent-Declared Boundaries as a Training Unit [0.0]
  We introduce collection-time semantic self-segmentation, in which a declarative contract has the acting agent expose its own boundaries while the trajectory is generated.<n>Because the agent names its conjecture, a reviewer can negate it by name, which lets our protocol manufacture wrong-cause-then-correction transitions that recorded work rarely contains.<n>We then ask what survives deleting the declaration. Given the cut points but not the hypothesis, a model attributes action blocks to their governing hypothesis at over twice chance.<n>Asked instead to place boundaries, a code-blind annotator matches 24 of 40 where random placement matches 11.5, while a mechanical test
  arXiv  Detail & Related papers  (2026-08-03T14:27:58Z)
• Explaining AI-Image Detection: What the Heatmap Actually Shows [65.01025489527173]
  We build a detector and attach an attribution map as its evidence.<n>We measure what that pair delivers on 186,527 images under controls.<n>Whether an attribution ranking exists at all depends on whether the detector reacts to the image.
  arXiv  Detail & Related papers  (2026-07-31T16:07:05Z)
• Looping Is Not Reliability: State-Bound Evidence and Typed Revision Contracts for Agentic Code Repair [36.56438114281786]
  We study the gap between finding a correct patch and retaining, verifying, and submitting it.<n>We derive an evidence-bound typed loop contract and instantiate its mechanically enforceable subset in a reference implementation.
  arXiv  Detail & Related papers  (2026-07-27T16:05:23Z)
• Adversarial Test-Hardening for AI-Written Code: An Instrument Autopsy and a Pre-Registered Causal Estimate of the Critic Loop [0.0]
  We study an adversarial test-hardening loop under a mechanical oracle.<n>A Tester model writes tests, mutation testing names surviving injected defects, and a Critic model writes tests to kill exactly those.<n>We release both protocols, all receipts, and the analysis code.
  arXiv  Detail & Related papers  (2026-07-25T02:38:07Z)
• Abliteration Is Not a Scalpel: Off-Target Effects of Refusal Removal on Decision Disposition Across Model Families [51.56484100374058]
  Abliteration - deleting a model's refusal direction from its weights - is the standard recipe behind popular "uncensored" open-weight models.<n>As a disposition probe we use 21,600 decisions under uncertainty, replayed through a frozen pipeline so the decision-layer model is the only variable.<n>Three effects replicate across both families (weeks-clustered CIs excluding zero)<n>A fourth effect reverses sign: the same operation makes Gemma-abliterated less confident and Qwen-abliterated more (family CIs non-overlapping)
  arXiv  Detail & Related papers  (2026-07-19T22:27:00Z)
• Falsifiable Release Gates for Self-Improving Systems [0.0]
  We describe falsifiable release gates, and a methodology to build and validate such systems.<n>Every new capability must pass a pre-specified, machine-verifiable acceptance suite before it ships.<n>No action goes to an effector without a safety-critical property capability token minted by a control ring.
  arXiv  Detail & Related papers  (2026-07-11T06:06:34Z)
• What Accuracy and Gradient Cosine Miss: Evaluating Feedback Alignment via Scale Stability, Reference Validity, and Depth Utility [48.10132234701036]
  We propose a diagnostic evaluation protocol based on three checks -- scale stability, reference validity, and depth utility -- together with per-layer rather than aggregate cosine reporting.<n>Across multiple architectures and methods, our protocol identifies all failures with wide calibration margins.
  arXiv  Detail & Related papers  (2026-06-19T06:04:17Z)
• Layer-Isolated Evaluation: Gating the Deterministic Scaffold of a Production LLM Agent with a No-LLM, Regression-Locked Test Harness [45.148328075418156]
  A deployed ordering agent is decomposed into a fixed taxonomy of layers.<n>The pure suite runs in CI on every change against a locked per-slice baseline.<n>We validate by controlled regression injection, degrading one layer at a time across seven non-safety layers.
  arXiv  Detail & Related papers  (2026-06-10T05:55:13Z)
• Catching One in Five: LLM-as-Judge Blind Spots in Production Multi-Turn Transaction Agents [45.148328075418156]
  We study a deployed multi-turn food-and-beverage ordering agent and measure how many genuine quality problems it catches.<n>Our blind-spot taxonomy shows the failure is structured, not random.<n>For production multi-turn agents, automated judging is a regression floor, not a substitute for human review.
  arXiv  Detail & Related papers  (2026-06-09T02:11:01Z)
• Distributional Energy-Based Models for Uncertainty-Aware Structured LLM Reasoning [40.342912574072024]
  Large Language Models produce structured outputs such as travel plans or code solutions.<n>Individual reasoning steps may appear correct while the output as a whole violates budgets, fails test cases, or contradicts earlier deductions.<n>We propose a learned quality scorer with deterministic analytical constraint penalties for verifying structured LLM outputs.
  arXiv  Detail & Related papers  (2026-05-15T17:08:27Z)
• Controlling Authority Retrieval: A Missing Retrieval Objective for Authority-Governed Knowledge [0.0]
  We call this CAR: retrieving the currently active authority for a semantic anchor q, that is, front(cl(A_k(q))).<n>We evaluate three real-world datasets: security advisories, SCOTUS overruling pairs, and FDA drug records.<n>A GPT-4omini experiment shows RAG produces explicit "not patched" claims for 39% of queries where a patch exists.
  arXiv  Detail & Related papers  (2026-04-15T23:56:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.