Boundary-Mutation Testing for Pattern-Based Secret Detection: A Rule-Level Method and Cross-Scanner Evaluation

• URL: http://arxiv.org/abs/2609.02983v1
• Date: Wed, 02 Sep 2026 13:41:50 GMT
• Title: Boundary-Mutation Testing for Pattern-Based Secret Detection: A Rule-Level Method and Cross-Scanner Evaluation
• Abstract summary: Boundary-mutation testing for pattern-based secret scanners.<n>Three scanners - a 43-rule open-source scanner, Gitleaks 8.21.2, and TruffleHog 3.82.13 - are tested.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Pattern-based secret scanners are commonly validated with example-based fixtures that fix one variable: the text surrounding a credential. We introduce boundary-mutation testing to vary that context, generating credentials from each rule's own regular expression, embedding them in realistic source contexts, and classifying outcomes at the rule level rather than the tool level, yielding three detection metrics. Applied to three scanners - a 43-rule open-source scanner, Gitleaks 8.21.2, and TruffleHog 3.82.13 - detection in the primary subject holds at >=0.9976 across ten contexts but collapses to 0.5233 when a credential ends in a hyphen. Five rules are affected: two, with fixed-count quantifiers, fail totally and deterministically; three, with variable-count quantifiers, backtrack and match a truncated credential; an entropy fallback rescues some failures but downgrades their severity. We validate a repair restoring full robustness with no new false positives, and report a caution: a plausible first attempt silently regressed two rules, caught only by re-running the same battery. Gitleaks has an unrelated, source-confirmed defect - a hard-coded terminator allowlist causing total misses for most credential types - while TruffleHog shows no boundary fragility but narrowest coverage. We report marginal, not conditional, failure probabilities: one common token format is structurally immune, another fails once in 64; on 292,527 lines of real code, the false-positive ordering inverts relative to the synthetic corpus. Because per-type detection is deterministic, the comparison unit is ten credential types, not hundreds of samples; no recall difference reaches significance, so we report the null result, not a ranking.

Related papers

• IBIB: A Protocol for Measuring Enterprise AI Systems by Serving Route, Not Model Identifier [0.0]
  All 18 audited benchmarks score advertised model identifiers.<n>A gold-blind capability-binding preflight verifies that a route can execute the evaluation contract before any task reaches it.<n>A reliability-inclusive first-pass scoring rule keeps failure in the score while keeping unsupported capability out.
  arXiv  Detail & Related papers  (2026-09-09T17:31:29Z)
• VeriPhy: Agentic Physical Reasoning for World Model Evaluation and Refinement [57.86962208273888]
  We present an auditable physical-verification system in which a text-only planner compiles the prompt into typed physical obligations.<n>Each action returns a provenance-carrying evidence record whose payload, when usable, is either a typed measurement or an explicitly tagged learned state.<n>We anchor evaluation in a 1,500-clip corpus of human-annotated flaw records that localize real generation failures in prompt reference, space, and time.
  arXiv  Detail & Related papers  (2026-09-02T20:36:45Z)
• Fidelity Is Not Enough: Dispatch-Level Instrumentation for Agentic Datasheet Extraction [6.419341495897254]
  We log every tool call in an agentic benchmark of 25 hand-curated claims over three components, with 12 more on a fourth, 37 in all.<n>We build two instruments: a rule-based failure-attribution classifier, and a silent-failure detector whose two rules check only which tools were called, never the extracted value.
  arXiv  Detail & Related papers  (2026-08-28T15:25:12Z)
• In-Cell Learning: Deployed Language Models Can Learn New Knowledge Without Changing a Single Stored Bit [4.726337413307433]
  We show that a 4-bit release stores each weight as an integer code and a shared scale, and leaves between the stored values an interval the quantizer discards.<n>We call in-cell learning the paradigm in which new knowledge is written only into that interval.<n>CellFill realizes the paradigm by construction: a low-rank position inside the cell, trained on the vendor's own 4-bit release.
  arXiv  Detail & Related papers  (2026-08-21T08:49:07Z)
• The Recall Trap: A Recall-Maximizing Retriever Configuration Reduces Issue Resolution in Fixed-Budget Code Context [45.88028371034407]
  We report a controlled case study in code repair, not a new phenomenon.<n>We toggle one flag (one-chunk-perfile deduplication) on an otherwise identical stack.<n>We map where it holds: it reverses on a lexical BM25 (3.2pp, significant cross-paradigm interaction)<n>We show that at a tight fixed budget, do not hard-deduplicate by file, and A/B packing policies against the task, not the metric flag was tuned.
  arXiv  Detail & Related papers  (2026-08-14T19:22:50Z)
• Fantastic Adaptive Taxonomies and How to Use Them [100.20614173157708]
  AdaMAST converts execution traces into a compact, evidence-grounded failure taxonomy.<n>No code is hand-authored, no trace human-annotated.<n>In agent-system search, taxonomy-coded diagnoses of failed candidates outperform free-form.
  arXiv  Detail & Related papers  (2026-07-17T17:41:16Z)
• How Much Human Label Variation Does Formal Semantic Structure Explain?: Group-Level Effects and Item-Level Ceilings in NLI [1.9177583131035885]
  Human label variation in natural language inference is increasingly treated as signal rather than noise.<n>We measure it on the 3,113 SNLI and MNLI items of ChaosNLI using a rule-based operator and monotonicity tagger validated against MED.<n>In this sample, formal semantic structure shifts how much annotators disagree by a small amount and does not detectably change what they disagree about.
  arXiv  Detail & Related papers  (2026-07-17T11:34:30Z)
• AdversaBench: Automated LLM Red-Teaming with Multi-Judge Confirmation and Cross-Model Transferability [16.27940179594792]
  AdversaBench is an end-to-end red-teaming pipeline that mutates seed prompts with five structured operators.<n>We report experiments on 45 seeds across three categories: reasoning, instruction-following, and tool use.
  arXiv  Detail & Related papers  (2026-06-23T13:50:51Z)
• What Accuracy and Gradient Cosine Miss: Evaluating Feedback Alignment via Scale Stability, Reference Validity, and Depth Utility [48.10132234701036]
  We propose a diagnostic evaluation protocol based on three checks -- scale stability, reference validity, and depth utility -- together with per-layer rather than aggregate cosine reporting.<n>Across multiple architectures and methods, our protocol identifies all failures with wide calibration margins.
  arXiv  Detail & Related papers  (2026-06-19T06:04:17Z)
• QuantileMark: A Message-Symmetric Multi-bit Watermark for LLMs [43.011999511057134]
  QuantileMark is a white-box watermark that embeds messages within a cumulative probability interval.<n>It partitions this interval into $M$ equal-mass bins and samples strictly from the bin assigned to the target symbol.<n>For detection, the verifier reconstructs the same partition under teacher forcing, computes posteriors over latent bins, and aggregates evidence for verification.
  arXiv  Detail & Related papers  (2026-04-15T12:22:40Z)
• Towards Token-Level Text Anomaly Detection [48.821180044375176]
  We introduce token-level anomaly detection, a novel paradigm that enables fine-grained localization of anomalies within text.<n>We formally define text anomalies at both document and token-levels, and propose a unified detection framework that operates across multiple levels.
  arXiv  Detail & Related papers  (2026-01-20T06:27:09Z)
• SpatialBench-UC: Uncertainty-Aware Evaluation of Spatial Prompt Following in Text-to-Image Generation [0.0]
  SpatialBench-UC is a small, reproducible benchmark for pairwise spatial relations.<n>We release a benchmark package, versioned prompts, pinned configs, per-sample checker outputs, and report tables.<n>We evaluate three baselines, Stable Diffusion 1.5, SD 1.5 BoxDiff, and SD 1.4 GLIGEN.
  arXiv  Detail & Related papers  (2026-01-19T23:37:10Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.