FinalityBench: An Effect-Level Benchmark for Agent Decisions Under Delayed and Conflicting Financial Finality

• URL: http://arxiv.org/abs/2609.04706v1
• Date: Fri, 04 Sep 2026 04:24:10 GMT
• Title: FinalityBench: An Effect-Level Benchmark for Agent Decisions Under Delayed and Conflicting Financial Finality
• Abstract summary: FinalityBench is an executable benchmark for a finality-gating decision.<n> Grading is on executed monetary effects: an episode is scored by the merchant's terminal economic position.<n>A runtime gating irreversible actions on an authoritative finality probe reaches 85.4% and, unlike every polling policy, loses nothing to pass5.
• Score: 1.350933217151331
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: A merchant's payment processor, ledger, ERP and bank feed are updated by messages that get delayed, duplicated, dropped and reordered, so for minutes at a time the four hold contradictory beliefs about the same order. An agent resolving the exception must decide whether to ship goods, re-submit a capture, refund or wait, knowing some of those cannot be undone. We present FinalityBench, an executable benchmark for that decision. It keeps a hidden canonical event log and derives each system's view from a separately faulted delivery stream, so disagreement follows from specified fault semantics rather than being authored. Grading is on executed monetary effects: an episode is scored by the merchant's terminal economic position, relative to a privileged reference told when the pending capture resolves. The corpus of 321 tasks includes 45 twin pairs (90 tasks): tasks whose four system views are identical at the decision instant, whose authoritative probes both return unknown, and whose eventual correct dispositions differ. That snapshot indistinguishability is checked under every evaluation seed rather than assumed; equivalence over all interaction traces is not claimed. Over 14,445 graded episodes from nine programmatic policies, ranking by single-task accuracy and by paired loss disagree in 7 places: a ship-on-first-sign policy is second-best by accuracy at 65.7% and worst in the suite by paired loss, because it cannot tell the two members apart. A runtime gating irreversible actions on an authoritative finality probe reaches 85.4% and, unlike every polling policy, loses nothing to pass^5; its residual loss is almost entirely one archetype, which prices finality information directly. Language models reach the same exact rate as the hand-written gate on a stratified subset, lose about twice as much money, and discover the finality-gating strategy without being told it.

Related papers

• IBIB: A Protocol for Measuring Enterprise AI Systems by Serving Route, Not Model Identifier [0.0]
  All 18 audited benchmarks score advertised model identifiers.<n>A gold-blind capability-binding preflight verifies that a route can execute the evaluation contract before any task reaches it.<n>A reliability-inclusive first-pass scoring rule keeps failure in the score while keeping unsupported capability out.
  arXiv  Detail & Related papers  (2026-09-09T17:31:29Z)
• Agent Mesh: Reliability Primitives for Non-Idempotent Agent Delegation - Identity Adequacy and Evidence Adequacy [0.0]
  We report a failure study of a production agentic software-delivery platform over 147 numbered incidents spanning 81 runs.<n>We quantify the consequences: a loop of fifty-four consecutive successful tool calls no error-rate breaker could see.<n>We derive seven reliability primitives whose enforcement unit is the delegation rather than the message.
  arXiv  Detail & Related papers  (2026-08-26T15:38:21Z)
• When Stale Constraints Go Unchecked: Budgeted Verification Failures in Inherited Agent Memory [0.0]
  Provenance links keep the evidence behind an inherited belief reachable; an agent with a verification budget must still choose which links to inspect.<n>We study a consolidated memory that states a decision constraint and whose source record has since been superseded by a record that withdraws it.<n>16 language models rarely re-verified a constraint that read as settled.<n>Once the constraint had been superseded, produced stale-consistent decisions in 77.3%, 74.7% and 74.7% of episodes across a primary run, a replication and a held-out domain.
  arXiv  Detail & Related papers  (2026-08-26T09:04:21Z)
• IndicQE-APE: A Benchmark for Quality Estimation and Automatic Post-Editing for Indic Languages [77.88427845181154]
  We consolidate the WMT 2020--2024 shared-task lineage with an extended English--Malayalam resource into indicqe: $126,754$ instances over nine directional pairs.<n>We benchmark six prompted LLMs and three COMET metrics on segment-level QE, and three systems on APE.
  arXiv  Detail & Related papers  (2026-08-17T09:50:52Z)
• DarwinX: Evolving Agent Harnesses Through Natural Selection [48.64258219266629]
  DarwinX treats self-evolution as selection over a population of harnesses with the model frozen.<n>General agent competence, not benchmark-specific patches, so it survives changes of task, verifier, and base model.
  arXiv  Detail & Related papers  (2026-07-31T05:34:02Z)
• Abliteration Is Not a Scalpel: Off-Target Effects of Refusal Removal on Decision Disposition Across Model Families [51.56484100374058]
  Abliteration - deleting a model's refusal direction from its weights - is the standard recipe behind popular "uncensored" open-weight models.<n>As a disposition probe we use 21,600 decisions under uncertainty, replayed through a frozen pipeline so the decision-layer model is the only variable.<n>Three effects replicate across both families (weeks-clustered CIs excluding zero)<n>A fourth effect reverses sign: the same operation makes Gemma-abliterated less confident and Qwen-abliterated more (family CIs non-overlapping)
  arXiv  Detail & Related papers  (2026-07-19T22:27:00Z)
• Fantastic Adaptive Taxonomies and How to Use Them [100.20614173157708]
  AdaMAST converts execution traces into a compact, evidence-grounded failure taxonomy.<n>No code is hand-authored, no trace human-annotated.<n>In agent-system search, taxonomy-coded diagnoses of failed candidates outperform free-form.
  arXiv  Detail & Related papers  (2026-07-17T17:41:16Z)
• STOCKTAKE: Measuring the Gap Between Perception and Action in LLM Agents with a Fair Oracle [0.0]
  LLM agents are increasingly evaluated on multi-week decision tasks in which the state that drives cost is never directly observed.<n>We introduce STOCKTAKE, a 26-week supply-chain replenishment benchmark built as a factored partially observable Markov decision process.
  arXiv  Detail & Related papers  (2026-07-15T09:08:27Z)
• What Accuracy and Gradient Cosine Miss: Evaluating Feedback Alignment via Scale Stability, Reference Validity, and Depth Utility [48.10132234701036]
  We propose a diagnostic evaluation protocol based on three checks -- scale stability, reference validity, and depth utility -- together with per-layer rather than aggregate cosine reporting.<n>Across multiple architectures and methods, our protocol identifies all failures with wide calibration margins.
  arXiv  Detail & Related papers  (2026-06-19T06:04:17Z)
• Catching One in Five: LLM-as-Judge Blind Spots in Production Multi-Turn Transaction Agents [45.148328075418156]
  We study a deployed multi-turn food-and-beverage ordering agent and measure how many genuine quality problems it catches.<n>Our blind-spot taxonomy shows the failure is structured, not random.<n>For production multi-turn agents, automated judging is a regression floor, not a substitute for human review.
  arXiv  Detail & Related papers  (2026-06-09T02:11:01Z)
• Strained Coherence: A Pre-Failure Signal in Coding Agent Execution Trajectories [0.0]
  LLM-based coding agents sometimes acknowledge a problem in their own reasoning and then proceed anyway.<n>We build a Claude Sonnet 4.6 judge that reads full trajectories and flags spans where the pattern occurs.<n>We evaluate it on 44 Terminal-bench-2 trajectories using a Qwen3.5-35B-A3B backbone.
  arXiv  Detail & Related papers  (2026-06-05T22:52:16Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.