Abstention Errors and Segment Support in CUAD: An Auditable Contract-Extraction Case Study

• URL: http://arxiv.org/abs/2609.13274v1
• Date: Mon, 07 Sep 2026 21:26:53 GMT
• Title: Abstention Errors and Segment Support in CUAD: An Auditable Contract-Extraction Case Study
• Abstract summary: This paper presents a retrospective, reproducible evaluation of a fixed public RoBERTa checkpoint on CUAD's published 102-contract test split.<n>Of 30,464 returned candidate strings, 19,562 occur on questions with no annotated answer.<n>The system answers 1,335 of 2,938 such questions, a 45.4% false-positive rate.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Contract-clause extraction benchmarks measure reference recovery, but interpreting a system for review assistance also requires measuring unnecessary output and the evidence available within categories. This paper presents a retrospective, reproducible evaluation of a fixed public RoBERTa checkpoint on CUAD's published 102-contract test split. Candidate generation is preserved from an earlier frozen run; matching errors are corrected and category thresholds are reselected on 62 training-split contracts using the original 90% recall target. The corrected operating point recovers 82.2% of reference spans at 8.2% CUAD precision. Of 30,464 returned candidate strings, 19,562 occur on questions with no annotated answer. The system answers 1,335 of 2,938 such questions, a 45.4% false-positive rate (95% contract-bootstrap interval: 43.3-47.9%). Matching candidates constitute 20.3% of returned strings, illustrating why reference-based precision and candidate-level review burden require different denominators. Only nine of 41 categories have at least 30 positive and 30 negative contracts; this count changes to 18 and two under support minima of 20 and 40. Two categories have no negative contracts, making their no-answer false-positive rates undefined. These findings describe one checkpoint, decoder, and threshold policy. Earlier test access and uncertain checkpoint training overlap limit confirmatory interpretation. The paper supplies the original and corrected analyses, predictions, and software checks as ancillary artifacts; it establishes neither a new release criterion nor production readiness.

Related papers

• Turkish MMLU Pro: Traceable Option Augmentation and Its Validity Limits in Turkish Multiple-Choice Evaluation [0.0]
  Adding answer options can lower multiple-choice scores without improving assessment validity.<n>Turkish MMLU Pro examines this distinction using 12,000 Turkish-source questions across 58 sections.
  arXiv  Detail & Related papers  (2026-09-14T12:28:09Z)
• Certifying Model Upgrades with Slice-Wise Non-Regression and Incumbent Fallback [2.8716825293247665]
  We study checkpoint selection subject to non-regression tolerances relative to a retained incumbent.<n>We give a reproducible release procedure that separates candidate search from independent, paired evaluation.<n>A joint release decision does not require a slice-count Bonferroni penalty, although certification power can still decrease with the number of slices.
  arXiv  Detail & Related papers  (2026-09-12T05:00:50Z)
• Where the Verifier Fails: A Category-Level Audit of Reward Signals in RLVR [0.0]
  Reinforcement learning with rewards (RLVR) and standard benchmark evaluation both rely on an automatic verifier that turns a free text answer into a binary reward.<n>Prior work reports that one evaluation harness accepts only about 94% of its own ground truth answers, blaming parsing.<n>We supply the decomposition. We apply metamorphic testing to the verifier rather than the model, generating certified equivalent answer variants.<n>We measure rejection per answer category across four widely used verifiers over 307,420 verdicts.
  arXiv  Detail & Related papers  (2026-09-01T14:57:59Z)
• MemToC: Benchmarking Memory-Tool Conflict Resolution in Large Language Models [49.47300047353726]
  We introduce MemToC, a benchmark for post-tool-return arbitration with executable tools.<n>MemToC comprises 6,504 evaluation episodes constructed from 542 quality-controlled factual questions.<n>Across five open-weight 7-9B models, tool returns strongly dominate elicited closed-book answers.
  arXiv  Detail & Related papers  (2026-08-26T18:22:03Z)
• When Many Answers Are Valid, Voting Fails: Symbolic Verification for Best-of-K Causal Reasoning in LLMs [42.32694162421078]
  We introduce CALVER, a training-free symbolic verifier that scores structured traces against Pearl's causal criteria.<n>On CLEAR find-one-valid queries that admit multiple graph-valid answers, CALVER reaches 42.1% where plurality, a reward model, an LLM judge, and model confidence remain near 30%.
  arXiv  Detail & Related papers  (2026-08-04T11:45:46Z)
• Share the Judge, Learn the Deferral: Where Specialization Helps LLM Evaluation [0.1269104766024433]
  We study 99,952 public, rubric-conditioned examples.<n>Supplying the correct rubric improves locked-test accuracy by 2.11 points over a response-only control.<n>On RewardBench 2, learned correctness heads route examples through a 0.6B-4B-8B cascade without changing any reward score.
  arXiv  Detail & Related papers  (2026-07-30T10:29:22Z)
• Visual Credit Audit for Multimodal Spatial Reasoning [70.16915309526443]
  Visual Credit Audit separates two Estimands: whether the benchmark image gives the model's declared decision more support than text-only and blank controls, and whether the model responds to relation-specific visual evidence.<n>Applying labels yields dependence-credited correctness (D-CC)<n>Across four open MLLMs and two spatial benchmarks, 12.73-26.25% of decisions are correct yet uncredited.
  arXiv  Detail & Related papers  (2026-07-29T15:55:31Z)
• Looping Is Not Reliability: State-Bound Evidence and Typed Revision Contracts for Agentic Code Repair [36.56438114281786]
  We study the gap between finding a correct patch and retaining, verifying, and submitting it.<n>We derive an evidence-bound typed loop contract and instantiate its mechanically enforceable subset in a reference implementation.
  arXiv  Detail & Related papers  (2026-07-27T16:05:23Z)
• Step-wise Rubric Rewards for LLM Reasoning [72.17879367869503]
  Reinforcement Learning with Verifiable Rewards (RLVR) is widely used to improve reasoning in large language models.<n>We find 18.2% of steps in correct responses are wrong yet positively rewarded.<n>Across six mathematical reasoning benchmarks, SRaR improves average accuracy over RaR by 3.57 points.
  arXiv  Detail & Related papers  (2026-05-17T07:08:14Z)
• Are Benchmark Tests Strong Enough? Mutation-Guided Diagnosis and Augmentation of Regression Suites [49.16055123488827]
  Insufficiently strong test suites can admit plausible yet semantically incorrect patches, inflating reported success rates.<n>We introduce STING, a framework for targeted test augmentation that uses semantically altered program variants as diagnostic stressors.<n>STING produces 1,014 validated tests spanning 211 instances and increases patch-region line and branch coverage by 10.8% and 9.5%.
  arXiv  Detail & Related papers  (2026-04-02T01:13:40Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.