ATANT v1.1: Positioning Continuity Evaluation Against Memory, Long-Context, and Agentic-Memory Benchmarks

• URL: http://arxiv.org/abs/2604.10981v2
• Date: Sun, 19 Apr 2026 17:25:53 GMT
• Title: ATANT v1.1: Positioning Continuity Evaluation Against Memory, Long-Context, and Agentic-Memory Benchmarks
• Abstract summary: ATANT v1.0 defined continuity as a system property with 7 required properties.<n>We show by structural analysis that none of these benchmarks measures continuity as defined in v1.0.<n>We publish our 8.8% LOCOMO score alongside our 96% ATANT cumulative-scale score as a calibration pair.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: ATANT v1.0 (arXiv:2604.06710) defined continuity as a system property with 7 required properties and introduced a 10-checkpoint, LLM-free evaluation methodology validated on a 250-story corpus. Since publication, a recurring reviewer and practitioner question has concerned not the framework itself but its relationship to a wider set of memory evaluations: LOCOMO, LongMemEval, BEAM, MemoryBench, Zep's evaluation suite, Letta/MemGPT's evaluations, and RULER. This companion paper, v1.1, does not modify the v1.0 standard. It closes a related-work gap that v1.0 left brief under page limits. We show by structural analysis that none of these benchmarks measures continuity as defined in v1.0: of the 7 required properties, the median existing eval covers 1 property, the mean covers 0.43 when partial credit is scored at 0.5, and no eval covers more than 2. We provide a cell-by-cell property-coverage matrix, identify methodological defects specific to each benchmark (including an empty-gold scoring bug in the LOCOMO reference implementation that renders 23% of its corpus unscorable by construction), and publish our reference implementation's LOCOMO score (8.8%) alongside the structural reason that number is uninformative about continuity. We publish our 8.8% LOCOMO score alongside our 96% ATANT cumulative-scale score as a calibration pair: the 87-point divergence is evidence that the two benchmarks measure different properties, not that one system is an order of magnitude better than another. The position v1.1 takes is not adversarial: each benchmark measures a real capability. The claim is that none of them can adjudicate continuity, and conflating them with continuity evaluation has led the field to under-invest in the properties v1.0 names.

Related papers

• Hindsight Memory-PRM: Supervising Memory Management with Auditable Hindsight Credit [48.811961095386]
  Hindsight Memory-PRM exploits this audit trail twice.<n> offline to train an operation-conditioned memory-utility critic.<n>Online, where retrievals, citations, and one controlled deletion-and-reanswer per probe settle an intervention-calibrated entry-level presence credit.
  arXiv  Detail & Related papers  (2026-08-30T06:53:13Z)
• What Aggregate Scores Miss: Measuring Item-Level Regressions in Commercial LLM API Migrations [1.6787220460106658]
  Software systems that depend on large language model APIs must migrate to successor versions when vendors deprecate older models.<n>Migration decisions typically rely on aggregate benchmark scores, which compress heterogeneous item-level behaviour into a single net figure.
  arXiv  Detail & Related papers  (2026-08-18T12:44:54Z)
• UpliftBench: Revealing Outcome-Regime and Objective Mismatch in Uplift Evaluation [0.0]
  UpliftBench evaluates 12 uplift estimators under an outer-test-isolated, multi-objective protocol.<n>We show the disagreement is substantially about metrics, not models.
  arXiv  Detail & Related papers  (2026-08-02T00:51:44Z)
• Context-Aware Distillation and Ablation for Text2DSL [78.29352305480285]
  We replace prompt-only synthetic generation with context-aware distillation.<n>A teacher large language model operates under an explicitly defined structured context.<n>This scales the verified PolkitBench corpus from 4,204 to 10,073 pairs.
  arXiv  Detail & Related papers  (2026-06-21T16:27:24Z)
• Let the Results Speak: A Replication-First Paradigm for LLM Behavioral Benchmarking [22.825786049667602]
  We propose a replication-first paradigm to anchor validity to a single human-rater consensus.<n>We certify the instrument via four properties -- reliability across K runs, cross-instrument replication across architecturally distinct judges, historical-footprint calibration via judges from earlier training cohorts, and pre-registered prediction.<n>We test it on emotional accompaniment by letting the self-evolve data-driven across: the dimensions are not pre-stipulated and the procedure stabilizes to a 9-dimension set.
  arXiv  Detail & Related papers  (2026-05-27T03:41:11Z)
• DiffScore: Text Evaluation Beyond Autoregressive Likelihood [79.18228156703695]
  We introduce DiffScore, an evaluation framework built on Masked Large Diffusion Language Models.<n>By measuring text recoverability across continuous masking rates, DiffScore eliminates positional bias.<n>Experiments across ten benchmarks show that DiffScore consistently outperforms autoregressive baselines in both zero-shot and fine-tuned settings.
  arXiv  Detail & Related papers  (2026-05-12T06:27:36Z)
• How Far Is Document Parsing from Solved? PureDocBench: A Source-TraceableBenchmark across Clean, Degraded, and Real-World Settings [56.70440596502351]
  The past year has seen over 20 open document parsing models, yet benchmarks almost exclusively on OmniDocBench.<n>We present PureDocBench, a benchmark thatrenders document images from HTML/CSS, covering 10 domains, 66 pages, and 1,475 pages.
  arXiv  Detail & Related papers  (2026-05-08T09:30:31Z)
• Reducing Maintenance Burden in Behaviour-Driven Development: A Paraphrase-Robust Duplicate-Step Detector with a 1.1M-Step Open Benchmark [1.9537983097153042]
  Behaviour-Driven Development suites accumulate step-duplicate duplication with documented maintenance cost.<n>We release the largest cross-organisational BDD step corpus to date.
  arXiv  Detail & Related papers  (2026-04-22T11:44:05Z)
• ImplicitMemBench: Measuring Unconscious Behavioral Adaptation in Large Language Models [60.14219417402433]
  Existing memory benchmarks for LLM agents evaluate explicit recall of facts, yet overlook implicit memory where experience becomes automated behavior without conscious retrieval.<n>We introduce ImplicitMemBench, the first systematic benchmark evaluating implicit memory through three constructs drawn from standard cognitive-science accounts of non-declarative memory.<n>Our 300-item suite employs a unified Learning/Priming-Interfere-Test protocol with first-attempt scoring.
  arXiv  Detail & Related papers  (2026-04-09T10:26:32Z)
• ATANT: An Evaluation Framework for AI Continuity [0.0]
  We present ATANT, an open evaluation framework for measuring continuity in AI systems.<n>We present a narrative test corpus of 250 stories comprising 1,835 verification questions across 6 life domains.<n>ATANT is system-agnostic, model-independent, and designed as a sequenced methodology for building and validating continuity systems.
  arXiv  Detail & Related papers  (2026-04-08T06:04:51Z)
• Are Benchmark Tests Strong Enough? Mutation-Guided Diagnosis and Augmentation of Regression Suites [49.16055123488827]
  Insufficiently strong test suites can admit plausible yet semantically incorrect patches, inflating reported success rates.<n>We introduce STING, a framework for targeted test augmentation that uses semantically altered program variants as diagnostic stressors.<n>STING produces 1,014 validated tests spanning 211 instances and increases patch-region line and branch coverage by 10.8% and 9.5%.
  arXiv  Detail & Related papers  (2026-04-02T01:13:40Z)
• BenchScope: How Many Independent Signals Does Your Benchmark Provide? [0.0]
  We introduce Effective Dimensionality (ED), a fast, population-conditional upper-bound diagnostic of measurement breadth.<n>ED is applied at per-instance granularity to 22 benchmarks across 8 domains and more than 8,400 model evaluations.<n>We show that relative ED rankings are stable under matched-dimension controls and that ED can flag redundant suite components.
  arXiv  Detail & Related papers  (2026-03-31T07:28:41Z)
• Graph-Native Cognitive Memory for AI Agents: Formal Belief Revision Semantics for Versioned Memory Architectures [0.6091702876917279]
  Kumiho is a graph-native cognitive memory architecture grounded in formal belief revision semantics.<n>The architecture implements a dual-store model (Redis working memory, Neo4j long-term graph) with hybrid fulltext and vector retrieval.
  arXiv  Detail & Related papers  (2026-03-18T00:59:49Z)
• vla-eval: A Unified Evaluation Harness for Vision-Language-Action Models [58.633451339058986]
  VLA models are typically evaluated using per benchmark scripts maintained independently by each model repository.<n>We present vla eval, an open source evaluation harness that decouples model inference from benchmark execution.<n>A complete evaluation requires only two commands: vla eval serve and vla eval run.
  arXiv  Detail & Related papers  (2026-03-14T14:38:53Z)
• QEDBENCH: Quantifying the Alignment Gap in Automated Evaluation of University-Level Mathematical Proofs [29.26861081722613]
  We demonstrate that standard "LLM-as-a-Judge" protocols suffer from a systematic Alignment Gap when applied to upper-undergraduate to early graduate level mathematics.<n>We introduce QEDBench, the first large-scale dual-rubric alignment benchmark to measure alignment with human experts on university-level math.<n>We reveal that certain frontier evaluators like Claude Opus 4.5, DeepSeek-V3, Qwen 2.5 Max, and Llama 4 Maverick exhibit significant positive bias.
  arXiv  Detail & Related papers  (2026-02-24T07:23:28Z)
• Assessing LLM Reliability on Temporally Recent Open-Domain Questions [15.456770184839726]
  Large Language Models (LLMs) are increasingly deployed for open-domain question answering.<n>We investigate how four open-source LLMs respond to 15,000 recent Reddit questions.<n>All models achieve over 99% cosine similarity with references despite less than 8% BLEU-1 overlap.
  arXiv  Detail & Related papers  (2026-01-17T21:33:27Z)
• TrustJudge: Inconsistencies of LLM-as-a-Judge and How to Alleviate Them [58.04324690859212]
  Large Language Models (LLMs) as automated evaluators (LLM-as-a-judge) has revealed critical inconsistencies in current evaluation frameworks.<n>We identify two fundamental types of inconsistencies: Score-Comparison Inconsistency and Pairwise Transitivity Inconsistency.<n>We propose TrustJudge, a probabilistic framework that addresses these limitations through two key innovations.
  arXiv  Detail & Related papers  (2025-09-25T13:04:29Z)
• Generating Benchmarks for Factuality Evaluation of Language Models [61.69950787311278]
  We propose FACTOR: Factual Assessment via Corpus TransfORmation, a scalable approach for evaluating LM factuality. FACTOR automatically transforms a factual corpus of interest into a benchmark evaluating an LM's propensity to generate true facts from the corpus vs. similar but incorrect statements. We show that: (i) our benchmark scores increase with model size and improve when the LM is augmented with retrieval; (ii) benchmark score and perplexity do not always agree on model ranking; (iii) when perplexity and benchmark score disagree, the latter better reflects factuality in open-ended generation.
  arXiv  Detail & Related papers  (2023-07-13T17:14:38Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.