ATANT: An Evaluation Framework for AI Continuity

• URL: http://arxiv.org/abs/2604.06710v1
• Date: Wed, 08 Apr 2026 06:04:51 GMT
• Title: ATANT: An Evaluation Framework for AI Continuity
• Abstract summary: We present ATANT, an open evaluation framework for measuring continuity in AI systems.<n>We present a narrative test corpus of 250 stories comprising 1,835 verification questions across 6 life domains.<n>ATANT is system-agnostic, model-independent, and designed as a sequenced methodology for building and validating continuity systems.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We present ATANT (Automated Test for Acceptance of Narrative Truth), an open evaluation framework for measuring continuity in AI systems: the ability to persist, update, disambiguate, and reconstruct meaningful context across time. While the AI industry has produced memory components (RAG pipelines, vector databases, long context windows, profile layers), no published framework formally defines or measures whether these components produce genuine continuity. We define continuity as a system property with 7 required properties, introduce a 10-checkpoint evaluation methodology that operates without an LLM in the evaluation loop, and present a narrative test corpus of 250 stories comprising 1,835 verification questions across 6 life domains. We evaluate a reference implementation across 5 test suite iterations, progressing from 58% (legacy architecture) to 100% in isolated mode (250 stories) and 100% in 50-story cumulative mode, with 96% at 250-story cumulative scale. The cumulative result is the primary measure: when 250 distinct life narratives coexist in the same database, the system must retrieve the correct fact for the correct context without cross-contamination. ATANT is system-agnostic, model-independent, and designed as a sequenced methodology for building and validating continuity systems. The framework specification, example stories, and evaluation protocol are available at https://github.com/Kenotic-Labs/ATANT. The full 250-story corpus will be released incrementally.

Related papers

• EvolveScaler: Synthesizing Information-Evolution Contexts via Executable State Machines and Natural-Language Rendering [57.87517047585447]
  We introduce EvolveScaler, a code-driven framework that defines information evolution before rendering it as natural language.<n>We instantiate EvolveScaler with 117 task prototypes and 159 final-question operators across five difficulty levels spanning approximately 7 to 1,200 events per instance.<n>On the very_long tier, the strongest model reaches 59.3% avg@5, while six models score below 10%.
  arXiv  Detail & Related papers  (2026-09-08T08:40:05Z)
• A Standardized Framework for Machine Learning in Power System Protection [8.877321413840214]
  This paper proposes a standardization-oriented framework that treats evaluation design as part of the scientific contribution.<n>It defines seven required study dimensions: protection objective, physical scope, observability, timing and decision windows, targets and sample validity, validation protocol, and evaluation outputs.
  arXiv  Detail & Related papers  (2026-08-20T15:35:36Z)
• Operationalising Multi-Dimensional Evaluation for Conversational Agents: A Scalable, Governed Pipeline with Selective Re-evaluation and Model Benchmarking [0.0]
  We present GenAI Evaluation, a governed, configuration-driven pipeline for large-scale evaluation of retail conversational systems.<n>The framework evaluates helpfulness, truthfulness, clarity, tone alignment, and translation-specific dimensions.<n>The pipeline achieved a macro F1 score of 0.93 and 89% human-acceptability accuracy for translation.
  arXiv  Detail & Related papers  (2026-07-13T19:01:56Z)
• Context-Aware Distillation and Ablation for Text2DSL [78.29352305480285]
  We replace prompt-only synthetic generation with context-aware distillation.<n>A teacher large language model operates under an explicitly defined structured context.<n>This scales the verified PolkitBench corpus from 4,204 to 10,073 pairs.
  arXiv  Detail & Related papers  (2026-06-21T16:27:24Z)
• MMAE: A Massive Multitask Audio Editing Benchmark [66.74229858407413]
  MMAE is the first comprehensive evaluation testbed designed for general-purpose instruction-based audio editing.<n>It comprises 2,000 high-fidelity samples paired with a pioneering rubric-based evaluation framework.<n>Our evaluation reveals that current systems remain far from achieving reliable edits.
  arXiv  Detail & Related papers  (2026-06-05T12:52:41Z)
• ATANT v1.1: Positioning Continuity Evaluation Against Memory, Long-Context, and Agentic-Memory Benchmarks [0.0]
  ATANT v1.0 defined continuity as a system property with 7 required properties.<n>We show by structural analysis that none of these benchmarks measures continuity as defined in v1.0.<n>We publish our 8.8% LOCOMO score alongside our 96% ATANT cumulative-scale score as a calibration pair.
  arXiv  Detail & Related papers  (2026-04-13T04:40:37Z)
• Context Engineering: A Practitioner Methodology for Structured Human-AI Collaboration [0.0]
  This paper introduces Context Engineering, a structured methodology for assembling, declaring, and sequencing the complete informational payload that accompanies a prompt to an AI tool.<n>In an observational study of 200 documented interactions across four AI tools, incomplete context was associated with 72% of iteration cycles.<n> Structured context assembly was associated with a reduction from 3.8 to 2.0 average iteration cycles per task and an improvement in first-pass acceptance from 32% to 55%.
  arXiv  Detail & Related papers  (2026-04-05T20:30:44Z)
• Encyclo-K: Evaluating LLMs with Dynamically Composed Knowledge Statements [78.87065404966002]
  Existing benchmarks predominantly curate questions at the question level.<n>We propose Encyclo-K, a statement-based benchmark that rethinks benchmark construction from the ground up.
  arXiv  Detail & Related papers  (2025-12-31T13:55:54Z)
• DramaBench: A Six-Dimensional Evaluation Framework for Drama Script Continuation [3.496854427764583]
  Drama script continuation requires models to maintain character consistency, advance plot coherently, and preserve dramatic structurecapabilities.<n>We present DramaBench, the first large-scale benchmark for evaluating drama script continuation across six independent dimensions.
  arXiv  Detail & Related papers  (2025-12-22T04:03:01Z)
• What Matters in Evaluating Book-Length Stories? A Systematic Study of Long Story Evaluation [59.626962970198434]
  We introduce the first large-scale benchmark, LongStoryEval, comprising 600 newly published books with an average length of 121K tokens (maximum 397K)<n>By analyzing all user-mentioned aspects, we propose an evaluation criteria structure and conduct experiments to identify the most significant aspects.<n>For evaluation methods, we compare the effectiveness of three types: aggregation-based, incremental-updated, and summary-based evaluations.
  arXiv  Detail & Related papers  (2025-12-14T20:53:29Z)
• Towards Realistic Evaluation of Commit Message Generation by Matching Online and Offline Settings [77.20838441870151]
  We use an online metric - the number of edits users introduce before committing the generated messages to the VCS - to select metrics for offline experiments.<n>We collect a dataset with 57 pairs consisting of commit messages generated by GPT-4 and their counterparts edited by human experts.<n>Our results indicate that edit distance exhibits the highest correlation with the online metric, whereas commonly used similarity metrics such as BLEU and METEOR demonstrate low correlation.
  arXiv  Detail & Related papers  (2024-10-15T20:32:07Z)
• T5Score: Discriminative Fine-tuning of Generative Evaluation Metrics [94.69907794006826]
  We present a framework that combines the best of both worlds, using both supervised and unsupervised signals from whatever data we have available. We operationalize this idea by training T5Score, a metric that uses these training signals with mT5 as the backbone. T5Score achieves the best performance on all datasets against existing top-scoring metrics at the segment level.
  arXiv  Detail & Related papers  (2022-12-12T06:29:04Z)
• Segment-level Metric Learning for Few-shot Bioacoustic Event Detection [56.59107110017436]
  We propose a segment-level few-shot learning framework that utilizes both the positive and negative events during model optimization. Our system achieves an F-measure of 62.73 on the DCASE 2022 challenge task 5 (DCASE2022-T5) validation set, outperforming the performance of the baseline prototypical network 34.02 by a large margin.
  arXiv  Detail & Related papers  (2022-07-15T22:41:30Z)
• Natural Answer Generation: From Factoid Answer to Full-length Answer using Grammar Correction [39.40116590327074]
  This paper proposes a system that outputs a full-length answer given a question and the extracted factoid answer as the input. A transformer-based Grammar Error Correction model GECToR ( 2020), is used as a post-processing step for better fluency. We compare our system with (i) Modified Pointer Generator (SOTA) and (ii) Fine-tuned DialoGPT for factoid questions.
  arXiv  Detail & Related papers  (2021-12-07T17:39:21Z)
• $Q^{2}$: Evaluating Factual Consistency in Knowledge-Grounded Dialogues via Question Generation and Question Answering [38.951535576102906]
  We propose an automatic evaluation metric for factual consistency in knowledge-grounded dialogue models. Our metric makes use of co-reference resolution and natural language inference capabilities. We curate a novel dataset of state-of-the-art dialogue system outputs for the Wizard-of-Wikipedia dataset.
  arXiv  Detail & Related papers  (2021-04-16T16:21:16Z)
• Yseop at SemEval-2020 Task 5: Cascaded BERT Language Model for Counterfactual Statement Analysis [0.0]
  We use a BERT base model for the classification task and build a hybrid BERT Multi-Layer Perceptron system to handle the sequence identification task. Our experiments show that while introducing syntactic and semantic features does little in improving the system in the classification task, using these types of features as cascaded linear inputs to fine-tune the sequence-delimiting ability of the model ensures it outperforms other similar-purpose complex systems like BiLSTM-CRF in the second task.
  arXiv  Detail & Related papers  (2020-05-18T08:19:18Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.