Beyond Perplexity: A Behavioral Evaluation Framework for Deployment-Memory Claims in LLM Test-Time Training

• URL: http://arxiv.org/abs/2607.00368v1
• Date: Wed, 01 Jul 2026 03:07:17 GMT
• Title: Beyond Perplexity: A Behavioral Evaluation Framework for Deployment-Memory Claims in LLM Test-Time Training
• Abstract summary: Large language model test-time training (TTT) is often evaluated through local proxy metrics.<n>We introduce a behavioral evaluation framework that calibrates TTT memory claims to the evidence that supports them.
• Score: 30.22002300048916
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Large language model test-time training (TTT) is often evaluated through local proxy metrics: models are updated on recent tokens, retrieved context, target-domain data, or verifiable task attempts, and then judged by perplexity, future-token loss, long-context performance, or reward. These metrics are well matched to claims about stream adaptation, domain adaptation, context compression, and reward-backed test-time improvement. They are weaker evidence, however, for a capability that TTT results are increasingly used to motivate: deployed assistant memory, personalization, or sparse post-deployment learning, which instead requires behavioral evidence such as later recall, paraphrase robustness, retention, locality, conflict handling, and use in downstream actions after the original support context is removed. We introduce a behavioral evaluation framework that calibrates TTT memory claims to the evidence that supports them. It has two components: a claim-calibrated evidence ladder that separates stream/domain adaptation, bridge internalization, and deployment-time behavioral learning; and an evaluation protocol with matched explicit-memory baselines and mutually exclusive failure categories. We validate the framework by auditing recent TTT and memory-adjacent work and by instantiating it as a controlled diagnostic in which, in a sparse nonce-fact setting, one-step LoRA updates lower support and answer loss across three Qwen3 model scales while generated free-form recall stays at zero, exposing a measurable gap between proxy improvement and deployment behavior. The framework gives authors and evaluators a concrete standard for aligning TTT memory claims with the evidence actually reported.

Related papers

• RepoReasoner: Evaluating Repository-Level Code Reasoning Ability of Long-Context Language Models [54.53236295237077]
  We introduce RepoReasoner, a benchmark for evaluating repository-level code reasoning.<n> Output Prediction measures fine-grained, stateful execution reasoning across files.<n>Call Chain Prediction evaluates high-level architectural dependency understanding under noisy context.
  arXiv  Detail & Related papers  (2026-07-28T17:12:41Z)
• Hidden Forgetting in Continual Multimodal Learning: When Accuracy Survives but Grounding Fails [6.708653593551749]
  We study whether a continually adapted MLLM can preserve not only what it answers, but how it uses visual, textual, OCR, chart, and document evidence.<n>We identify emphhidden evidence-use forgetting, where answer accuracy is retained while the model silently shifts toward different or less grounded evidence channels.<n>We propose textscRCL, a replay-free reliance-constrained continual learning framework.
  arXiv  Detail & Related papers  (2026-07-02T10:50:37Z)
• EVAF: A Test-Retest Protocol for Selective Parametric Consolidation [0.0]
  We introduce EVAF, an Echo-Valence Attractor Field mechanism for gated LoRA consolidation.<n>Across GPT-2 and TinyLlama, EVAF preferentially consolidates high-valence, high-surprise experiences.<n>Test-retest measurements show stronger post-interference behavioral persistence than frozen, retrieval-only, and ungated continual-update baselines.
  arXiv  Detail & Related papers  (2026-06-29T07:53:50Z)
• The Evaluation Game: Beyond Static LLM Benchmarking [2.168753020470345]
  We introduce a game-theoretic framework in which the interaction between an evaluator and a trainer is formalized as a two-player game.<n>We provide empirical evidence supporting locality-dependence of the model.<n>Our framework recasts the central object of adversarial evaluation: a benchmark is not a static set of prompts but an orbit under the evaluator's group action.
  arXiv  Detail & Related papers  (2026-05-19T05:22:21Z)
• Back to Source: Open-Set Continual Test-Time Adaptation via Domain Compensation [18.70132691409063]
  Test-Time Adaptation aims to mitigate distributional shifts between training and test domains during inference time.<n>The coupling of domain and semantic shifts often collapses the feature space, severely degrading both classification and out-of-distribution detection.<n>We propose DOmain COmpensation (DOCO), a lightweight and effective framework that robustly performs domain adaptation and OOD detection in a synergistic, closed loop.
  arXiv  Detail & Related papers  (2026-04-23T15:29:29Z)
• FRESCO: Benchmarking and Optimizing Re-rankers for Evolving Semantic Conflict in Retrieval-Augmented Generation [73.22935457705057]
  We introduce FRESCO, a benchmark for evaluating re-rankers in temporally dynamic contexts.<n>By pairing recency-seeking queries with historical Wikipedia revisions, FRESCO tests whether re-rankers can prioritize factually recent evidence while maintaining semantic relevance.<n>Our evaluation reveals a consistent failure mode across existing re-rankers: a strong bias toward older, semantically rich documents, even when they are factually obsolete.
  arXiv  Detail & Related papers  (2026-04-14T17:04:25Z)
• Pretrain-then-Adapt: Uncertainty-Aware Test-Time Adaptation for Text-based Person Search [26.52140212357349]
  Text-based person search faces inherent limitations due to data scarcity, driven by stringent privacy constraints and the high cost of manual annotation.<n>Existing methods usually rely on a Pretrain-then-Finetune paradigm, where models are first pretrained on synthetic person-caption data.<n>We propose a new Pretrain-then-Adapt paradigm that eliminates reliance on extensive target-domain supervision through an offline test-time adaptation manner.
  arXiv  Detail & Related papers  (2026-04-07T07:48:15Z)
• The Unlearning Mirage: A Dynamic Framework for Evaluating LLM Unlearning [54.67958855362658]
  We propose a dynamic framework that stress tests unlearning using complex structured queries.<n>Our approach first elicits knowledge from the target model (pre-unlearning) and constructs targeted probes, ranging from simple queries to multi-hop chains.<n>Our framework enables practical and scalable evaluation of unlearning methods without the need for manual construction of forget test sets.
  arXiv  Detail & Related papers  (2026-03-11T19:51:33Z)
• Refinement Provenance Inference: Detecting LLM-Refined Training Prompts from Model Behavior [58.751981587234916]
  This paper formalizes the Refinement Provenance Inference (RPI) audit task as Refinement Provenance Inference (RPI)<n>We propose RePro, a logit-based framework that fuses teacher-forced likelihood features with logit-ranking signals.<n>During training, RePro learns a transferable representation via shadow fine-tuning, and uses a lightweight linear head to infer provenance on unseen victims without training-data access.
  arXiv  Detail & Related papers  (2026-01-05T10:16:41Z)
• Memory in Large Language Models: Mechanisms, Evaluation and Evolution [8.158439933515131]
  We propose a four-part taxonomy (parametric, contextual, external, procedural/episodic) and a memory quadruple (location, persistence, write/access path, controllability)<n>For updating and forgetting, we present DMM Gov: coordinating DAPT/TAPT, PEFT, model editing (ROME, MEND, MEMIT, SERAC), and RAG to form an auditable loop.<n>This yields a reproducible, comparable, and governable coordinate system for research and deployment.
  arXiv  Detail & Related papers  (2025-09-23T10:06:58Z)
• BoostAdapter: Improving Vision-Language Test-Time Adaptation via Regional Bootstrapping [64.8477128397529]
  We propose a training-required and training-free test-time adaptation framework. We maintain a light-weight key-value memory for feature retrieval from instance-agnostic historical samples and instance-aware boosting samples. We theoretically justify the rationality behind our method and empirically verify its effectiveness on both the out-of-distribution and the cross-domain datasets.
  arXiv  Detail & Related papers  (2024-10-20T15:58:43Z)
• Mitigating Catastrophic Forgetting in Task-Incremental Continual Learning with Adaptive Classification Criterion [50.03041373044267]
  We propose a Supervised Contrastive learning framework with adaptive classification criterion for Continual Learning. Experiments show that CFL achieves state-of-the-art performance and has a stronger ability to overcome compared with the classification baselines.
  arXiv  Detail & Related papers  (2023-05-20T19:22:40Z)
• A Closer Look at Temporal Sentence Grounding in Videos: Datasets and Metrics [70.45937234489044]
  We re- organize two widely-used TSGV datasets (Charades-STA and ActivityNet Captions) to make it different from the training split. We introduce a new evaluation metric "dR@$n$,IoU@$m$" to calibrate the basic IoU scores. All the results demonstrate that the re-organized datasets and new metric can better monitor the progress in TSGV.
  arXiv  Detail & Related papers  (2021-01-22T09:59:30Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.