Beyond Static Snapshots: A Grounded Evaluation Framework for Language Models at the Agentic Frontier

• URL: http://arxiv.org/abs/2604.17573v1
• Date: Sun, 19 Apr 2026 18:28:32 GMT
• Title: Beyond Static Snapshots: A Grounded Evaluation Framework for Language Models at the Agentic Frontier
• Abstract summary: We present ISOPro, a simulation-based fine-tuning and evaluation system.<n>It replaces the learned reward model with a deterministic ground-truth verifier.<n>It operates on LoRA adapter weights updatable on CPU, reducing the hardware barrier by an order of magnitude.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We argue that current evaluation frameworks for large language models (LLMs) suffer from four systematic failures that make them structurally inadequate for assessing deployed, agentic systems: distributional invalidity (evaluation inputs do not reflect real interaction distributions), temporal invalidity (evaluations are post-hoc rather than training-integrated), scope invalidity (evaluations measure single-turn outputs rather than long-horizon trajectories), and process invalidity (evaluations assess outputs rather than reasoning). These failures compound critically in RLHF, where reward models are evaluated under conditions that do not hold during RL training, making reward hacking a predictable consequence of evaluation design rather than a training pathology. We propose the Grounded Continuous Evaluation (GCE) framework and present ISOPro, a simulation-based fine-tuning and evaluation system. ISOPro replaces the learned reward model with a deterministic ground-truth verifier, eliminating reward hacking by construction in verifiable-reward domains, and operates on LoRA adapter weights updatable on CPU, reducing the hardware barrier by an order of magnitude. We validate ISOPro on a resource-constrained scheduling domain with six difficulty tiers, demonstrating capability emergence visible only through continuous evaluation, an implicit curriculum that forms without researcher curation, and a 3x accuracy improvement over zero-shot baselines, all on consumer hardware with 0.216% trainable parameters.

Related papers

• CAST: Critique-Aware Supervision for Training Reliable Long-Horizon Tool-Calling Agents [66.9575676266637]
  Large language model (LLM) agents are increasingly deployed in long-horizon, interactive, and stateful environments.<n>In these settings, a single wrong action, such as refunding the wrong purchase, can cause irreversible task failure and must be intercepted before execution.<n>We develop CAST, a critique-aware training framework that converts sparse task outcomes into action-level supervision for critique learning and policy optimization.
  arXiv  Detail & Related papers  (2026-08-31T01:59:51Z)
• Evaluating Agentic AI in the Wild: Failure Modes, Drift Patterns, and a Production Evaluation Framework [0.0]
  We present a taxonomy of seven failure modes unique to production agentic systems.<n>Standard metrics fail to detect four of the seven failure modes entirely.<n>We propose PAEF, a five-dimension evaluation framework with an open-source reference implementation.
  arXiv  Detail & Related papers  (2026-05-02T21:02:08Z)
• The Illusion of Certainty: Decoupling Capability and Calibration in On-Policy Distillation [67.26315138466312]
  We propose a calibration-aware OPD framework, CaOPD, that estimates empirical confidence from model rollouts.<n>Our findings highlight that capability distillation does not imply confidence, and that confidence should be treated as an essential objective in post-training.
  arXiv  Detail & Related papers  (2026-04-18T04:43:40Z)
• Can Vision Language Models Judge Action Quality? An Empirical Evaluation [5.688409551177917]
  Action Quality Assessment (AQA) has broad applications in physical therapy, sports coaching, and competitive judging.<n>Although Vision Language Models (VLMs) hold considerable promise for AQA, their actual performance in this domain remains largely uncharacterised.<n>We present a comprehensive evaluation of state-of-the-art VLMs across activity domains (e.g. fitness, figure skating, diving), tasks, representations, and prompting strategies.
  arXiv  Detail & Related papers  (2026-04-09T14:29:19Z)
• Correct Answers from Sound Reasoning: Verifiable Process Supervision for Language Models [94.68358825189738]
  We propose a post-training framework for verifiable domains that jointly optimize prediction accuracy and reasoning quality.<n>We evaluate VPS on chess, a controlled testbed where reasoning steps can be deterministically verified against engine signals.<n>VPS preserves accuracy while significantly improving reasoning quality, reducing win-rate error by up to 30% and restoring consistency to near saturation.
  arXiv  Detail & Related papers  (2026-04-03T15:19:46Z)
• Evaluating Uplift Modeling under Structural Biases: Insights into Metric Stability and Model Robustness [8.135022024189306]
  In personalized marketing, uplift models estimate incremental effects by modeling how customer behavior changes under alternative treatments.<n>Real-world data often exhibit biases, such as selection bias, spillover effects, and unobserved confounding.
  arXiv  Detail & Related papers  (2026-03-21T11:54:25Z)
• Evaluation-Aware Reinforcement Learning [10.594563233900004]
  Policy evaluation is often a prerequisite for deploying safety- and performance-critical systems.<n>We propose evaluation-aware reinforcement learning (EvA-RL), in which a policy is trained to maximize expected return.<n>We show that EvA-RL can substantially reduce evaluation error while maintaining competitive returns.
  arXiv  Detail & Related papers  (2025-09-23T18:17:21Z)
• Weak Supervision Performance Evaluation via Partial Identification [46.73061437177238]
  Programmatic Weak Supervision (PWS) enables supervised model training without direct access to ground truth labels. We present a novel method to address this challenge by framing model evaluation as a partial identification problem. Our approach derives reliable bounds on key metrics without requiring labeled data, overcoming core limitations in current weak supervision evaluation techniques.
  arXiv  Detail & Related papers  (2023-12-07T07:15:11Z)
• Exploring validation metrics for offline model-based optimisation with diffusion models [50.404829846182764]
  In model-based optimisation (MBO) we are interested in using machine learning to design candidates that maximise some measure of reward with respect to a black box function called the (ground truth) oracle. While an approximation to the ground oracle can be trained and used in place of it during model validation to measure the mean reward over generated candidates, the evaluation is approximate and vulnerable to adversarial examples. This is encapsulated under our proposed evaluation framework which is also designed to measure extrapolation.
  arXiv  Detail & Related papers  (2022-11-19T16:57:37Z)
• Uncertainty Weighted Actor-Critic for Offline Reinforcement Learning [63.53407136812255]
  Offline Reinforcement Learning promises to learn effective policies from previously-collected, static datasets without the need for exploration. Existing Q-learning and actor-critic based off-policy RL algorithms fail when bootstrapping from out-of-distribution (OOD) actions or states. We propose Uncertainty Weighted Actor-Critic (UWAC), an algorithm that detects OOD state-action pairs and down-weights their contribution in the training objectives accordingly.
  arXiv  Detail & Related papers  (2021-05-17T20:16:46Z)
• Model Rectification via Unknown Unknowns Extraction from Deployment Samples [8.0497115494227]
  We propose a general algorithmic framework that aims to perform a post-training model rectification at deployment time in a supervised way. RTSCV extracts unknown unknowns (u.u.s) We show that RTSCV consistently outperforms state-of-the-art approaches.
  arXiv  Detail & Related papers  (2021-02-08T11:46:19Z)
• Unsupervised Domain Adaptation for Speech Recognition via Uncertainty Driven Self-Training [55.824641135682725]
  Domain adaptation experiments using WSJ as a source domain and TED-LIUM 3 as well as SWITCHBOARD show that up to 80% of the performance of a system trained on ground-truth data can be recovered.
  arXiv  Detail & Related papers  (2020-11-26T18:51:26Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.