CLQT: A Closed-Loop, Cost-Aware, Strategy-Consistent Benchmark for Diagnostic Evaluation of LLM Portfolio-Management Agents
- URL: http://arxiv.org/abs/2606.29771v1
- Date: Mon, 29 Jun 2026 04:26:30 GMT
- Title: CLQT: A Closed-Loop, Cost-Aware, Strategy-Consistent Benchmark for Diagnostic Evaluation of LLM Portfolio-Management Agents
- Abstract summary: We introduce CLQT, which reframes closed-loop trading evaluation as diagnosis rather than ranking.<n> CLQT is a fully closed-loop, cost-aware, strategy-consistent, temporally-gated environment.
- Score: 0.9924185669370708
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: LLM agents are increasingly cast as autonomous portfolio managers, and benchmarks have moved from financial question-answering to sequential trading. Yet most still rank agents by returns over a fixed window -- a weak proxy, since a period's return is dominated by the market path and apparent alpha can dissolve once look-ahead leakage is controlled. Such a ranking certifies neither sound reasoning, nor a consistent strategy, nor a durable edge. We introduce CLQT, which reframes closed-loop trading evaluation as diagnosis rather than ranking: an instrument that localizes where and why an agent's process succeeds or fails. CLQT is a fully closed-loop, cost-aware, strategy-consistent, temporally-gated environment whose agents run a five-stage cycle: gather, synthesize, allocate, execute, reflect. Each round emits a complete DecisionRound sealed into a recompute-verifiable hash chain, so every metric is reconstructable from the trail. Six pillars form the substrate: a hard TimeGate, institutional transaction- and financing-cost modeling, strategy-consistency scoring, three-tier memory, a Model-Context-Protocol tool layer, and mandate-aware synthesis. The same agent runs as a constrained committee of specialized roles or a single full-autonomy orchestrator, making process scaffolding an experimental variable. From the audit trail we compute a five-axis capability scorecard (APM-CS: Coherence, Acuity, Composure, Discipline, Reliability), with Coherence judged partly by a held-out, out-of-cohort LLM to curb self-preference bias. We validate it on a contamination-controlled multi-model backtest with an ablation grid and a live broker track on unseen, post-cutoff data, against a repeated-run noise floor. CLQT separates outcome from capability, yielding not a model ranking but a durable, extensible map of agent competencies and limitations.
Related papers
- From Sequence to Structure: Relational Uncertainty Propagation for LLM Agents [75.69180947909148]
We propose a trajectory-level UQ framework for large language model (LLM) agents.<n>RuPA represents an execution history as a directed trajectory graph in which reasoning states, tool interactions, and environment feedback are nodes connected by temporal and semantic dependency edges.<n>We evaluate RUPA on representative agent benchmarks, including $$-2, Terminal-Bench-2, and GAIA, using 6 open-source LLMs spanning multiple model families.
arXiv Detail & Related papers (2026-08-17T01:40:14Z) - AgentOPSD: Recursive Self-Distillation for Agentic Reinforcement Learning [58.76655851910778]
AgentOPSD is a critic-free, recursive method for turn-level credit assignment in agentic reinforcement learning.<n>We evaluate AgentOPSD on ALFWorld, WebShop, and Search-QA using Qwen2.5 models at two scales (3B and 7B)
arXiv Detail & Related papers (2026-08-06T13:00:59Z) - ARCO: Adaptive Rubric with Co-Evolution for Multi-Step LLM-Based Agents [12.091072999307622]
ARCO is an adaptive rubric framework for reinforcement learning.<n>It improves the best EM in every setting over strong outcome-, rubric-, and process-reward baselines.<n>Its rubrics are step-specific, robust to design choices, and useful for diagnosing agent behavior.
arXiv Detail & Related papers (2026-06-19T09:38:02Z) - Certified Policy Optimisation for Nested Causal Bandits via PAC-Bayes Risk [0.0]
We formalise the problem class as Nested Contextual Causal Bandits (NCCBs)<n>Our main theoretical result is a causal PAC-Bayesian excess-risk bound that certifies any candidate deployment policy.
arXiv Detail & Related papers (2026-05-28T11:39:51Z) - TERMS-Bench: Diagnosing LLM Negotiation Agents Beyond Deal Rate [34.67393151003599]
Negotiation is a canonical testbed for agentic language models.<n>Unlike math or code, it has no intrinsic verifier.<n>We introduce Terms-Bench, a Bayesian-game framework that makes the environment itself the verifier.
arXiv Detail & Related papers (2026-05-13T06:22:50Z) - ComplexMCP: Evaluation of LLM Agents in Dynamic, Interdependent, and Large-Scale Tool Sandbox [61.862814740220806]
$textbfComplexMCP$ is a benchmark designed to evaluate agents in rigorous conditions.<n>Built on the Model Context Protocol (MCP), $textbfComplexMCP$ provides over 300 meticulously tested tools.
arXiv Detail & Related papers (2026-05-11T16:20:51Z) - AgenticAITA: A Proof-Of-Concept About Deliberative Multi-Agent Reasoning for Autonomous Trading Systems [0.0]
AGENTICAITA is an agentic AI framework that replaces the traditional signal then execute paradigm with a fully autonomous loop.<n>The framework demonstrates operational correctness of the deliberative pipeline, achieving 157 zero-intervention invocations across 76 assets.
arXiv Detail & Related papers (2026-05-01T16:25:43Z) - Generate, Filter, Control, Replay: A Comprehensive Survey of Rollout Strategies for LLM Reinforcement Learning [79.88942231770629]
Reinforcement learning (RL) has become a central post-training tool for improving the reasoning abilities of large language models (LLMs)<n>We formalize rollout pipelines with unified notation and introduce Generate-Filter-Control-Replay (GFCR)<n>We synthesize methods spanning RL with verifiable rewards, process supervision, judge-based gating, guided and tree/segment rollouts, adaptive compute allocation, early-exit and partial rollouts, throughput optimization, and replay/recomposition for self-improvement.
arXiv Detail & Related papers (2026-04-08T00:53:29Z) - LLM Readiness Harness: Evaluation, Observability, and CI Gates for LLM/RAG Applications [51.56484100374058]
We present a readiness harness for LLM and RAG applications that turns evaluation into a deployment decision workflow.<n>The system combines automated benchmarks, OpenTelemetry observability, and CI quality gates under a minimal API contract.<n>We evaluate the harness on ticket-routing and BEIR tasks with full Azure matrix coverage.
arXiv Detail & Related papers (2026-03-28T18:03:32Z) - CARE What Fails: Contrastive Anchored-REflection for Verifiable Multimodal [84.71254539482369]
Group-relative reinforcement learning with verifiable rewards (RLVR) often wastes the most informative data it already has the failures.<n>We present CARE, a failure-centric post-training framework for multimodal reasoning that turns errors into supervision.<n> CARE improves accuracy and training smoothness while explicitly increasing the share of learning signal that comes from failures.
arXiv Detail & Related papers (2025-12-22T16:34:21Z) - Practical RAG Evaluation: A Rarity-Aware Set-Based Metric and Cost-Latency-Quality Trade-offs [0.0]
This paper addresses the guessing game in building production RAG.<n>There is no standardized, reproducible way to build and audit golden sets.<n>Rath-gs (MIT) is a lean golden-set pipeline with Plackett-Luce listwise refinement.
arXiv Detail & Related papers (2025-11-12T18:49:21Z) - MALT: Improving Reasoning with Multi-Agent LLM Training [67.76186488361685]
MALT (Multi-Agent LLM Training) is a novel post-training strategy that divides the reasoning process into generation, verification, and refinement steps.<n>On MATH, GSM8K, and CSQA, MALT surpasses the same baseline LLM with a relative improvement of 15.66%, 7.42%, and 9.40% respectively.
arXiv Detail & Related papers (2024-12-02T19:30:36Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.