OpenPM: Auditable Point-in-Time Evaluation for LLM Portfolio-Management Agents

• URL: http://arxiv.org/abs/2608.09988v1
• Date: Thu, 06 Aug 2026 16:04:44 GMT
• Title: OpenPM: Auditable Point-in-Time Evaluation for LLM Portfolio-Management Agents
• Abstract summary: We present OpenPM, an auditable point-in-time evaluation framework for portfolio-management agents.<n>In OpenPM, an agent manages a $1M long-only book over the S&P 500 universe using market data at five-minute intervals.<n>Each run produces audit artifacts, including a contamination certificate, a cost-sensitivity curve, and a constraint-adherence report.
• Score: 39.82139666997448
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large language models are increasingly used to read markets, assess risk, and allocate capital. However, reported results for LLM trading agents can be inflated by look-ahead leakage, optimistic execution, and risk mandates that are described but not enforced. We present OpenPM, an auditable point-in-time evaluation framework for LLM portfolio-management agents. In OpenPM, an agent manages a \$1M long-only book over the S\&P 500 universe using market data at five-minute intervals. Every record visible to the agent must be available at the decision time. Natural-language risk mandates are converted into typed constraints and enforced on the executed portfolio. Each run produces audit artifacts, including a contamination certificate, a cost-sensitivity curve, and a constraint-adherence report. We also build a reference agent named the tiered allocator, where typed analysts score candidates, a constructor LLM proposes weights, and a deterministic critic guarantees feasibility. We isolate constructor behavior by capturing analyst evidence once and replaying it across constructor models. In our short-window case study, stronger constructors show modest and model-dependent gains over equal weighting on the same pool, but analyst quality matters more than constructor choice, and turnover is the main cost driver. All returns are upper bounds on a single frozen window without market impact, not validated alpha.

Related papers

• CLQT: A Closed-Loop, Cost-Aware, Strategy-Consistent Benchmark for Diagnostic Evaluation of LLM Portfolio-Management Agents [0.9924185669370708]
  We introduce CLQT, which reframes closed-loop trading evaluation as diagnosis rather than ranking.<n> CLQT is a fully closed-loop, cost-aware, strategy-consistent, temporally-gated environment.
  arXiv  Detail & Related papers  (2026-06-29T04:26:30Z)
• Steering LLM Viewpoints through Fabricated Evidence Injection [61.22850755490993]
  We introduce a two-phase attack framework that first repackages misleading statements with fabricated rationales, then instruct target LLMs to incorporate these viewpoints when responding to relevant queries.<n>Experiments on BBQ, ToxiGen, and our specialized dataset reveal that commercial LLMs without external safety classifiers remain highly vulnerable.<n>Building on this, we explore multiple defense strategies, among which a tailored safety policy enablesgpt-oss-safeguard to achieve 81% detection rate.
  arXiv  Detail & Related papers  (2026-06-04T14:48:23Z)
• From Knowing to Doing: A Memory-Controlled Benchmark for LLM Trading Agents on Stock Markets [34.699961175708815]
  KTD-Fin is an end-to-end stock-market trading benchmark.<n>It separates historical market memory from investment decision-making.<n>It incorporates a Barra-style performance attribution framework.
  arXiv  Detail & Related papers  (2026-05-27T11:57:10Z)
• The Alpha Illusion: Reported Alpha from LLM Trading Agents Should Not Be Treated as Deployment Evidence [23.772113719172896]
  Report headline Sharpe ratios from end-to-end LLM trading agents should not be treated as deployment evidence.<n>Report robustness is not tradable probability, narrative reasoning is not numerical execution, and model priors may become undisclosed implicit factor exposures.
  arXiv  Detail & Related papers  (2026-05-16T09:14:35Z)
• ValueBlindBench: Agreement-Gated Stress Testing of LLM-Judged Investment Rationales Before Returns Are Observable [1.8689252029357564]
  This paper introduces ValueBlindBench, a preregistered agreement-gated stress-test protocol.<n>ValueBlindBench decides when LLM-judged investment-rationale claims are publishable, qualified, or invalid.
  arXiv  Detail & Related papers  (2026-04-28T05:04:20Z)
• LLM Readiness Harness: Evaluation, Observability, and CI Gates for LLM/RAG Applications [51.56484100374058]
  We present a readiness harness for LLM and RAG applications that turns evaluation into a deployment decision workflow.<n>The system combines automated benchmarks, OpenTelemetry observability, and CI quality gates under a minimal API contract.<n>We evaluate the harness on ticket-routing and BEIR tasks with full Azure matrix coverage.
  arXiv  Detail & Related papers  (2026-03-28T18:03:32Z)
• Reasoning with Confidence: Efficient Verification of LLM Reasoning Steps via Uncertainty Heads [104.9566359759396]
  We propose a lightweight alternative for step-level reasoning verification based on data-driven uncertainty scores.<n>Our findings suggest that the internal states of LLMs encode their uncertainty and can serve as reliable signals for reasoning verification.
  arXiv  Detail & Related papers  (2025-11-09T03:38:29Z)
• Standard Benchmarks Fail - Auditing LLM Agents in Finance Must Prioritize Risk [31.43947127076459]
  Standard benchmarks fixate on how well large language model (LLM) agents perform in finance, yet say little about whether they are safe to deploy.<n>We argue that accuracy metrics and return-based scores provide an illusion of reliability, overlooking vulnerabilities such as hallucinated facts, stale data, and adversarial prompt manipulation.
  arXiv  Detail & Related papers  (2025-02-21T12:56:15Z)
• Empowering Many, Biasing a Few: Generalist Credit Scoring through Large Language Models [53.620827459684094]
  Large Language Models (LLMs) have great potential for credit scoring tasks, with strong generalization ability across multiple tasks. We propose the first open-source comprehensive framework for exploring LLMs for credit scoring. We then propose the first Credit and Risk Assessment Large Language Model (CALM) by instruction tuning, tailored to the nuanced demands of various financial risk assessment tasks.
  arXiv  Detail & Related papers  (2023-10-01T03:50:34Z)
• Do-Not-Answer: A Dataset for Evaluating Safeguards in LLMs [59.596335292426105]
  This paper collects the first open-source dataset to evaluate safeguards in large language models. We train several BERT-like classifiers to achieve results comparable with GPT-4 on automatic safety evaluation.
  arXiv  Detail & Related papers  (2023-08-25T14:02:12Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.