Market-Bench: Evaluating Large Language Models on Introductory Quantitative Trading and Market Dynamics

• URL: http://arxiv.org/abs/2512.12264v1
• Date: Sat, 13 Dec 2025 10:07:31 GMT
• Title: Market-Bench: Evaluating Large Language Models on Introductory Quantitative Trading and Market Dynamics
• Abstract summary: We introduce MARKET-BENCH, a benchmark that evaluates large language models (LLMs) on introductory quantitative trading tasks.<n>Each instance specifies one of three canonical strategies.<n>Models must produce code whose P&L, drawdown, and position paths match a verifiable reference implementation.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We introduce MARKET-BENCH, a benchmark that evaluates large language models (LLMs) on introductory quantitative trading tasks by asking them to construct executable backtesters from natural-language strategy descriptions and market assumptions. Each instance specifies one of three canonical strategies -- scheduled trading on Microsoft (NASDAQ: MSFT), pairs trading on Coca-Cola (NASDAQ: KO) and Pepsi (NASDAQ: PEP), or delta hedging on MSFT -- and models must produce code whose P\&L, drawdown, and position paths match a verifiable reference implementation. We assess twelve state-of-the-art models using a multi-round pass@k metric that separates structural reliability (whether the backtest runs) from numerical accuracy (mean absolute error of the backtest metrics). While most models reliably execute the simplest strategy (average pass@3 of 0.80), errors vary by orders of magnitude across models and tasks: Gemini 3 Pro and Claude 4.5 Sonnet combine strong reliability with low error on simpler strategies, GPT-5.1 Codex-Max achieves perfect pass@1 on the first two strategies and the lowest best-run error on the easiest task, and Qwen3 Max attains perfect pass@3 yet sometimes produces inaccurate P\&L paths. These results show that current LLMs can scaffold basic trading infrastructure but still struggle to reason robustly about prices, inventory, and risk; we release MARKET-BENCH and a public leaderboard at https://marketbench.ai.

Related papers

• AI Trading: Evaluating Large Language Models for Technical Market Analysis [0.0]
  Large Language Models (LLMs) have emerged as powerful tools for processing the heterogeneous information environments of modern financial markets.<n>This paper presents a systematic, comparative evaluation of five prominent LLMs: GPT-4 Turbo, Claude 3 Opus, Gemini 1.5 Pro, Llama 3 70B, and the domain-specialized FinGPT.<n>We conclude that while LLMs hold genuine promise within AI trading systems, robust deployment requires careful task decomposition, rigorous backtesting protocols, and domain-aware fine-tuning strategies.
  arXiv  Detail & Related papers  (2026-07-16T19:38:23Z)
• SciOrch: Learning to Orchestrate Expert LLMs for Solving Frontier Multimodal Scientific Reasoning Tasks [65.14594927081983]
  We present SciOrch, a framework that trains a lightweight 8B model to orchestrate frontier LLMs for scientific reasoning.<n>On a 240-question test set spanning SGI-Reasoning and Scientists' First exam, SciOrch reaches 56.66% average accuracy.<n>It also attains the best accuracy on both SGI and SFE with less than half the API cost of typical multi-agent methods.
  arXiv  Detail & Related papers  (2026-06-14T15:45:34Z)
• QuantCode-Bench: A Benchmark for Evaluating the Ability of Large Language Models to Generate Executable Algorithmic Trading Strategies [0.04660328753262074]
  We present QuantCode-Bench, a benchmark for the systematic evaluation of modern LLMs in generating strategies for the Backtrader framework.<n>We show that the main limitations of current models are not related to syntax, but rather to the correct operationalization of trading logic, proper API usage, and adherence to task semantics.
  arXiv  Detail & Related papers  (2026-04-16T15:31:42Z)
• SysTradeBench: An Iterative Build-Test-Patch Benchmark for Strategy-to-Code Trading Systems with Drift-Aware Diagnostics [25.854191624941677]
  We introduce SysTradeBench, an auditable benchmark for benchmarking strategy-to-code trading systems.<n>A sandboxed harness runs determinism and anti-leakage checks, detects rule drift, and returns evidence bundles to support constrained patches.<n>Top models achieve validity above 91.7 percent with strong aggregate scores, but evidence-driven iteration also induces code convergence by Iter2.
  arXiv  Detail & Related papers  (2026-04-06T16:16:24Z)
• TraderBench: How Robust Are AI Agents in Adversarial Capital Markets? [8.661756660747042]
  TraderBench is a benchmark for evaluating AI agents in finance.<n>It combines expert-verified static tasks (knowledge retrieval, analytical reasoning) with adversarial trading simulations.<n>Two novel tracks: crypto trading with four progressive market-manipulation transforms, and options derivatives scoring across P&L accuracy, Greeks, and risk management.
  arXiv  Detail & Related papers  (2026-02-27T20:06:28Z)
• One LLM to Train Them All: Multi-Task Learning Framework for Fact-Checking [7.856998585396422]
  Large language models (LLMs) are reshaping automated fact-checking (AFC) by enabling unified, end-to-end verification pipelines.<n>We propose textbfmulti-task learning (MTL) as a more efficient alternative that fine-tunes a single model to perform claim detection, evidence ranking, and stance detection jointly.
  arXiv  Detail & Related papers  (2026-01-16T13:44:25Z)
• Reliable Decision Support with LLMs: A Framework for Evaluating Consistency in Binary Text Classification Applications [0.7124971549479361]
  This study introduces a framework for evaluating consistency in large language model (LLM) binary text classification.<n>We determine sample size requirements, develop metrics for invalid responses, and evaluate intra- and inter-rater reliability.
  arXiv  Detail & Related papers  (2025-05-20T21:12:58Z)
• FinTSB: A Comprehensive and Practical Benchmark for Financial Time Series Forecasting [58.70072722290475]
  Financial time series (FinTS) record the behavior of human-brain-augmented decision-making.<n>FinTSB is a comprehensive and practical benchmark for financial time series forecasting.
  arXiv  Detail & Related papers  (2025-02-26T05:19:16Z)
• Benchmarking Post-Training Quantization in LLMs: Comprehensive Taxonomy, Unified Evaluation, and Comparative Analysis [89.60263788590893]
  Post-training Quantization (PTQ) technique has been extensively adopted for large language models (LLMs) compression.<n>Existing algorithms focus primarily on performance, overlooking the trade-off among model size, performance, and quantization bitwidth.<n>We provide a novel benchmark for LLMs PTQ in this paper.
  arXiv  Detail & Related papers  (2025-02-18T07:35:35Z)
• Do Large Language Model Benchmarks Test Reliability? [66.1783478365998]
  We investigate how well current benchmarks quantify model reliability.<n>Motivated by this gap in the evaluation of reliability, we propose the concept of so-called platinum benchmarks.<n>We evaluate a wide range of models on these platinum benchmarks and find that, indeed, frontier LLMs still exhibit failures on simple tasks.
  arXiv  Detail & Related papers  (2025-02-05T18:58:19Z)
• FCMR: Robust Evaluation of Financial Cross-Modal Multi-Hop Reasoning [5.65203350495478]
  We present Financial Cross-Modal Multi-Hop Reasoning (FCMR), a benchmark to analyze the reasoning capabilities of MLLMs.<n>FCMR is categorized into three difficulty levels-Easy, Medium, and Hard-facilitating a step-by-step evaluation.<n>Experiments on this new benchmark reveal that even state-of-the-art MLLMs struggle, with the best-performing model achieving only 30.4% accuracy on the most challenging tier.
  arXiv  Detail & Related papers  (2024-12-17T05:50:55Z)
• TAT-LLM: A Specialized Language Model for Discrete Reasoning over Tabular and Textual Data [73.29220562541204]
  We consider harnessing the amazing power of language models (LLMs) to solve our task. We develop a TAT-LLM language model by fine-tuning LLaMA 2 with the training data generated automatically from existing expert-annotated datasets.
  arXiv  Detail & Related papers  (2024-01-24T04:28:50Z)
• Prompting GPT-3 To Be Reliable [117.23966502293796]
  This work decomposes reliability into four facets: generalizability, fairness, calibration, and factuality. We find that GPT-3 outperforms smaller-scale supervised models by large margins on all these facets.
  arXiv  Detail & Related papers  (2022-10-17T14:52:39Z)
• Trader-Company Method: A Metaheuristic for Interpretable Stock Price Prediction [3.9189409002585562]
  There are several challenges in financial markets hindering practical applications of machine learning-based models. We propose the Trader-Company method, a novel evolutionary model that mimics the roles of a financial institute and traders. Our method predicts future stock returns by aggregating suggestions from multiple weak learners called Traders.
  arXiv  Detail & Related papers  (2020-12-18T13:19:27Z)
• AvgOut: A Simple Output-Probability Measure to Eliminate Dull Responses [97.50616524350123]
  We build dialogue models that are dynamically aware of what utterances or tokens are dull without any feature-engineering. The first model, MinAvgOut, directly maximizes the diversity score through the output distributions of each batch. The second model, Label Fine-Tuning (LFT), prepends to the source sequence a label continuously scaled by the diversity score to control the diversity level. The third model, RL, adopts Reinforcement Learning and treats the diversity score as a reward signal.
  arXiv  Detail & Related papers  (2020-01-15T18:32:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.