StakeBench: Evaluating Language Understanding Grounded in Market Commitment

• URL: http://arxiv.org/abs/2605.26074v1
• Date: Mon, 25 May 2026 17:38:30 GMT
• Title: StakeBench: Evaluating Language Understanding Grounded in Market Commitment
• Abstract summary: StakeBench is an evaluation framework for language understanding grounded in market commitment.<n> StakeBench links 560,876 comments from 2,261 resolved markets to verified position, action, and market-odds records.<n>Four diagnostic tasks test whether models detect market commitment, identify the revealed side, anticipate future action, and perform collective odds projection.
• Score: 5.902169076249955
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Existing financial NLP benchmarks often rely on labels supplied by outside observers, measuring how language is perceived rather than what speakers have committed to in the market. We introduce StakeBench, an evaluation framework for language understanding grounded in market commitment. StakeBench links 560,876 comments from 2,261 resolved markets to verified position, action, and market-odds records across Polymarket and Manifold. Supervision is derived from observable market behavior. Position sides, post-comment trading actions, and market-odds trajectories replace human annotation. Four diagnostic tasks test whether models detect market commitment, identify the revealed side, anticipate future action, and perform collective odds projection. Three commitment-aware metrics measure alignment with revealed preferences rather than perceived sentiment. Validity audits and explicit interpretation boundaries help distinguish observable commitment signals from latent belief and causal market-odds impact. Across 15 LLMs and 18 topics and platform settings, models partially recover position-side signals, with Directed Accuracy from 0.506 to 0.599, but show structural failures on later tasks. Ten of the fifteen models collapse to one or two action labels in future action anticipation, and no model consistently improves on the naive odds-direction baseline in collective odds projection. Model scale is not correlated with performance, finance-domain tuning does not improve revealed-side identification, and platform incentives strongly shape higher-order results. StakeBench is packaged with evaluation code and dataset under CC-BY 4.0.

Related papers

• Repurposing Deep Limit Order Book Forecasting for Scenario-Conditioned Market Impact Modeling [49.590388401787834]
  We introduce a model-agnostic framework that compares a trained forecaster's predictive distributions before and after injecting mechanically valid counterfactual messages.<n>A Transformer-based forecaster recovered scenario rankings with a Spearman correlation of 0.99 and 97.2% directional agreement with realized historical outcomes.
  arXiv  Detail & Related papers  (2026-09-15T10:06:03Z)
• NextFund: A Unified Performance Tracking Platform for Agentic Portfolio Management [0.9906305486517417]
  We introduce NextFund, an evaluation platform that makes financial-agent behavior observable under live market conditions.<n>The platform couples time-consistent market access, coordinated multi-agent analysis, and persistent logging of the full decision path from observation to trade.<n>We present NextFund on Hong Kong, U.S., and China A-share equities, illustrating how inspectable decision histories enable fairer benchmarking and more actionable diagnosis.
  arXiv  Detail & Related papers  (2026-07-13T06:22:21Z)
• Martingale Doppelgänger-Eval: An Identification Framework for Auditing Candlestick Understanding in Vision-Language Models [7.600410227599106]
  Martingale Doppelgnger-Eval is a public shadow-market benchmark for auditing whether vision-language models (VLMs) use candlestick evidence rather than extrapolate past trends.<n>A structural behavioral model identifies null-market bias, trend sensitivity, evidence sensitivity, prompt/renderer fragility, and evidence faithfulness.
  arXiv  Detail & Related papers  (2026-06-16T02:10:37Z)
• CIAware-Bench: Benchmarking Control Intervention Awareness Across Frontier LLMs [100.38986535324284]
  We introduce textbfCIAware-Bench, a benchmark for measuring textbfcontrol textbfintervention (CI) awareness across frontier models.<n>CIAware-Bench tests whether models can distinguish their own trajectories from those modified by a control intervention.
  arXiv  Detail & Related papers  (2026-06-09T16:24:16Z)
• Foresight Arena: An On-Chain Benchmark for Evaluating AI Forecasting Agents [0.0]
  We introduce Foresight Arena, the first permissionless, on-chain benchmark for evaluating AI forecasting agents on real-world prediction markets.<n>Performance is measured by the Brier Score and a novel Alpha Score -- proper scoring rules that incentivize honest probability reporting.<n>We show that detecting a true edge of $* = 0.02$ at 80% power requires approximately 350 resolved binary predictions.
  arXiv  Detail & Related papers  (2026-05-01T05:33:10Z)
• Price as Focal Point: Prediction Markets,Conditional Reflexivity, and the Politics of Common Knowledge [0.0]
  We show that the social force of a market signal depends less on its size than on its persistence, the breadth of responding trader types, and cross-platform consensus.<n>We introduce a Signal Credibility Index (SCI) as a microstructure-grounded criterion for predicting when price moves acquire behavioral traction.
  arXiv  Detail & Related papers  (2026-04-27T08:02:34Z)
• Objective Mispricing Detection for Shortlisting Undervalued Football Players via Market Dynamics and News Signals [1.6312989763677892]
  We present a practical, reproducible framework for identifying undervalued football players grounded in objective mispricing.<n>We estimate an expected market value from structured data (historical market dynamics, biographical and contract features, transfer history) and compare it to the observed valuation to define mispricing.<n>We then assess whether news-derived Natural Language Processing (NLP) features complement market signals for shortlisting undervalued players.
  arXiv  Detail & Related papers  (2026-03-18T13:03:23Z)
• TraderBench: How Robust Are AI Agents in Adversarial Capital Markets? [8.661756660747042]
  TraderBench is a benchmark for evaluating AI agents in finance.<n>It combines expert-verified static tasks (knowledge retrieval, analytical reasoning) with adversarial trading simulations.<n>Two novel tracks: crypto trading with four progressive market-manipulation transforms, and options derivatives scoring across P&L accuracy, Greeks, and risk management.
  arXiv  Detail & Related papers  (2026-02-27T20:06:28Z)
• CLUE: Non-parametric Verification from Experience via Hidden-State Clustering [64.50919789875233]
  We show that correctness of a solution is encoded as a geometrically separable signature within the trajectory of hidden activations.<n>ClUE consistently outperforms LLM-as-a-judge baselines and matches or exceeds modern confidence-based methods in reranking candidates.
  arXiv  Detail & Related papers  (2025-10-02T02:14:33Z)
• Are You Getting What You Pay For? Auditing Model Substitution in LLM APIs [71.7892165868749]
  Commercial Large Language Model (LLM) APIs create a fundamental trust problem.<n>Users pay for specific models but have no guarantee that providers deliver them faithfully.<n>We formalize this model substitution problem and evaluate detection methods under realistic adversarial conditions.<n>We propose and evaluate the use of Trusted Execution Environments (TEEs) as one practical and robust solution.
  arXiv  Detail & Related papers  (2025-04-07T03:57:41Z)
• Eliminating Position Bias of Language Models: A Mechanistic Approach [119.34143323054143]
  Position bias has proven to be a prevalent issue of modern language models (LMs)<n>Our mechanistic analysis attributes the position bias to two components employed in nearly all state-of-the-art LMs: causal attention and relative positional encodings.<n>By eliminating position bias, models achieve better performance and reliability in downstream tasks, including LM-as-a-judge, retrieval-augmented QA, molecule generation, and math reasoning.
  arXiv  Detail & Related papers  (2024-07-01T09:06:57Z)
• Delving into Probabilistic Uncertainty for Unsupervised Domain Adaptive Person Re-Identification [54.174146346387204]
  We propose an approach named probabilistic uncertainty guided progressive label refinery (P$2$LR) for domain adaptive person re-identification. A quantitative criterion is established to measure the uncertainty of pseudo labels and facilitate the network training. Our method outperforms the baseline by 6.5% mAP on the Duke2Market task, while surpassing the state-of-the-art method by 2.5% mAP on the Market2MSMT task.
  arXiv  Detail & Related papers  (2021-12-28T07:40:12Z)
• A Sentiment Analysis Approach to the Prediction of Market Volatility [62.997667081978825]
  We have explored the relationship between sentiment extracted from financial news and tweets and FTSE100 movements. The sentiment captured from news headlines could be used as a signal to predict market returns; the same does not apply for volatility. We developed an accurate classifier for the prediction of market volatility in response to the arrival of new information.
  arXiv  Detail & Related papers  (2020-12-10T01:15:48Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.