Same Day, Same Story; One Day Ahead, a Different Signal: The Dual Validity of Financial Sentiment

• URL: http://arxiv.org/abs/2609.11144v1
• Date: Thu, 10 Sep 2026 06:37:33 GMT
• Title: Same Day, Same Story; One Day Ahead, a Different Signal: The Dual Validity of Financial Sentiment
• Abstract summary: We find that the relationship between construct and predictive validity depends on the sampling convention and score representation.<n>In a conversation that is 17.6% spam, message volume predicts neither market damage nor settlement size.
• Score: 5.121117558582558
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Financial NLP has a standard workflow: validate a sentiment tool against human labels, then trust it to extract market signal. This assumes the two evaluations measure the same thing. We test that assumption in a setting where both can be measured at once: a corpus of securities class actions (2002-2025) linking 70,500 X messages to abnormal stock returns, with a single-annotator human labelled gold sample. Running five instruments (VADER, Loughran-McDonald, FinBERT, Twitter-RoBERTa, and an LLM annotator) through one identical pipeline, we find that the relationship between construct and predictive validity depends on the sampling convention and score representation. Under conventional method-specific sampling, human agreement aligns more closely with graded same-day associations than with one-day leads. On a fixed-n panel, however, agreement has similar graded rank correlations at both horizons, while the coarse ordering remains weak. Benchmark agreement therefore establishes semantic validity but does not by itself determine predictive rankings. In a conversation that is 17.6% spam, message volume predicts neither market damage nor settlement size.

Related papers

• Difference-in-Differences on a Censored Rating Scale Can Manufacture an Effect: Evidence from a Pre-Registered LLM-Judge Audit [15.81584544074437]
  Audits of LLM judges certify a bias by contrasting matched conditions.<n>We show that this endpoint is not identified on the scale that reports it.<n>We derive the mechanism in closed form and show that its contribution is measurable from an audit's own ratings.
  arXiv  Detail & Related papers  (2026-08-27T16:10:16Z)
• One Human, $N$ Agents: Audit-Budget Allocation for LLM Agent Fleets under Miscalibrated, Correlated Confidence [41.99844472131922]
  A single human must audit $N$ LLM agents under a budget of $B ll N$ audits per round.<n>We model this as budgeted noisy inspection over a two-level Gaussian copula and locate the miscalibration threshold $*$ past which confidence-ranked auditing is emphworse than random.
  arXiv  Detail & Related papers  (2026-07-30T14:52:36Z)
• AgentFairBench: Do LLM Agents Discriminate When They Act? [2.3004655342211078]
  AgentFairBench is a cheap, reproducible, multi-domain benchmark for demographic disparity in the actions of LLM agents.<n>It spans three regulator-anchored domains: hiring, lending, and medical triage.<n>A NumPy-only harness computes counterfactual flip rate, mean absolute score difference (MASD), action-rate disparity, and tool-invocation disparity.
  arXiv  Detail & Related papers  (2026-06-15T13:50:26Z)
• StakeBench: Evaluating Language Understanding Grounded in Market Commitment [5.902169076249955]
  StakeBench is an evaluation framework for language understanding grounded in market commitment.<n> StakeBench links 560,876 comments from 2,261 resolved markets to verified position, action, and market-odds records.<n>Four diagnostic tasks test whether models detect market commitment, identify the revealed side, anticipate future action, and perform collective odds projection.
  arXiv  Detail & Related papers  (2026-05-25T17:38:30Z)
• Agreement Metrics for LLM-as-Judge Evaluation: What to Report and Why [34.429649156970015]
  A survey of 24 recent LLM-as-judge papers finds metric choice entangled with the judgment scale, tie handling, invalid outputs, and abstention handling.<n>Most of the reported numbers are redundant: Pearson's $r$, Spearman's $$, Kendall's $_b$, the phi coefficient $$, and the Matthews Correlation Coefficient all reduce to a single number.
  arXiv  Detail & Related papers  (2026-05-25T07:31:44Z)
• Measurement Risk in Supervised Financial NLP: Rubric and Metric Sensitivity on JF-ICR [1.7107991816118835]
  We study the measurement risk on Japanese Financial Implicit-Commitment Recognition.<n>We find that rubric wording materially changes model-assigned labels.<n>Not every metric remains informative under the JF-ICR class distribution.
  arXiv  Detail & Related papers  (2026-04-30T03:39:14Z)
• Correction and Corruption: A Two-Rate View of Error Flow in LLM Protocols [51.56484100374058]
  We propose a paired-outcome measurement interface for auditing a single protocol step on exact-match tasks.<n>For each instance, the interface records a baseline correctness bit and a post-step correctness bit.<n>These rates predict accuracy changes and define a reusable empirical interface testable across seeds, mixtures, and pipelines.
  arXiv  Detail & Related papers  (2026-04-20T13:25:40Z)
• Consistency Checks for Language Model Forecasters [54.62507816753479]
  We measure the performance of forecasters in terms of the consistency of their predictions on different logically-related questions.<n>We build an automated evaluation system that generates a set of base questions, instantiates consistency checks from these questions, elicits predictions of the forecaster, and measures the consistency of the predictions.
  arXiv  Detail & Related papers  (2024-12-24T16:51:35Z)
• Beyond correlation: The Impact of Human Uncertainty in Measuring the Effectiveness of Automatic Evaluation and LLM-as-a-Judge [51.93909886542317]
  We show how *relying on a single aggregate correlation score* can obscure fundamental differences between human labels and those from automatic evaluation.<n>We propose stratifying data by human label uncertainty to provide a more robust analysis of automatic evaluation performance.
  arXiv  Detail & Related papers  (2024-10-03T03:08:29Z)
• Uncertainty in Language Models: Assessment through Rank-Calibration [65.10149293133846]
  Language Models (LMs) have shown promising performance in natural language generation. It is crucial to correctly quantify their uncertainty in responding to given inputs. We develop a novel and practical framework, termed $Rank$-$Calibration$, to assess uncertainty and confidence measures for LMs.
  arXiv  Detail & Related papers  (2024-04-04T02:31:05Z)
• Measuring Fairness of Text Classifiers via Prediction Sensitivity [63.56554964580627]
  ACCUMULATED PREDICTION SENSITIVITY measures fairness in machine learning models based on the model's prediction sensitivity to perturbations in input features. We show that the metric can be theoretically linked with a specific notion of group fairness (statistical parity) and individual fairness.
  arXiv  Detail & Related papers  (2022-03-16T15:00:33Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.