Benchmarking Language Models for Statistical Problem Formulation

• URL: http://arxiv.org/abs/2609.01982v2
• Date: Sat, 05 Sep 2026 01:28:39 GMT
• Title: Benchmarking Language Models for Statistical Problem Formulation
• Abstract summary: StatFormBench is a benchmark built from five cross-domain statistics textbooks and a data science case library.<n>It contains 1,013 samples spanning 20 coarse-grained and 85 fine-grained statistical problem categories.<n>No model performs consistently best across the two subtasks, while enhanced prompting strategies yield only limited or inconsistent gains.
• Score: 15.739640186203474
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Large language models (LLMs) are increasingly used as assistants for statistical and data science work, yet existing evaluations largely assume the analysis target is already specified. In practice, users arrive with informal goals and heterogeneous data, leaving the model to decide what statistical task is implied and which data are relevant. We first formalize this upstream step as Statistical Problem Formulation and decompose it into two subtasks: (1) Statistical Problem Classification and (2) Variable Identification & Role Assignment. We then introduce StatFormBench, a benchmark built from five cross-domain statistics textbooks and a data science case library, covering diverse problem types, data representations, and scenario styles. It contains 1,013 samples spanning 20 coarse-grained and 85 fine-grained statistical problem categories. Across 14 open- and closed-source LLMs, the best zero-shot models reach only 72.0 fine-grained classification accuracy and 63.2 variable set overlap. No model performs consistently best across the two subtasks, while enhanced prompting strategies yield only limited or inconsistent gains. We release the benchmark data on Hugging Face at https://huggingface.co/datasets/THU-CongLab/StatFormBench and the evaluation code on GitHub at https://github.com/THU-CongLab/StatFormBench.

Related papers

• StatABench: Dataset and Framework for Evaluating Statistical Analysis Capabilities of LLMs [26.57002473586959]
  We introduce StatABench, a benchmark designed to assess large language models' statistical analysis capabilities.<n>StatABench comprises two complementary components: Stat-Closed, containing 404 questions across 18 statistical topics in multiple formats, and Stat-Open, featuring 30 complex open-ended modeling tasks adapted from professional competitions.<n>We evaluate diverse LLMs using the LangChain MCP framework and multiple data science agents, and assess Stat-Open solutions via a validated LLM-as-Judge protocol.
  arXiv  Detail & Related papers  (2026-06-22T07:57:53Z)
• StatEval: A Comprehensive Benchmark for Large Language Models in Statistics [18.64342811887586]
  StatEval is the first comprehensive benchmark dedicated to statistics, spanning both breadth and depth across difficulty levels.<n>It consists of 13,817 foundational problems covering undergraduate and graduate curricula, together with 2374 research-level proof tasks extracted from leading journals.<n>We propose a robust evaluation framework tailored to both computational and proof-based tasks, enabling fine-grained assessment of reasoning ability.
  arXiv  Detail & Related papers  (2025-10-10T16:28:43Z)
• LimiX: Unleashing Structured-Data Modeling Capability for Generalist Intelligence [61.46575527504109]
  LimiX-16M and LimiX-2M treat structured data as a joint distribution over variables and missingness.<n>We evaluate LimiX models across 11 large structured-data benchmarks with broad regimes of sample size, feature dimensionality, class number, categorical-to-numerical feature ratio, missingness, and sample-to-feature ratios.
  arXiv  Detail & Related papers  (2025-09-03T17:39:08Z)
• Training on the Benchmark Is Not All You Need [52.01920740114261]
  We propose a simple and effective data leakage detection method based on the contents of multiple-choice options.<n>Our method is able to work under gray-box conditions without access to model training data or weights.<n>We evaluate the degree of data leakage of 35 mainstream open-source LLMs on four benchmark datasets.
  arXiv  Detail & Related papers  (2024-09-03T11:09:44Z)
• Variance Alignment Score: A Simple But Tough-to-Beat Data Selection Method for Multimodal Contrastive Learning [17.40655778450583]
  We propose a principled metric named Variance Alignment Score (VAS), which has the form $langle Sigma_texttest, Sigma_irangle$. We show that applying VAS and CLIP scores together can outperform baselines by a margin of $1.3%$ on 38 evaluation sets for noisy dataset DataComp and $2.5%$ on VTAB for high-quality dataset CC12M.
  arXiv  Detail & Related papers  (2024-02-03T06:29:04Z)
• A Statistical View of Column Subset Selection [47.65143789184956]
  We consider the problem of selecting a small subset of representative variables from a large dataset. We show how to efficiently (1) perform CSS using only summary statistics from the original dataset; (2) perform CSS in the presence of missing and/or censored data; and (3) select the subset size for CSS in a hypothesis testing framework.
  arXiv  Detail & Related papers  (2023-07-24T15:42:33Z)
• Subsampling Suffices for Adaptive Data Analysis [8.231050911072755]
  Most classical techniques assume that the dataset is independent of the analyst's query and break down in the common setting where a dataset is reused for multiple, adaptively chosen, queries. We identify a remarkably simple set of assumptions under which the queries will continue to be representative even when chosen adaptively. The simplicity of this subsampling-based framework allows it to model a variety of real-world scenarios not covered by prior work.
  arXiv  Detail & Related papers  (2023-02-17T02:47:54Z)
• Canary in a Coalmine: Better Membership Inference with Ensembled Adversarial Queries [53.222218035435006]
  We use adversarial tools to optimize for queries that are discriminative and diverse. Our improvements achieve significantly more accurate membership inference than existing methods.
  arXiv  Detail & Related papers  (2022-10-19T17:46:50Z)
• RIFLE: Imputation and Robust Inference from Low Order Marginals [10.082738539201804]
  We develop a statistical inference framework for regression and classification in the presence of missing data without imputation. Our framework, RIFLE, estimates low-order moments of the underlying data distribution with corresponding confidence intervals to learn a distributionally robust model. Our experiments demonstrate that RIFLE outperforms other benchmark algorithms when the percentage of missing values is high and/or when the number of data points is relatively small.
  arXiv  Detail & Related papers  (2021-09-01T23:17:30Z)
• Examining and Combating Spurious Features under Distribution Shift [94.31956965507085]
  We define and analyze robust and spurious representations using the information-theoretic concept of minimal sufficient statistics. We prove that even when there is only bias of the input distribution, models can still pick up spurious features from their training data. Inspired by our analysis, we demonstrate that group DRO can fail when groups do not directly account for various spurious correlations.
  arXiv  Detail & Related papers  (2021-06-14T05:39:09Z)
• Overcoming Statistical Shortcuts for Open-ended Visual Counting [54.858754825838865]
  We aim to develop models that learn a proper mechanism of counting regardless of the output label. First, we propose the Modifying Count Distribution protocol, which penalizes models that over-rely on statistical shortcuts. Secondly, we introduce the Spatial Counting Network (SCN), which is dedicated to visual analysis and counting based on natural language questions.
  arXiv  Detail & Related papers  (2020-06-17T18:02:01Z)
• Meta-Learned Confidence for Few-shot Learning [60.6086305523402]
  A popular transductive inference technique for few-shot metric-based approaches, is to update the prototype of each class with the mean of the most confident query examples. We propose to meta-learn the confidence for each query sample, to assign optimal weights to unlabeled queries. We validate our few-shot learning model with meta-learned confidence on four benchmark datasets.
  arXiv  Detail & Related papers  (2020-02-27T10:22:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.