Why Large Language Models Fail at Tabular Prediction

• URL: http://arxiv.org/abs/2608.02412v1
• Date: Mon, 03 Aug 2026 15:52:51 GMT
• Title: Why Large Language Models Fail at Tabular Prediction
• Abstract summary: Large language models (LLMs) have become the default tool for a remarkable range of tasks.<n>But they have had conspicuously little success at one of the most common machine learning workloads.
• Score: 1.1908340728628395
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Large language models (LLMs) have become the default tool for a remarkable range of tasks, yet they have had conspicuously little success at one of the most common machine learning workloads: predictive analytics over tabular data. This gap is the founding premise of the fast-growing field of tabular foundation models, but the question of why generic LLMs fail has remained open. We study a frontier LLM in its purest inference regime - a single generation pass over a prompt containing the full training and test data, with no tools, no agentic scaffolding, and no fine-tuning - and systematically evaluate five hypotheses for the failure: (a) an inability to handle noisy or non-linearly-separable data; (b) the linearised CSV format obscuring column structure; (c) the tokenisation of numeric values; (d) the number of test points classified per query; and (e) the dimensionality of the input. Controlled experiments falsify (a)-(d). Dimensionality, in contrast, is decisive: sweeping random linear projections of thirty-one benchmark datasets, the LLM is the only method among nine whose accuracy decreases as dimensionality grows, while every classical baseline stays flat or improves. A behavioural comparison against 252 configured classical models finds that in two dimensions the LLM predicts like a local, distance-based method (up to 91.6% grid agreement), but in higher dimensions no classical model - even when augmented with tuned, dimension-dependent noise - reproduces its predictions. We do not claim to have identified the internal mechanism; our results show, more modestly, that the LLM's capability dissolves with dimension in a way no noise-corrupted classical learner mimics - which explains why LLMs, so capable elsewhere, keep losing to fifty-year-old baselines on tables, while leaving the mechanism of the prediction as an open question.

Related papers

• From Residuals to Reasons: LLM-Guided Mechanism Inference from Tabular Data [10.795877029195845]
  A persistent challenge in machine learning for scientific applications is jointly achieving prediction and understanding.<n>We introduce Multi-Agent Residual In-Context Learning (MARICL)<n>MARICL improves consistently over its base model on all datasets.
  arXiv  Detail & Related papers  (2026-05-21T15:34:08Z)
• FORESTLLM: Large Language Models Make Random Forest Great on Few-shot Tabular Learning [20.27406245916013]
  We propose a novel framework that unifies the structural inductive biases of decision forests with the semantic reasoning capabilities of large language models (LLMs)<n>Our method is two-fold. First, we introduce a semantic splitting criterion in which the LLM evaluates candidate partitions based on their coherence over both labeled and unlabeled data, enabling the induction of more robust and generalizable tree structures under few-shot supervision.<n>Second, we propose a one-time in-context inference mechanism for leaf node stabilization, where the LLM distills the decision path and its supporting examples into a concise, deterministic prediction, replacing noisy empirical estimates with semantically informed outputs
  arXiv  Detail & Related papers  (2026-01-16T14:08:51Z)
• An Empirical Investigation of Robustness in Large Language Models under Tabular Distortions [8.023379679609151]
  Large language models (LLMs) fail when tabular data in an otherwise canonical representation is subjected to semantic and structural distortions.<n>Only when provided with an explicit prior, via a system prompt, do models partially adjust their reasoning strategies and correct some distortions, though not consistently or completely.
  arXiv  Detail & Related papers  (2026-01-08T15:10:32Z)
• Predicting Emergent Capabilities by Finetuning [98.9684114851891]
  We find that finetuning language models can shift the point in scaling at which emergence occurs towards less capable models. We validate this approach using four standard NLP benchmarks. We find that, in some cases, we can accurately predict whether models trained with up to 4x more compute have emerged.
  arXiv  Detail & Related papers  (2024-11-25T01:48:09Z)
• Can adversarial attacks by large language models be attributed? [1.2289361708127877]
  We show that certain classes of Large Language Models (LLMs) are not identifiable from their outputs alone.<n>We quantify the explosion in the number of plausible model origins for a given output in recent years.
  arXiv  Detail & Related papers  (2024-11-12T18:28:57Z)
• Large Language Models Must Be Taught to Know What They Don't Know [97.90008709512921]
  We show that fine-tuning on a small dataset of correct and incorrect answers can create an uncertainty estimate with good generalization and small computational overhead.<n>We also investigate the mechanisms that enable reliable uncertainty estimation, finding that many models can be used as general-purpose uncertainty estimators.
  arXiv  Detail & Related papers  (2024-06-12T16:41:31Z)
• Cycles of Thought: Measuring LLM Confidence through Stable Explanations [53.15438489398938]
  Large language models (LLMs) can reach and even surpass human-level accuracy on a variety of benchmarks, but their overconfidence in incorrect responses is still a well-documented failure mode. We propose a framework for measuring an LLM's uncertainty with respect to the distribution of generated explanations for an answer.
  arXiv  Detail & Related papers  (2024-06-05T16:35:30Z)
• Characterizing Truthfulness in Large Language Model Generations with Local Intrinsic Dimension [63.330262740414646]
  We study how to characterize and predict the truthfulness of texts generated from large language models (LLMs) We suggest investigating internal activations and quantifying LLM's truthfulness using the local intrinsic dimension (LID) of model activations.
  arXiv  Detail & Related papers  (2024-02-28T04:56:21Z)
• Imputation-Free Learning from Incomplete Observations [73.15386629370111]
  We introduce the importance of guided gradient descent (IGSGD) method to train inference from inputs containing missing values without imputation. We employ reinforcement learning (RL) to adjust the gradients used to train the models via back-propagation. Our imputation-free predictions outperform the traditional two-step imputation-based predictions using state-of-the-art imputation methods.
  arXiv  Detail & Related papers  (2021-07-05T12:44:39Z)
• Meta-Learned Confidence for Few-shot Learning [60.6086305523402]
  A popular transductive inference technique for few-shot metric-based approaches, is to update the prototype of each class with the mean of the most confident query examples. We propose to meta-learn the confidence for each query sample, to assign optimal weights to unlabeled queries. We validate our few-shot learning model with meta-learned confidence on four benchmark datasets.
  arXiv  Detail & Related papers  (2020-02-27T10:22:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.