LLMs on Tabular Data with Limited Semantics: Evidence from Industrial Car Retrofit Prediction

• URL: http://arxiv.org/abs/2606.15314v1
• Date: Sat, 13 Jun 2026 14:13:41 GMT
• Title: LLMs on Tabular Data with Limited Semantics: Evidence from Industrial Car Retrofit Prediction
• Abstract summary: We study an industrial dataset linking a prototype-registration system with a retrofit-management system.<n>We compare strong tabular machine learning baselines with three LLM-based strategies on row-serialized inputs.<n>On the monthly benchmark, lag-based machine learning outperforms time-series foundation models.
• Score: 4.679788938455094
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Industrial retrofit planning depends on structured operational data rather than free text: planners must estimate whether a newly registered prototype will require a retrofit, which retrofit package it will need, and how long the work will take. We study an industrial dataset linking a prototype-registration system (284,271 vehicles) with a retrofit-management system (48,716 cleaned visits), and compare strong tabular machine learning baselines with three LLM-based strategies on row-serialized inputs: embedding features (Amazon Titan), direct prompted classification (Claude Sonnet 4), and an ML+LLM stacking approach. Across binary occurrence prediction, 15-way retrofit-type classification, per-visit duration regression, and an aggregated monthly benchmark, classical tree ensembles remain the strongest standalone models. However, the LLM results reveal a consistent pattern: embeddings remain useful on tables (binary AUC = 0.982), direct prompting collapses once semantic signal is stripped by hashing (binary AUC = 0.500; multiclass weighted F1 = 0.018), and hybrid stacking yields the best manually built multiclass model (weighted F1 = 0.626). On the monthly benchmark, lag-based machine learning outperforms time-series foundation models, though Chronos-small remains competitive in zero-shot forecasting. The results suggest that on privacy-constrained industrial tables, LLMs are more effective as complementary components than as replacements for strong tabular baselines.

Related papers

• MLLM-DataEngine: Closing the Loop of Multimodal Instruction Tuning Data Generation [51.15583711678273]
  We propose a novel closed-loop system that bridges data generation, model training, and evaluation.<n>Within each loop iteration, the MLLM-DataEngine first analyzes the weakness of the model based on the evaluation results.<n>We find MLLM-DataEngine could boost the MLLMs capability in a targeted and automatic manner without human participants.
  arXiv  Detail & Related papers  (2026-07-07T08:35:38Z)
• Is TabPFN the Silver Bullet for Insurance Pricing? [4.5497948012757865]
  Tabular foundation models (TFMs) present a fundamentally different inference paradigm.<n>By pre-training on large collections of synthetic datasets, TFMs enable inference on new data through in-context learning.<n>This paper presents a first empirical evaluation of TabPFN for motor insurance pricing.
  arXiv  Detail & Related papers  (2026-05-21T13:59:06Z)
• SLE-FNO: Single-Layer Extensions for Task-Agnostic Continual Learning in Fourier Neural Operators [0.4369550829556577]
  New experimental conditions or simulation regimes may differ significantly, requiring extrapolation and model updates without re-access to prior data.<n>This creates a need for continual learning (CL) frameworks that can adapt to distribution shifts while preventing catastrophic forgetting.<n>We introduce a new architecture-based approach (SLE-FNO) combining a Single-Layer Extension (SLE) with the Fourier Neural Operator (FNO) to support efficient CL.
  arXiv  Detail & Related papers  (2026-03-20T18:30:38Z)
• SPaRFT: Self-Paced Reinforcement Fine-Tuning for Large Language Models [51.74498855100541]
  Large language models (LLMs) have shown strong reasoning capabilities when fine-tuned with reinforcement learning (RL)<n>We propose textbfSPaRFT, a self-paced learning framework that enables efficient learning based on the capability of the model being trained.
  arXiv  Detail & Related papers  (2025-08-07T03:50:48Z)
• TabICL: A Tabular Foundation Model for In-Context Learning on Large Data [15.08819125687632]
  We introduce TabICL, a tabular foundation model for classification, pretrained on synthetic datasets with up to 60K samples.<n>Across 200 classification datasets from the TALENT benchmark, TabICL is on par with TabPFNv2 while being systematically faster (up to 10 times)<n>On 53 datasets with over 10K samples, TabICL surpasses both TabPFNv2 and CatBoost, demonstrating the potential of ICL for large data.
  arXiv  Detail & Related papers  (2025-02-08T13:25:04Z)
• TabDPT: Scaling Tabular Foundation Models on Real Data [20.00390825519329]
  We propose an approach to combine ICL-based retrieval with self supervised learning to train foundation models.<n>We show that incorporating real data during the pre-training phase can lead to significantly faster training and better generalization to unseen data.<n>Our resulting model, TabDPT, achieves top performance on both regression (CTR23) and classification (CC18) benchmarks.
  arXiv  Detail & Related papers  (2024-10-23T18:00:00Z)
• Self-Augmented Preference Optimization: Off-Policy Paradigms for Language Model Alignment [104.18002641195442]
  We introduce Self-Augmented Preference Optimization (SAPO), an effective and scalable training paradigm that does not require existing paired data. Building on the self-play concept, which autonomously generates negative responses, we further incorporate an off-policy learning pipeline to enhance data exploration and exploitation.
  arXiv  Detail & Related papers  (2024-05-31T14:21:04Z)
• Squeezing Lemons with Hammers: An Evaluation of AutoML and Tabular Deep Learning for Data-Scarce Classification Applications [2.663744975320783]
  We find that L2-regularized logistic regression performs similar to state-of-the-art automated machine learning (AutoML) frameworks. We recommend to consider logistic regression as the first choice for data-scarce applications.
  arXiv  Detail & Related papers  (2024-05-13T11:43:38Z)
• The Languini Kitchen: Enabling Language Modelling Research at Different Scales of Compute [66.84421705029624]
  We introduce an experimental protocol that enables model comparisons based on equivalent compute, measured in accelerator hours. We pre-process an existing large, diverse, and high-quality dataset of books that surpasses existing academic benchmarks in quality, diversity, and document length. This work also provides two baseline models: a feed-forward model derived from the GPT-2 architecture and a recurrent model in the form of a novel LSTM with ten-fold throughput.
  arXiv  Detail & Related papers  (2023-09-20T10:31:17Z)
• MLLM-DataEngine: An Iterative Refinement Approach for MLLM [62.30753425449056]
  We propose a novel closed-loop system that bridges data generation, model training, and evaluation. Within each loop, the MLLM-DataEngine first analyze the weakness of the model based on the evaluation results. For targeting, we propose an Adaptive Bad-case Sampling module, which adjusts the ratio of different types of data. For quality, we resort to GPT-4 to generate high-quality data with each given data type.
  arXiv  Detail & Related papers  (2023-08-25T01:41:04Z)
• Branch-Train-Merge: Embarrassingly Parallel Training of Expert Language Models [106.65127123304842]
  Branch-Train-Merge (BTM) is an efficient algorithm for parallel training of large language models (LLMs) BTM learns a set of independent expert LMs (ELMs), each specialized to a different textual domain. Experiments show that BTM improves in- and out-of-domain perplexities as compared to GPT-style Transformer LMs.
  arXiv  Detail & Related papers  (2022-08-05T17:46:38Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.