MulTaBench: Benchmarking Multimodal Tabular Learning with Text and Image

• URL: http://arxiv.org/abs/2605.10616v1
• Date: Mon, 11 May 2026 14:12:05 GMT
• Title: MulTaBench: Benchmarking Multimodal Tabular Learning with Text and Image
• Abstract summary: We introduce MulTaBench, a benchmark of 40 datasets split equally between image-tabular and text-tabular tasks.<n>We show that the gains from target-aware representation tuning generalize across both text and image modalities.
• Score: 52.47434184153733
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Tabular Foundation Models have recently established the state of the art in supervised tabular learning, by leveraging pretraining to learn generalizable representations of numerical and categorical structured data. However, they lack native support for unstructured modalities such as text and image, and rely on frozen, pretrained embeddings to process them. On established Multimodal Tabular Learning benchmarks, we show that tuning the embeddings to the task improves performance. Existing benchmarks, however, often focus on the mere co-occurrence of modalities; this leads to high variance across datasets and masks the benefits of task-specific tuning. To address this gap, we introduce MulTaBench, a benchmark of 40 datasets, split equally between image-tabular and text-tabular tasks. We focus on predictive tasks where the modalities provide complementary predictive signal, and where generic embeddings lose critical information, necessitating Target-Aware Representations that are aligned with the task. Our experimental results demonstrate that the gains from target-aware representation tuning generalize across both text and image modalities, several tabular learners, encoder scales, and embedding dimensions. MulTaBench constitutes the largest image-tabular benchmarking effort to date, spanning high-impact domains such as healthcare and e-commerce. It is designed to enable the research of novel architectures which incorporate joint modeling and target-aware representations, paving the way for the development of novel Multimodal Tabular Foundation Models.

Related papers

• Deeply Interleaved Text-Image Contexts for Multimodal LLMs Assessment [57.41731607488101]
  TIC-Bench is designed to evaluate the capability of models to integrate text-image clues and recover the ground truth facts within deeply interleaved contexts.<n>We evaluated 10 state-of-the-art MLLMs and observed a substantial performance gap compared to human experts.
  arXiv  Detail & Related papers  (2026-09-02T13:19:51Z)
• TabSTAR: A Tabular Foundation Model for Tabular Data with Text Fields [12.860878027211522]
  Tabular Foundation Models can leverage real-world knowledge and generalize across diverse datasets.<n>We introduce TabSTAR: a Tabular Foundation Model with Semantically Target-Aware Representations.
  arXiv  Detail & Related papers  (2025-05-23T17:34:28Z)
• Leveraging Foundation Models for Multimodal Graph-Based Action Recognition [2.066890710233268]
  We introduce a graph-based framework that integrates a vision-temporal foundation leveraging VideoMAE for dynamic visual encoding and BERT for contextual textual embedding.<n>We show that our method consistently outperforms state-of-the-art baselines on diverse benchmark datasets.
  arXiv  Detail & Related papers  (2025-05-21T07:15:14Z)
• Make Still Further Progress: Chain of Thoughts for Tabular Data Leaderboard [27.224577475861214]
  Tabular data, a fundamental data format in machine learning, is predominantly utilized in competitions and real-world applications.<n>We propose an in-context ensemble framework for tabular prediction that leverages large language models.<n>Our method constructs a context around each test instance using its nearest neighbors and the predictions from a pool of external models.
  arXiv  Detail & Related papers  (2025-05-19T17:52:58Z)
• Representation Learning for Tabular Data: A Comprehensive Survey [23.606506938919605]
  Tabular data, structured as rows and columns, is among the most prevalent data types in machine learning classification and regression applications.<n>Deep Neural Networks (DNNs) have recently demonstrated promising results through their capability of representation learning.<n>We organize existing methods into three main categories according to their generalization capabilities.
  arXiv  Detail & Related papers  (2025-04-17T17:58:23Z)
• Towards Text-Image Interleaved Retrieval [49.96332254241075]
  We introduce the text-image interleaved retrieval (TIIR) task, where the query and document are interleaved text-image sequences.<n>We construct a TIIR benchmark based on naturally interleaved wikiHow tutorials, where a specific pipeline is designed to generate interleaved queries.<n>We propose a novel Matryoshka Multimodal Embedder (MME), which compresses the number of visual tokens at different granularity.
  arXiv  Detail & Related papers  (2025-02-18T12:00:47Z)
• Boosting Short Text Classification with Multi-Source Information Exploration and Dual-Level Contrastive Learning [12.377363857246602]
  We propose a novel model named MI-DELIGHT for short text classification.<n>It first performs multi-source information exploration to alleviate the sparsity issues.<n>Then, the graph learning approach is adopted to learn the representation of short texts.
  arXiv  Detail & Related papers  (2025-01-16T00:26:15Z)
• Flex: End-to-End Text-Instructed Visual Navigation from Foundation Model Features [59.892436892964376]
  We investigate the minimal data requirements and architectural adaptations necessary to achieve robust closed-loop performance with vision-based control policies.<n>Our findings are synthesized in Flex (Fly lexically), a framework that uses pre-trained Vision Language Models (VLMs) as frozen patch-wise feature extractors.<n>We demonstrate the effectiveness of this approach on a quadrotor fly-to-target task, where agents trained via behavior cloning successfully generalize to real-world scenes.
  arXiv  Detail & Related papers  (2024-10-16T19:59:31Z)
• Knowledge-Aware Reasoning over Multimodal Semi-structured Tables [85.24395216111462]
  This study investigates whether current AI models can perform knowledge-aware reasoning on multimodal structured data. We introduce MMTabQA, a new dataset designed for this purpose. Our experiments highlight substantial challenges for current AI models in effectively integrating and interpreting multiple text and image inputs.
  arXiv  Detail & Related papers  (2024-08-25T15:17:43Z)
• UniDiff: Advancing Vision-Language Models with Generative and Discriminative Learning [86.91893533388628]
  This paper presents UniDiff, a unified multi-modal model that integrates image-text contrastive learning (ITC), text-conditioned image synthesis learning (IS), and reciprocal semantic consistency modeling (RSC) UniDiff demonstrates versatility in both multi-modal understanding and generative tasks.
  arXiv  Detail & Related papers  (2023-06-01T15:39:38Z)
• High-Modality Multimodal Transformer: Quantifying Modality & Interaction Heterogeneity for High-Modality Representation Learning [112.51498431119616]
  This paper studies efficient representation learning for high-modality scenarios involving a large set of diverse modalities. A single model, HighMMT, scales up to 10 modalities (text, image, audio, video, sensors, proprioception, speech, time-series, sets, and tables) and 15 tasks from 5 research areas.
  arXiv  Detail & Related papers  (2022-03-02T18:56:20Z)
• Improving Image Captioning with Better Use of Captions [65.39641077768488]
  We present a novel image captioning architecture to better explore semantics available in captions and leverage that to enhance both image representation and caption generation. Our models first construct caption-guided visual relationship graphs that introduce beneficial inductive bias using weakly supervised multi-instance learning. During generation, the model further incorporates visual relationships using multi-task learning for jointly predicting word and object/predicate tag sequences.
  arXiv  Detail & Related papers  (2020-06-21T14:10:47Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.