ViTaB-A: Evaluating Multimodal Large Language Models on Visual Table Attribution

• URL: http://arxiv.org/abs/2602.15769v1
• Date: Tue, 17 Feb 2026 18:01:35 GMT
• Title: ViTaB-A: Evaluating Multimodal Large Language Models on Visual Table Attribution
• Abstract summary: Multimodal Large Language Models (mLLMs) are often used to answer questions in structured data.<n>We study structured data attribution/citation, which is the ability of the models to point to the specific rows and columns that support an answer.
• Score: 37.049064455196486
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Multimodal Large Language Models (mLLMs) are often used to answer questions in structured data such as tables in Markdown, JSON, and images. While these models can often give correct answers, users also need to know where those answers come from. In this work, we study structured data attribution/citation, which is the ability of the models to point to the specific rows and columns that support an answer. We evaluate several mLLMs across different table formats and prompting strategies. Our results show a clear gap between question answering and evidence attribution. Although question answering accuracy remains moderate, attribution accuracy is much lower, near random for JSON inputs, across all models. We also find that models are more reliable at citing rows than columns, and struggle more with textual formats than images. Finally, we observe notable differences across model families. Overall, our findings show that current mLLMs are unreliable at providing fine-grained, trustworthy attribution for structured data, which limits their usage in applications requiring transparency and traceability.

Related papers

• Prompt Robustness Is Task-Dependent: Comparing Objective and Belief-Style Questions in LLM Evaluation [1.7613484075466104]
  We ask whether prompt robustness differs between objective questions with fixed answers and subjective questions that ask for opinions or values.<n>We evaluate four instruction-tuned model families on three objective datasets and three subjective datasets.<n>Using a binomial generalized estimating equation, we find significant effects of model, dataset, prompt category, and their interactions.
  arXiv  Detail & Related papers  (2026-07-06T18:47:47Z)
• Don't Always Pick the Highest-Performing Model: An Information Theoretic View of LLM Ensemble Selection [8.266188814122605]
  Large language models (LLMs) are often ensembled together to improve overall reliability and robustness, but in practice models are strongly correlated.<n>We formulate budgeted ensemble selection as maximizing the mutual information between the true label and predictions of the selected models.<n>Motivated by these, we propose a simple greedy mutual-information selection algorithm that estimates the required information terms directly from data.
  arXiv  Detail & Related papers  (2026-02-08T15:05:22Z)
• TRivia: Self-supervised Fine-tuning of Vision-Language Models for Table Recognition [54.85932472676512]
  Table recognition (TR) aims to transform table images into semi-structured representations such as HTML or markdown.<n>We introduce TRivia, a self-supervised fine-tuning method that enables pretrained vision-language models to learn TR directly from unlabeled table images.<n>We present TRivia-3B, an open-sourced, compact, and state-of-the-art TR model that surpasses existing systems.
  arXiv  Detail & Related papers  (2025-12-01T03:49:00Z)
• Checkmate: interpretable and explainable RSVQA is the endgame [5.445304535169411]
  We introduce a novel RSVQA dataset, Chessboard, designed to minimize biases through 3'123'253 questions.<n>Each answer is linked to one or more cells within the image, enabling fine-grained visual reasoning.<n>We develop an explainable and interpretable model called Checkmate that identifies the image cells most relevant to its decisions.
  arXiv  Detail & Related papers  (2025-08-18T16:59:43Z)
• Surface Fairness, Deep Bias: A Comparative Study of Bias in Language Models [45.41676783204022]
  We investigate various proxy measures of bias in large language models (LLMs)<n>We find that evaluating models with pre-prompted personae on a multi-subject benchmark (MMLU) leads to negligible and mostly random differences in scores.<n>With the recent trend for LLM assistant memory and personalization, these problems open up from a different angle.
  arXiv  Detail & Related papers  (2025-06-12T08:47:40Z)
• MINERVA: Evaluating Complex Video Reasoning [72.12644008002566]
  We provide a new video reasoning dataset called MINERVA for modern multimodal models.<n>Our dataset is multimodal, diverse in terms of video domain and length, and consists of complex multi-step questions.<n>We perform fine-grained error analysis to identify common failure modes across various models, and create a taxonomy of reasoning errors.
  arXiv  Detail & Related papers  (2025-05-01T17:41:49Z)
• The Mighty ToRR: A Benchmark for Table Reasoning and Robustness [45.420943398134845]
  ToRR is a benchmark for Table Reasoning and Robustness, measuring model performance and robustness on table-related tasks.<n>We present a leaderboard as well as comprehensive analyses of the results of leading models over ToRR.
  arXiv  Detail & Related papers  (2025-02-26T18:56:38Z)
• Ranked from Within: Ranking Large Multimodal Models Without Labels [73.96543593298426]
  We show that uncertainty scores derived from softmax distributions provide a robust basis for ranking models across various tasks.<n>This facilitates the ranking of LMMs on unlabeled data, providing a practical approach for selecting models for diverse target domains without requiring manual annotation.
  arXiv  Detail & Related papers  (2024-12-09T13:05:43Z)
• Answer, Assemble, Ace: Understanding How LMs Answer Multiple Choice Questions [103.20281438405111]
  Multiple-choice question answering (MCQA) is a key competence of performant transformer language models.<n>We employ vocabulary projection and activation patching methods to localize key hidden states that encode relevant information for predicting the correct answer.<n>We show that subsequent layers increase the probability of the predicted answer symbol in vocabulary space, and that this probability increase is associated with a sparse set of attention heads with unique roles.
  arXiv  Detail & Related papers  (2024-07-21T00:10:23Z)
• Changing Answer Order Can Decrease MMLU Accuracy [18.774650080306944]
  We investigate the robustness of the accuracy measurement on a widely used multiple choice question answering dataset, MMLU. When shuffling the answer label contents, we find that all explored models decrease in accuracy on MMLU, but not every model is equally sensitive.
  arXiv  Detail & Related papers  (2024-06-27T18:21:32Z)
• CALM : A Multi-task Benchmark for Comprehensive Assessment of Language Model Bias [7.28980829208179]
  Comprehensive Assessment of Language Models (CALM) for robust measurement of two types of universally relevant sociodemographic bias, gender and race. Our empirical evaluation shows that CALM bias scores are more robust and far less sensitive than previous bias measurements to perturbations in the templates.
  arXiv  Detail & Related papers  (2023-08-24T03:53:55Z)
• Getting MoRE out of Mixture of Language Model Reasoning Experts [71.61176122960464]
  We propose a Mixture-of-Reasoning-Experts (MoRE) framework that ensembles diverse specialized language models. We specialize the backbone language model with prompts optimized for different reasoning categories, including factual, multihop, mathematical, and commonsense reasoning. Our human study confirms that presenting expert predictions and the answer selection process helps annotators more accurately calibrate when to trust the system's output.
  arXiv  Detail & Related papers  (2023-05-24T02:00:51Z)
• Robust Question Answering Through Sub-part Alignment [53.94003466761305]
  We model question answering as an alignment problem. We train our model on SQuAD v1.1 and test it on several adversarial and out-of-domain datasets.
  arXiv  Detail & Related papers  (2020-04-30T09:10:57Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.