Lost in Translation and Noise: A Deep Dive into the Failure Modes of VLMs on Real-World Tables

• URL: http://arxiv.org/abs/2511.17238v1
• Date: Fri, 21 Nov 2025 13:32:56 GMT
• Title: Lost in Translation and Noise: A Deep Dive into the Failure Modes of VLMs on Real-World Tables
• Abstract summary: Existing datasets for QA, such as WikiTableQuestions and FinQA, are overwhelmingly monolingual (English)<n>MirageTVQA provides a benchmark for measuring and driving progress towards more robust VLM models for table reasoning.
• Score: 0.1749935196721634
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The impressive performance of VLMs is largely measured on benchmarks that fail to capture the complexities of real-world scenarios. Existing datasets for tabular QA, such as WikiTableQuestions and FinQA, are overwhelmingly monolingual (English) and present tables in a digitally perfect, clean format. This creates a significant gap between research and practice. To address this, we present \textbf{MirageTVQA}, a new benchmark designed to evaluate VLMs on these exact dimensions. Featuring nearly 60,000 QA pairs across 24 languages, MirageTVQA challenges models with tables that are not only multilingual but also visually imperfect, incorporating realistic noise to mimic scanned documents. Our evaluation of the leading VLMs reveals two primary failure points: a severe degradation in performance (over 35\% drop for the best models) when faced with visual noise and a consistent English-first bias where reasoning abilities fail to transfer to other languages. MirageTVQA provides a benchmark for measuring and driving progress towards more robust VLM models for table reasoning. The dataset and the code are available at: https://github.com/anshulsc/MirageTVQA.

Related papers

• EMBLEM: Enhancing Multi-script Table Detection through Masking [14.822713626646332]
  We introduce MANDALA (Multi-script Annotated Documents for Table Detection), a dataset of 2,323 table-containing pages spanning 18 languages and 15 scripts across diverse domains.<n>We also propose EMBLEM, a masking-based paradigm for Multi-script Table Detection (MTD)<n>EMBLEM generates masked images that conceal script- and font-specific details, enabling models pre-trained on abundant English documents to focus on script-agnostic page layout.
  arXiv  Detail & Related papers  (2026-09-08T06:57:50Z)
• TALENT: Table VQA via Augmented Language-Enhanced Natural-text Transcription [15.734561144090641]
  Table Visual Question Answering (Table VQA) is typically addressed by large vision-language models (VLMs)<n>We propose TALENT, a lightweight framework that leverages dual representations of tables.
  arXiv  Detail & Related papers  (2025-10-08T14:56:42Z)
• Demystifying the Visual Quality Paradox in Multimodal Large Language Models [49.154146792279946]
  Recent Multimodal Large Language Models (MLLMs) excel on benchmark vision-language tasks, yet little is known about how input visual quality shapes their responses.<n>We conduct the first systematic study spanning leading MLLMs and a suite of vision-language benchmarks.<n>We uncover a visual-quality paradox: model, task, and even individual-instance performance can improve when images deviate from human-perceived fidelity.
  arXiv  Detail & Related papers  (2025-06-18T17:14:07Z)
• TableEval: A Real-World Benchmark for Complex, Multilingual, and Multi-Structured Table Question Answering [41.12322828986314]
  Existing TableQA benchmarks are often limited by their focus on simple flat tables and suffer from data leakage.<n>We introduce TableEval, a new benchmark designed to evaluate LLMs on realistic TableQA tasks.<n>To minimize the risk of data leakage, we collect all data from recent real-world documents.
  arXiv  Detail & Related papers  (2025-06-04T13:39:01Z)
• LENS: Multi-level Evaluation of Multimodal Reasoning with Large Language Models [59.0256377330646]
  Lens is a benchmark with 3.4K contemporary images and 60K+ human-authored questions covering eight tasks and 12 daily scenarios.<n>This dataset intrinsically supports to evaluate MLLMs to handle image-invariable prompts, from basic perception to compositional reasoning.<n>We evaluate 15+ frontier MLLMs such as Qwen2.5-VL-72B, InternVL3-78B, GPT-4o and two reasoning models QVQ-72B-preview and Kimi-VL.
  arXiv  Detail & Related papers  (2025-05-21T15:06:59Z)
• Game-RL: Synthesizing Multimodal Verifiable Game Data to Boost VLMs' General Reasoning [89.93384726755106]
  Vision-language reinforcement learning (RL) has primarily focused on narrow domains.<n>We find video games inherently provide rich visual elements and mechanics that are easy to verify.<n>To fully use the multimodal and verifiable reward in video games, we propose Game-RL.
  arXiv  Detail & Related papers  (2025-05-20T03:47:44Z)
• VisualSimpleQA: A Benchmark for Decoupled Evaluation of Large Vision-Language Models in Fact-Seeking Question Answering [28.045285777736876]
  We introduce VisualSimpleQA, a multimodal fact-seeking benchmark with two key features.<n>It enables streamlined and decoupled evaluation of LVLMs in visual and linguistic modalities.<n> Experiments on 15 LVLMs show that even state-of-the-art models such as GPT-4o achieve merely 60%+ correctness.
  arXiv  Detail & Related papers  (2025-03-09T07:25:32Z)
• Protecting multimodal large language models against misleading visualizations [94.71976205962527]
  We show that questionanswering (QA) accuracy on misleading visualizations drops on average to the level of the random baseline.<n>We introduce the first inference-time methods to improve QA performance on misleading visualizations, without compromising accuracy on non-misleading ones.<n>We find that two methods, table-based QA and redrawing the visualization, are effective, with improvements of up to 19.6 percentage points.
  arXiv  Detail & Related papers  (2025-02-27T20:22:34Z)
• VHELM: A Holistic Evaluation of Vision Language Models [75.88987277686914]
  We present the Holistic Evaluation of Vision Language Models (VHELM) VHELM aggregates various datasets to cover one or more of the 9 aspects: visual perception, knowledge, reasoning, bias, fairness, multilinguality, robustness, toxicity, and safety. Our framework is designed to be lightweight and automatic so that evaluation runs are cheap and fast.
  arXiv  Detail & Related papers  (2024-10-09T17:46:34Z)
• Revisiting Multi-Modal LLM Evaluation [29.094387692681337]
  We pioneer evaluating recent MLLMs (LLaVA 1.5, LLaVA-NeXT, BLIP2, InstructBLIP, GPT-4V, and GPT-4o) on datasets designed to address weaknesses in earlier ones. Our code is integrated into the widely used LAVIS framework for MLLM evaluation, enabling the rapid assessment of future MLLMs.
  arXiv  Detail & Related papers  (2024-08-09T20:55:46Z)
• The Devil is in the Errors: Leveraging Large Language Models for Fine-grained Machine Translation Evaluation [93.01964988474755]
  AutoMQM is a prompting technique which asks large language models to identify and categorize errors in translations. We study the impact of labeled data through in-context learning and finetuning. We then evaluate AutoMQM with PaLM-2 models, and we find that it improves performance compared to just prompting for scores.
  arXiv  Detail & Related papers  (2023-08-14T17:17:21Z)
• MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models [111.51612340032052]
  Multimodal Large Language Model (MLLM) relies on the powerful LLM to perform multimodal tasks.<n>This paper presents the first comprehensive MLLM Evaluation benchmark MME.<n>It measures both perception and cognition abilities on a total of 14 subtasks.
  arXiv  Detail & Related papers  (2023-06-23T09:22:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.