HighlightBench: Benchmarking Markup-Driven Table Reasoning in Scientific Documents

• URL: http://arxiv.org/abs/2603.26784v1
• Date: Wed, 25 Mar 2026 06:15:40 GMT
• Title: HighlightBench: Benchmarking Markup-Driven Table Reasoning in Scientific Documents
• Abstract summary: multimodal large language models (MLLMs) have made substantial progress in document understanding, but their ability to treat such cues as explicit logical directives remains under-explored.<n>We introduce HighlightBench, a diagnostic benchmark for markup-driven table understanding that decomposes evaluation into five task families.<n>We show that even strong models remain unstable when visual cues must be consistently aligned with symbolic reasoning under structured output constraints.
• Score: 88.16322879912458
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Visual markups such as highlights, underlines, and bold text are common in table-centric documents. Although multimodal large language models (MLLMs) have made substantial progress in document understanding, their ability to treat such cues as explicit logical directives remains under-explored. More importantly, existing evaluations cannot distinguish whether a model fails to see the markup or fails to reason with it. This creates a key blind spot in assessing markup-conditioned behavior over tables. To address this gap, we introduce HighlightBench, a diagnostic benchmark for markup-driven table understanding that decomposes evaluation into five task families: Markup Grounding, Constrained Retrieval, Local Relations, Aggregation \& Comparison, and Consistency \& Missingness. We further provide a reference pipeline that makes intermediate decisions explicit, enabling reproducible baselines and finer-grained attribution of errors along the perception-to-execution chain. Experiments show that even strong models remain unstable when visual cues must be consistently aligned with symbolic reasoning under structured output constraints.

Related papers

• TABVERSE: Benchmarking Cross-Format Table Understanding in LLMs and VLMs [50.96120467118319]
  Large Language Models (LLMs) and Vision-Language Models (VLMs) are increasingly evaluated on table reasoning tasks.<n>We introduce TABVERSE, a controlled multimodal table benchmark that aligns the same table content across multiple structural formats.<n>Our results show that representation choice substantially affects table understanding.
  arXiv  Detail & Related papers  (2026-06-08T14:52:46Z)
• Faithful or Fabricated? A Causal Framework for Rationalization Bias in LLM Judges [19.04544141221272]
  Large language models (LLMs) are increasingly used as automatic judges for summarization and dialogue evaluation.<n>We ask whether LLM judges are cue-invariant, i.e., whether their rankings and explanations remain stable when non-evidential cues are perturbed.<n>We introduce a suite of cue interventions (Blind, Truth, Flip, Placebo, Reveal-After) and tie-aware metrics that quantify outcome anchoring and rationale anchoring.
  arXiv  Detail & Related papers  (2026-05-13T07:00:16Z)
• IF-RewardBench: Benchmarking Judge Models for Instruction-Following Evaluation [85.56193980646981]
  We propose IF-RewardBench, a comprehensive meta-evaluation benchmark for instruction-following.<n>For each instruction, we construct a preference graph containing all pairwise preferences among multiple responses.<n>Experiments on IF-RewardBench reveal significant deficiencies in current judge models.
  arXiv  Detail & Related papers  (2026-03-05T02:21:17Z)
• FMBench: Adaptive Large Language Model Output Formatting [49.52930069696333]
  We present FMBench, a benchmark for adaptive Markdown output formatting.<n>Experiments on two model families show that SFT consistently improves semantic alignment.<n>Results also reveal an inherent trade-off between semantic and structural objectives.
  arXiv  Detail & Related papers  (2026-02-06T04:42:06Z)
• Visually Prompted Benchmarks Are Surprisingly Fragile [82.98001690512461]
  Key challenge in evaluating VLMs is testing their ability to analyze visual content independently from their textual priors.<n>We demonstrate how details in benchmark setup, including visual marker design and dataset size, have a significant influence on model performance and leaderboard rankings.<n>To mitigate this instability, we curate existing datasets to create VPBench, a larger visually prompted benchmark with 16 visual marker variants.
  arXiv  Detail & Related papers  (2025-12-19T18:26:58Z)
• TabReX : Tabular Referenceless eXplainable Evaluation [15.411207072791806]
  TabReX is a reference-less, property-driven framework for evaluating tables generated by large language models.<n>It computes interpretable, rubric-aware scores that quantify structural and factual fidelity.<n>To asses robustness, we introduce TabReX-Bench, a large-scale benchmark spanning six domains and twelve planner-driven perturbation types.
  arXiv  Detail & Related papers  (2025-12-17T19:20:20Z)
• DAG-Math: Graph-Guided Mathematical Reasoning in LLMs [54.231935013127206]
  Large Language Models (LLMs) demonstrate strong performance on mathematical problems when prompted with Chain-of-Thought (CoT)<n>We propose modeling CoT as a certain rule-based process over directed acyclic graphs (DAGs)<n>We introduce logical closeness, a metric that quantifies how well a model's CoT trajectory adheres to the DAG structure.
  arXiv  Detail & Related papers  (2025-10-19T21:05:17Z)
• GRAFT: GRaPH and Table Reasoning for Textual Alignment -- A Benchmark for Structured Instruction Following and Visual Reasoning [0.0]
  GRAFT is a structured multimodal benchmark for evaluating models on instruction-following visual reasoning, and visual-textual alignment.<n>It features generated charts and synthetically rendered tables, created with Python libraries to ensure control over data semantics, and clarity.
  arXiv  Detail & Related papers  (2025-08-21T16:13:49Z)
• TabXEval: Why this is a Bad Table? An eXhaustive Rubric for Table Evaluation [10.212570261759204]
  We propose a rubric-based evaluation framework that integrates multi-level structural descriptors with fine-grained contextual signals.<n>We introduce TabXEval, an eXhaustive and eXplainable two-phase evaluation framework.<n>We evaluate TabXEval on TabXBench, a diverse, multi-domain benchmark featuring realistic table perturbations and human annotations.
  arXiv  Detail & Related papers  (2025-05-28T09:50:29Z)
• SEMv2: Table Separation Line Detection Based on Instance Segmentation [96.36188168694781]
  We propose an accurate table structure recognizer, termed SEMv2 (SEM: Split, Embed and Merge) We address the table separation line instance-level discrimination problem and introduce a table separation line detection strategy based on conditional convolution. To comprehensively evaluate the SEMv2, we also present a more challenging dataset for table structure recognition, dubbed iFLYTAB.
  arXiv  Detail & Related papers  (2023-03-08T05:15:01Z)
• Does Recommend-Revise Produce Reliable Annotations? An Analysis on Missing Instances in DocRED [60.39125850987604]
  We show that a textit-revise scheme results in false negative samples and an obvious bias towards popular entities and relations. The relabeled dataset is released to serve as a more reliable test set of document RE models.
  arXiv  Detail & Related papers  (2022-04-17T11:29:01Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.