PlotChain: Deterministic Checkpointed Evaluation of Multimodal LLMs on Engineering Plot Reading

• URL: http://arxiv.org/abs/2602.13232v1
• Date: Thu, 29 Jan 2026 06:08:19 GMT
• Title: PlotChain: Deterministic Checkpointed Evaluation of Multimodal LLMs on Engineering Plot Reading
• Abstract summary: PlotChain is a deterministic, generator-based benchmark for evaluating large language models (MLLMs)<n>PlotChain contains 15 plot families with 450 rendered plots (30 per family)<n>Top models achieve 80.42% (Gemini 2.5 Pro), 79.84% (GPT-4.1), and 78.21% (Claude Sonnet 4.5) overall field-level pass rates, while GPT-4o trails at 61.59%.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We present PlotChain, a deterministic, generator-based benchmark for evaluating multimodal large language models (MLLMs) on engineering plot reading-recovering quantitative values from classic plots (e.g., Bode/FFT, step response, stress-strain, pump curves) rather than OCR-only extraction or free-form captioning. PlotChain contains 15 plot families with 450 rendered plots (30 per family), where every item is produced from known parameters and paired with exact ground truth computed directly from the generating process. A central contribution is checkpoint-based diagnostic evaluation: in addition to final targets, each item includes intermediate 'cp_' fields that isolate sub-skills (e.g., reading cutoff frequency or peak magnitude) and enable failure localization within a plot family. We evaluate four state-of-the-art MLLMs under a standardized, deterministic protocol (temperature = 0 and a strict JSON-only numeric output schema) and score predictions using per-field tolerances designed to reflect human plot-reading precision. Under the 'plotread' tolerance policy, the top models achieve 80.42% (Gemini 2.5 Pro), 79.84% (GPT-4.1), and 78.21% (Claude Sonnet 4.5) overall field-level pass rates, while GPT-4o trails at 61.59%. Despite strong performance on many families, frequency-domain tasks remain brittle: bandpass response stays low (<= 23%), and FFT spectrum remains challenging. We release the generator, dataset, raw model outputs, scoring code, and manifests with checksums to support fully reproducible runs and retrospective rescoring under alternative tolerance policies.

Related papers

• Exact Finite Attention Responses From RoPE Derivatives [0.0]
  We derive exact local responses for attention interventions, allowing candidate edits to be scored from a cached baseline and one backward pass.<n>The resulting predictions achieve 95.36--96.52% sign accuracy across 92,160 executed positional edits.
  arXiv  Detail & Related papers  (2026-09-12T20:15:37Z)
• Anchoring Bias in LLM-as-a-Judge Systems: Prior Scores Compromise Evaluation Independence [0.7340017786387767]
  We show that prior scores, even when included only as context metadata, anchor judgments and systematically shift ratings toward their values.<n>On categorical industry data with human-labeled ground truth, anchored metadata blocks 48% of error corrections and flips 10.18% of correct judgments toward an assigned wrong label.
  arXiv  Detail & Related papers  (2026-08-26T14:41:24Z)
• Position Bias is Hidden Behind Ceiling Effects: A Permutation Diagnostic for LLM Benchmarks [0.0]
  I introduce inspect_permute, an open-source extension to inspect_ai- evaluation framework.<n>It runs exhaustive answer_order permutations per question.<n>It reports the chi-squared / Cramer V signature of position bias with bootstrap confidence intervals.
  arXiv  Detail & Related papers  (2026-07-23T02:45:02Z)
• FourierQK: Spectral Preprocessing of Query-Key Projections Improves Transformer Attention [0.0]
  spectral preprocessing of Q/K projections improves transformer attention on character-level language modelling.<n>Results are verified by a shuffled-validation diagnostic against positional leakage.<n>This work defines an architectural boundary between bilateral spectral attention and causal attention at wordscale tokenisation.
  arXiv  Detail & Related papers  (2026-07-08T14:43:00Z)
• Constrained Decoding for Diffusion Language Models via Efficient Inference over Finite Automata [57.27430779838529]
  We present an exact and tractable algorithm for sampling from the constrained mean-field posterior under any constraint expressible as a finite automaton.<n>The approach guarantees constraint satisfaction by construction, supports both greedy and sampling-based decoding, and is compatible with parallel and block-wise decoding.<n> Empirical evaluations on Dream-7B and LLaDA-8 show substantial accuracy gains across various tasks.
  arXiv  Detail & Related papers  (2026-07-08T05:48:57Z)
• DoPE: Denoising Rotary Position Embedding [60.779039511252584]
  Rotary Position Embedding (RoPE) in Transformer models has inherent limits that weaken length.<n>We reinterpret the attention map with positional encoding as a noisy feature map, and propose Denoising Positional extrapolation page (DoPE)<n>DoPE is a training-free method based on truncated matrix entropy to detect outlier frequency bands in the feature map.
  arXiv  Detail & Related papers  (2025-11-12T09:32:35Z)
• Ensemble Threshold Calibration for Stable Sensitivity Control [0.0]
  We present an end-to-end framework that achieves exact recall with sub-percent variance over tens of millions of geometry pairs.<n>Our approach consistently hits a recall target within a small error, decreases redundant verifications relative to other calibrations, and runs end-to-end on a single TPU v3 core.
  arXiv  Detail & Related papers  (2025-10-02T15:22:28Z)
• An analysis of the noise schedule for score-based generative models [7.180235086275926]
  Score-based generative models (SGMs) aim at estimating a target data distribution by learning score functions using only noise-perturbed samples from the target.<n>Recent literature has focused extensively on assessing the error between the target and estimated distributions, gauging the generative quality through the Kullback-Leibler (KL) divergence and Wasserstein distances.<n>We establish an upper bound for the KL divergence between the target and the estimated distributions, explicitly depending on any time-dependent noise schedule.
  arXiv  Detail & Related papers  (2024-02-07T08:24:35Z)
• A Meta-Learning Approach to Predicting Performance and Data Requirements [163.4412093478316]
  We propose an approach to estimate the number of samples required for a model to reach a target performance. We find that the power law, the de facto principle to estimate model performance, leads to large error when using a small dataset. We introduce a novel piecewise power law (PPL) that handles the two data differently.
  arXiv  Detail & Related papers  (2023-03-02T21:48:22Z)
• QuTE: decentralized multiple testing on sensor networks with false discovery rate control [93.1040521878626]
  This paper designs methods for decentralized multiple hypothesis testing on graphs equipped with provable guarantees on the false discovery rate (FDR)<n>We consider the setting where distinct agents reside on the nodes of an undirected graph, and each agent possesses p-values corresponding to one or more hypotheses local to its node.<n>Each agent must individually decide whether to reject one or more of its local hypotheses by only communicating with its neighbors, with the joint aim that the global FDR over the entire graph must be controlled at a predefined level.
  arXiv  Detail & Related papers  (2022-10-09T19:48:39Z)
• Content Popularity Prediction Based on Quantized Federated Bayesian Learning in Fog Radio Access Networks [76.16527095195893]
  We investigate the content popularity prediction problem in cache-enabled fog radio access networks (F-RANs) In order to predict the content popularity with high accuracy and low complexity, we propose a Gaussian process based regressor to model the content request pattern. We utilize Bayesian learning to train the model parameters, which is robust to overfitting.
  arXiv  Detail & Related papers  (2022-06-23T03:05:12Z)
• $\texttt{FedBC}$: Calibrating Global and Local Models via Federated Learning Beyond Consensus [66.62731854746856]
  In federated learning (FL), the objective of collaboratively learning a global model through aggregation of model updates across devices tends to oppose the goal of personalization via local information. In this work, we calibrate this tradeoff in a quantitative manner through a multi-criterion-based optimization. We demonstrate that $texttFedBC$ balances the global and local model test accuracy metrics across a suite datasets.
  arXiv  Detail & Related papers  (2022-06-22T02:42:04Z)
• Sparse Feature Selection Makes Batch Reinforcement Learning More Sample Efficient [62.24615324523435]
  This paper provides a statistical analysis of high-dimensional batch Reinforcement Learning (RL) using sparse linear function approximation. When there is a large number of candidate features, our result sheds light on the fact that sparsity-aware methods can make batch RL more sample efficient.
  arXiv  Detail & Related papers  (2020-11-08T16:48:02Z)
• MOCHA: A Dataset for Training and Evaluating Generative Reading Comprehension Metrics [55.85042753772513]
  We introduce a benchmark for training and evaluating generative reading comprehension metrics: MOdeling Correctness with Human. s. Using MOCHA, we train a Learned Evaluation metric for Reading Pearson, LERC, to mimic human judgement scores. LERC outperforms baseline metrics by 10 to 36 absolute points on held-out annotations. When we evaluate on minimal pairs, LERC achieves 80% accuracy, outperforming baselines by 14 to 26 absolute percentage points while leaving significant room for improvement.
  arXiv  Detail & Related papers  (2020-10-07T20:22:54Z)
• Continuous Release of Data Streams under both Centralized and Local Differential Privacy [30.998501044718548]
  We study the problem of publishing a stream of real-valued data satisfying differential privacy (DP) One major challenge is that the maximal possible value can be quite large. We develop a method that uses the Exponential Mechanism with a quality function that approximates well the utility goal.
  arXiv  Detail & Related papers  (2020-05-24T14:25:49Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.