QCalEval: Benchmarking Vision-Language Models for Quantum Calibration Plot Understanding

• URL: http://arxiv.org/abs/2604.25884v1
• Date: Tue, 28 Apr 2026 17:28:33 GMT
• Title: QCalEval: Benchmarking Vision-Language Models for Quantum Calibration Plot Understanding
• Abstract summary: We introduce QCalEval, the first VLM benchmark for quantum calibration plots.<n> 243 samples across 87 scenario types from 22 experiment families, spanning superconducting qubits and neutral atoms, were evaluated.<n>The best general-purpose zero-shot model reaches a mean score of 72.3, and many open-weight models degrade under multi-image in-context learning.
• Score: 37.18078731710843
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Quantum computing calibration depends on interpreting experimental data, and calibration plots provide the most universal human-readable representation for this task, yet no systematic evaluation exists of how well vision-language models (VLMs) interpret them. We introduce QCalEval, the first VLM benchmark for quantum calibration plots: 243 samples across 87 scenario types from 22 experiment families, spanning superconducting qubits and neutral atoms, evaluated on six question types in both zero-shot and in-context learning settings. The best general-purpose zero-shot model reaches a mean score of 72.3, and many open-weight models degrade under multi-image in-context learning, whereas frontier closed models improve substantially. A supervised fine-tuning ablation at the 9-billion-parameter scale shows that SFT improves zero-shot performance but cannot close the multimodal in-context learning gap. As a reference case study, we release NVIDIA Ising Calibration 1, an open-weight model based on Qwen3.5-35B-A3B that reaches 74.7 zero-shot average score.

Related papers

• When Does Few-Shot Prompting Help? A Systematic Empirical Study of Shot-Count Effects Across Model Scale, Architecture, and Output Parsing Robustness [0.0]
  Few-shot prompting, the practice of prepending a small number of input-output demonstration pairs to a query before presenting it to a large language model, is widely adopted in NLP.<n>Yet little systematic work investigates how shot count interacts with model scale, architecture, and output format compliance in classification performance.
  arXiv  Detail & Related papers  (2026-07-25T00:31:40Z)
• Iterative Visual Thinking: Teaching Vision-Language Models Spatial Self-Correction through Visual Feedback [0.023227405857540805]
  Vision-language models (VLMs) achieve strong singleshot spatial grounding, yet lack any mechanism to observe and correct their own predictions.<n>We propose Iterative Visual Thinking (IVT), a closed-loop framework in which the model predicts a bounding box, observes the prediction rendered on the image, and iteratively refines through visual feedback.<n>All training uses only 2,400 samples on a single GPU, demonstrating that spatial self-correction is a learnable capability that can be instilled at modest scale.
  arXiv  Detail & Related papers  (2026-06-11T10:27:29Z)
• Quantization Undoes Alignment: Bias Emergence in Compressed LLMs Across Models and Precision Levels [0.0]
  Large Language Models are routinely compressed via post-training quantization to reduce inference costs and memory footprint for cloud and edge deployment.<n>Existing studies typically compare only two conditions, rely on aggregate bias metrics, and evaluate a single model family.<n>We conduct a controlled empirical study of three instruction-tuned models at five precision levels (BF16 through 3-bit) on 12,148 BBQ bias benchmark items.
  arXiv  Detail & Related papers  (2026-05-02T05:41:47Z)
• MIRROR: A Hierarchical Benchmark for Metacognitive Calibration in Large Language Models [0.0]
  MIRROR is a benchmark that evaluates whether large language models can use self-knowledge to make better decisions.<n>We evaluate 16 models from 8 labs across approximately 250,000 evaluation instances.
  arXiv  Detail & Related papers  (2026-04-15T08:41:12Z)
• Mamba-3: Improved Sequence Modeling using State Space Principles [74.41028882099846]
  We introduce three core methodological improvements inspired by the state space model (SSM) viewpoint of linear models.<n>Together with architectural refinements, our Mamba-3 model achieves significant gains across retrieval, state-tracking, and downstream language modeling tasks.
  arXiv  Detail & Related papers  (2026-03-16T17:30:08Z)
• VB: Visibility Benchmark for Visibility and Perspective Reasoning in Images [0.0]
  We present VB, a benchmark that tests whether vision-language models can determine what is and is not visible in a photograph.<n>Items are organized into 100 families using a 2x2 design that crosses a minimal image edit with a minimal text edit.<n>We score models on confidence-aware accuracy with abstention (CAA), minimal-edit flip rate (MEFR), confidence-ranked selective prediction (SelRank), and second-order perspective reasoning.
  arXiv  Detail & Related papers  (2026-03-03T23:03:11Z)
• UrbanAlign: Post-hoc Semantic Calibration for VLM-Human Preference Alignment [17.430091762374744]
  We show that for subjective perception tasks, this alignment can be achieved without any model training.<n>We propose a training-free post-hoc concept-bottleneck pipeline consisting of three tightly coupled stages.
  arXiv  Detail & Related papers  (2026-02-23T02:24:55Z)
• Do Large Language Models Know What They Don't Know? Kalshibench: A New Benchmark for Evaluating Epistemic Calibration via Prediction Markets [0.0]
  A well-calibrated model should express confidence that matches its actual accuracy -- when it claims 80% confidence, it should be correct 80% of the time.<n>We introduce textbfKalshiBench, a benchmark of 300 prediction market questions from Kalshi, a CFTC-regulated exchange.<n>We evaluate five frontier models -- Claude Opus 4.5, GPT-5.2, DeepSeek-V3.2, Qwen3-235B, and Kimi-K2 -- and find textbfsystematic overconfidence across all models.
  arXiv  Detail & Related papers  (2025-12-17T23:23:06Z)
• Agentic World Modeling for 6G: Near-Real-Time Generative State-Space Reasoning [70.56067503630486]
  We argue that sixth-generation (6G) intelligence is not fluent token prediction but calibrated the capacity to imagine and choose.<n>We show that WM-MS3M cuts mean absolute error (MAE) by 1.69% versus MS3M with 32% fewer parameters and similar latency, and achieves 35-80% lower root mean squared error (RMSE) than attention/hybrid baselines with 2.3-4.1x faster inference.
  arXiv  Detail & Related papers  (2025-11-04T17:22:22Z)
• Goedel-Prover-V2: Scaling Formal Theorem Proving with Scaffolded Data Synthesis and Self-Correction [95.91743732150233]
  Goedel-Prover-V2, a series of open-source language models, set a new state-of-the-art in automated theorem proving.<n>We generate synthetic tasks of increasing difficulty to train the model to master increasingly complex theorems.<n>Goedel-Prover-V2-32B achieves 88.1% on MiniF2F at pass@32 in standard mode and 90.4% in self-correction mode.
  arXiv  Detail & Related papers  (2025-08-05T16:28:22Z)
• BridgeVLA: Input-Output Alignment for Efficient 3D Manipulation Learning with Vision-Language Models [37.699828966838986]
  BridgeVLA is a novel 3D VLA model that projects 3D inputs to multiple 2D images, ensuring input alignment with the VLM backbone.<n>It utilizes 2D heatmaps for action prediction, unifying the input and output spaces within a consistent 2D image space.<n>It is able to achieve a success rate of 96.8% on 10+ tasks with only 3 trajectories per task, highlighting its extraordinary sample efficiency.
  arXiv  Detail & Related papers  (2025-06-09T17:36:34Z)
• GWQ: Gradient-Aware Weight Quantization for Large Language Models [56.22507677736051]
  Large language models (LLMs) show impressive performance in solving complex language tasks.<n> compressing LLMs to low bits can enable to deploy on resource-constrained devices.<n>We propose gradient-aware weight quantization (GWQ), the first quantization approach for low-bit weight quantization.
  arXiv  Detail & Related papers  (2024-10-30T11:16:04Z)
• MM-Vet v2: A Challenging Benchmark to Evaluate Large Multimodal Models for Integrated Capabilities [146.4724093405187]
  We introduce MM-Vet v2, which includes a new "image-text sequence understanding" capability called "image-text sequence understanding"<n>Using MM-Vet v2 to benchmark large multimodal models, we found that Claude 3.5 Sonnet is the best model with a score of 71.8, slightly outperforming GPT-4o which scored 71.0.
  arXiv  Detail & Related papers  (2024-08-01T17:59:54Z)
• The Languini Kitchen: Enabling Language Modelling Research at Different Scales of Compute [66.84421705029624]
  We introduce an experimental protocol that enables model comparisons based on equivalent compute, measured in accelerator hours. We pre-process an existing large, diverse, and high-quality dataset of books that surpasses existing academic benchmarks in quality, diversity, and document length. This work also provides two baseline models: a feed-forward model derived from the GPT-2 architecture and a recurrent model in the form of a novel LSTM with ten-fold throughput.
  arXiv  Detail & Related papers  (2023-09-20T10:31:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.