SAFE-Cascade: Cost-Adaptive Vision-Language Routing for Chart Question Answering

• URL: http://arxiv.org/abs/2606.19646v1
• Date: Wed, 17 Jun 2026 23:00:17 GMT
• Title: SAFE-Cascade: Cost-Adaptive Vision-Language Routing for Chart Question Answering
• Abstract summary: We demonstrate SAFE- Cascade, an interactive system for cost-adaptive chart question answering.<n>Given a chart image and a natural-language question, SAFE- Cascade first extracts chart text with OCR.<n>It obtains a provisional answer from a text-only language model, and then uses a learned router to decide whether to accept the text answer or escalate to a VLM.
• Score: 16.639536340934715
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Vision-language models (VLMs) are powerful for chart question answering, but invoking a VLM for every query can be unnecessarily expensive when many questions are answerable from OCR text and lightweight language reasoning. We demonstrate SAFE-Cascade, an interactive system for cost-adaptive chart question answering. Given a chart image and a natural-language question, SAFE-Cascade first extracts chart text with OCR, obtains a provisional answer from a text-only language model, and then uses a learned router to decide whether to accept the text answer or escalate to a VLM. The demo exposes this decision process to users: OCR evidence, text-only answer, routing probability, escalation decision, final answer, estimated cost, and estimated latency are shown side by side. SAFE-Cascade is designed as a transparent interface for understanding when visual grounding is actually needed. Users can upload or select charts, ask questions, inspect the evidence used by each pathway, compare text-only and VLM answers, and adjust the escalation threshold to explore the accuracy-cost frontier. The system is implemented with Azure Document Intelligence for OCR, gpt-5-mini as the text-only model, gemini-2.5-flash-image as the VLM, and a Random Forest router trained on inference-time features. On a held-out ChartQA test split of 375 examples from a 2,500-example experiment, SAFE-Cascade achieves 69.1% unified accuracy with 73.1% VLM invocation, compared with 67.7% accuracy and 100% VLM invocation for the full-VLM baseline. The observed +1.4 percentage-point difference is statistically uncertain, so we interpret SAFE-Cascade as matching full-VLM performance while reducing VLM calls by 26.9% and estimated cost by 9.3%. The demonstration shows how selective modality routing can make multimodal knowledge systems more transparent, tunable, and cost-aware.

Related papers

• SABRE: Scalable and Automated Benchmarking of VLMs under Stress [10.315515647818009]
  Vision-language models (VLMs) are improving rapidly, but benchmark development lags behind.<n>We present SABRE, a scalable, automated pipeline that converts a Test Primer into structured specifications.<n>We instantiate SABRE-Prior to test whether VLMs follow visual evidence instead of relying on world priors.
  arXiv  Detail & Related papers  (2026-08-07T17:21:04Z)
• Detect Before You Leap: Mirage Detection in Vision-Language Models [2.6212127510234797]
  Vision-language models (VLMs) can produce confident visual answers even when the required visual evidence is missing, blank, or unrelated to the question.<n>We study pre-release mirage detection: given an image-question pair, the goal is to determine whether a VLM should answer or abstain.<n>We propose Text-Conditioned Layer-wise Internal Alignment (TC-LIA), a model-agnostic method that probes patch-token representations across the layers of a CLIP ViT-H/14 vision encoder.
  arXiv  Detail & Related papers  (2026-05-29T23:51:35Z)
• Index Light, Reason Deep: Deferred Visual Ingestion for Visual-Dense Document Question Answering [3.0028612998967894]
  This paper proposes a demand-side ingestion strategy for multimodal document question answering.<n>It achieves comparable overall accuracy at zero ingestion VLM cost.<n>It also supports interactive refinement and progressive caching, transforming the "QA accuracy" problem into a "page localization" problem.
  arXiv  Detail & Related papers  (2026-02-15T14:23:50Z)
• Retrieval-Based Interleaved Visual Chain-of-Thought in Real-World Driving Scenarios [69.00444996464662]
  We propose RIV-CoT, a Retrieval-Based Interleaved Visual Chain-of-Thought method that enables vision-language models to reason using visual crops corresponding to relevant entities.<n>Our experiments demonstrate that RIV-CoT improves answer accuracy by 3.1% and reasoning accuracy by 4.6% over vanilla CoT prompting.
  arXiv  Detail & Related papers  (2025-01-08T18:31:16Z)
• Right this way: Can VLMs Guide Us to See More to Answer Questions? [11.693356269848517]
  In question-answering scenarios, humans assess whether the available information is sufficient and seek additional information if necessary. In contrast, Vision Language Models (VLMs) typically generate direct, one-shot responses without evaluating the sufficiency of the information. This study demonstrates the potential to narrow the gap between information assessment and acquisition in VLMs, bringing their performance closer to humans.
  arXiv  Detail & Related papers  (2024-11-01T06:43:54Z)
• Trust but Verify: Programmatic VLM Evaluation in the Wild [62.14071929143684]
  Programmatic VLM Evaluation (PROVE) is a new benchmarking paradigm for evaluating VLM responses to open-ended queries. We benchmark the helpfulness-truthfulness trade-offs of a range ofVLMs on PROVE, finding that very few are in-fact able to achieve a good balance between the two.
  arXiv  Detail & Related papers  (2024-10-17T01:19:18Z)
• Rephrase, Augment, Reason: Visual Grounding of Questions for Vision-Language Models [59.05769810380928]
  Rephrase, Augment and Reason (RepARe) is a gradient-free framework that extracts salient details about the image using the underlying vision-language model. We show that RepARe can result in a 3.85% (absolute) increase in zero-shot accuracy on VQAv2, 6.41%, and 7.94% points increase on A-OKVQA, and VizWiz respectively.
  arXiv  Detail & Related papers  (2023-10-09T16:57:57Z)
• Improving Selective Visual Question Answering by Learning from Your Peers [74.20167944693424]
  Visual Question Answering (VQA) models can have difficulties abstaining from answering when they are wrong. We propose Learning from Your Peers (LYP) approach for training multimodal selection functions for making abstention decisions. Our approach uses predictions from models trained on distinct subsets of the training data as targets for optimizing a Selective VQA model.
  arXiv  Detail & Related papers  (2023-06-14T21:22:01Z)
• From Images to Textual Prompts: Zero-shot VQA with Frozen Large Language Models [111.42052290293965]
  Large language models (LLMs) have demonstrated excellent zero-shot generalization to new language tasks. End-to-end training on vision and language data may bridge the disconnections, but is inflexible and computationally expensive. We propose emphImg2Prompt, a plug-and-play module that provides the prompts that can bridge the aforementioned modality and task disconnections.
  arXiv  Detail & Related papers  (2022-12-21T08:39:36Z)
• Reliable Visual Question Answering: Abstain Rather Than Answer Incorrectly [100.60560477391732]
  We promote a problem formulation for reliable visual question answering (VQA) We analyze both their coverage, the portion of questions answered, and risk, the error on that portion. We find that although the best performing models achieve over 71% accuracy on the VQA v2 dataset, introducing the option to abstain limits them to answering less than 8% of the questions to achieve a low risk of error (i.e., 1%) This motivates us to utilize a multimodal selection function to directly estimate the correctness of the predicted answers, which we show can triple the coverage from, for example, 5.0% to 16.7% at
  arXiv  Detail & Related papers  (2022-04-28T16:51:27Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.