SABRE: Scalable and Automated Benchmarking of VLMs under Stress

• URL: http://arxiv.org/abs/2608.07435v1
• Date: Fri, 07 Aug 2026 17:21:04 GMT
• Title: SABRE: Scalable and Automated Benchmarking of VLMs under Stress
• Abstract summary: Vision-language models (VLMs) are improving rapidly, but benchmark development lags behind.<n>We present SABRE, a scalable, automated pipeline that converts a Test Primer into structured specifications.<n>We instantiate SABRE-Prior to test whether VLMs follow visual evidence instead of relying on world priors.
• Score: 10.315515647818009
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Vision-language models (VLMs) are improving rapidly, but benchmark development lags behind, making weaknesses hard to identify. Building stress tests is costly: samples must satisfy controlled conditions, remain answerable, and challenge current models. We present SABRE, a scalable, automated pipeline that converts a Test Primer (a Markdown Task Design with Data Schema) into structured specifications, generated or edited images, and question-answer pairs. Automated filtering removes candidates solved by a Filtering VLM, while human review verifies candidate validity and supports annotation correction and localized image repair. We instantiate SABRE-Prior to test whether VLMs follow visual evidence instead of relying on world priors -- learned expectations about familiar objects and scenes. Its 600 images and 1,000 questions span Context (unexpected entities in familiar scenes), Texture (counterfactual materials), Attribute (noncanonical component counts), and Language Elicitation (answers suggested by language but unsupported by the image). Across six VLMs, macro-average accuracy ranges from 17.8% to 31.3% (22.6% mean). A real-image Attribute control is comparably difficult for the Filtering VLM. SABRE-Counting and SABRE-Spatial pilots show that the workflow supports other stress-test settings. These results establish SABRE as a reusable framework for constructing and refreshing VLM stress tests rather than a single fixed benchmark.

Related papers

• Harnessing Image Question Dependence for Better VLM Test-time Reinforcement Learning [66.13367818892503]
  We propose TTIQ, a test-time reinforcement learning framework that harnesses image-question dependence for better vlm adaptation.<n> TTIQ teacher-forces each sampled response under the original image-question pair and its image- and question-ablated variants.<n>It combines image and question dependence with confidence to construct a response-level reward that favors jointly grounded responses.
  arXiv  Detail & Related papers  (2026-09-09T22:54:15Z)
• Evaluating VLMs for Autonomous Agent-Driven Geometry Clipping Detection in Video Game QA [6.404338288187941]
  We study the use of Vision-Language Models (VLMs) for anomaly detection in an agent-driven game Quality Assurance pipeline.<n>We benchmark six recent VLMs under a zero-shot prompting setting and analyse their sensitivity to four prompt variants.<n>Our results show that while the VLMs can capture visual cues associated with geometry clipping, they all produce substantial false positives on visually ambiguous frames.
  arXiv  Detail & Related papers  (2026-07-28T16:10:47Z)
• What to Test Next: Interpretable Coverage Gap Discovery in Driving VLMs [52.50210189669399]
  Driving vision-language models (VLMs) must accurately understand scenes across diverse conditions defined by Operational Design Domains (ODDs)<n>We propose SliceScorer, a deterministic scoring rule for missing-slice recommendation.<n>We show that SliceNav surfaces high-risk coverage gaps more effectively than prior slice-discovery methods.
  arXiv  Detail & Related papers  (2026-06-01T03:18:01Z)
• Employing Vision-Language Models for Face Image Quality Assessment [0.023141219541346198]
  Face Image Quality Assessment (FIQA) is a crucial control step in biometric pipelines.<n>State-of-the-art FIQA methods achieve high utility but typically operate as "black boxes"<n>We investigate the potential of off-the-shelf Vision-Language Models (VLMs) to bridge this gap by performing FIQA in a zero-shot setting.
  arXiv  Detail & Related papers  (2026-05-17T14:57:52Z)
• Test-Time Hinting for Black-Box Vision-Language Models [11.005886421208709]
  Test-Time Hinting improves VLM performance via a single VLM call and requiring only black-box API access.<n>We show that Test-Time Hinting improves the accuracy of multiple closed-weight VLMs on natural-image VQA benchmarks.
  arXiv  Detail & Related papers  (2026-05-13T14:35:22Z)
• Stress Tests REVEAL Fragile Temporal and Visual Grounding in Video-Language Models [18.243585941034116]
  Video-Language Models (VidLMs) robustly account for video content, temporal sequence, and motion.<n>We introduce REVEAL, a diagnostic benchmark that probes fundamental weaknesses of contemporary Vids.<n>We find that these models confidently describe reversed scenes as forward, answer questions while neglecting video content, agree with false claims, struggle with basic camera motion, and fail to aggregate temporal scalable information.
  arXiv  Detail & Related papers  (2026-02-11T17:39:14Z)
• FineGRAIN: Evaluating Failure Modes of Text-to-Image Models with Vision Language Model Judges [85.24983823102262]
  We propose a structured methodology for evaluating text-to-image (T2I) models and vision language models (VLMs)<n>We test whether VLMs can identify 27 specific failure modes in the images generated by T2I models conditioned on challenging prompts.<n>Our findings suggest that current metrics are insufficient to capture these nuanced errors.
  arXiv  Detail & Related papers  (2025-12-01T19:46:03Z)
• Lost in Translation and Noise: A Deep Dive into the Failure Modes of VLMs on Real-World Tables [0.1749935196721634]
  Existing datasets for QA, such as WikiTableQuestions and FinQA, are overwhelmingly monolingual (English)<n>MirageTVQA provides a benchmark for measuring and driving progress towards more robust VLM models for table reasoning.
  arXiv  Detail & Related papers  (2025-11-21T13:32:56Z)
• CARES: Context-Aware Resolution Selector for VLMs [29.734101330721263]
  Large vision-language models (VLMs) commonly process images at native or high resolution to remain effective across tasks.<n>We introduce emphCARES, a lightweight preprocessing module that predicts the emphminimal sufficient input resolution.<n> CARES preserves task performance while reducing compute by up to 80%.
  arXiv  Detail & Related papers  (2025-10-22T11:44:31Z)
• DySS: Dynamic Queries and State-Space Learning for Efficient 3D Object Detection from Multi-Camera Videos [53.52664872583893]
  Camera-based 3D object detection in Bird's Eye View (BEV) is one of the most important perception tasks in autonomous driving.<n>We propose DySS, a novel method that employs state-space learning and dynamic queries.<n>Our proposed DySS achieves both superior detection performance and efficient inference.
  arXiv  Detail & Related papers  (2025-06-11T23:49:56Z)
• Toward Robust Hyper-Detailed Image Captioning: A Multiagent Approach and Dual Evaluation Metrics for Factuality and Coverage [50.84150600032693]
  Multimodal large language models (MLLMs) excel at generating highly detailed captions but often produce hallucinations.<n>We propose a multiagent approach that leverages LLM-MLLM collaboration to correct given captions.<n>Our proposed method significantly enhances the factual accuracy of captions, even improving those generated by GPT-4V.
  arXiv  Detail & Related papers  (2024-12-20T01:37:22Z)
• Trust but Verify: Programmatic VLM Evaluation in the Wild [62.14071929143684]
  Programmatic VLM Evaluation (PROVE) is a new benchmarking paradigm for evaluating VLM responses to open-ended queries. We benchmark the helpfulness-truthfulness trade-offs of a range ofVLMs on PROVE, finding that very few are in-fact able to achieve a good balance between the two.
  arXiv  Detail & Related papers  (2024-10-17T01:19:18Z)
• Test-Time Self-Adaptive Small Language Models for Question Answering [63.91013329169796]
  We show and investigate the capabilities of smaller self-adaptive LMs, only with unlabeled test data. Our proposed self-adaption strategy demonstrates significant performance improvements on benchmark QA datasets.
  arXiv  Detail & Related papers  (2023-10-20T06:49:32Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.