DistortBench: Benchmarking Vision Language Models on Image Distortion Identification

• URL: http://arxiv.org/abs/2604.19966v1
• Date: Tue, 21 Apr 2026 20:20:59 GMT
• Title: DistortBench: Benchmarking Vision Language Models on Image Distortion Identification
• Abstract summary: Vision-language models (VLMs) are increasingly used in settings where sensitivity to low-level image degradations matters.<n>We present DistortBench, a diagnostic benchmark for no-reference distortion perception in VLMs.
• Score: 1.1609102090344634
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Vision-language models (VLMs) are increasingly used in settings where sensitivity to low-level image degradations matters, including content moderation, image restoration, and quality monitoring. Yet their ability to recognize distortion type and severity remains poorly understood. We present DistortBench, a diagnostic benchmark for no-reference distortion perception in VLMs. DistortBench contains 13,500 four-choice questions covering 27 distortion types, six perceptual categories, and five severity levels: 25 distortions inherit KADID-10k calibrations, while two added rotation distortions use monotonic angle-based levels. We evaluate 18 VLMs, including 17 open-weight models from five families and one proprietary model. Despite strong performance on high-level vision-language tasks, the best model reaches only 61.9% accuracy, just below the human majority-vote baseline of 65.7% (average individual: 60.2%), indicating that low-level perceptual understanding remains a major weakness of current VLMs. Our analysis further reveals weak and non-monotonic scaling with model size, performance drops in most base--thinking pairs, and distinct severity-response patterns across model families. We hope DistortBench will serve as a useful benchmark for measuring and improving low-level visual perception in VLMs.

Related papers

• VDiff-Bench: A Challenging Benchmark for Fine-Grained Image Difference Identification [53.25222629244709]
  VDiff-Bench is a benchmark for fine-grained Image Difference Identification.<n>Each question corresponds to two image inputs with 4 choices: the true difference, two hard negative descriptions, and a "no difference" distractor.<n>Experiments with 11 state-of-the-art open- and closed-source MLLMs show that fine-grained visual comparison remains brittle.
  arXiv  Detail & Related papers  (2026-09-05T20:20:24Z)
• The Visual Insensitivity Gap: Diagnosing When Vision-Language Models Fail to Use Visual Evidence [1.7844382164707184]
  We name this phenomenon the Visual Insensitivity Gap and quantify it with a per-sample Visual Sensitivity Index (VSI)<n>We show this assumption fails on 40%--97% of samples across six VLMs and three perceptual benchmarks.
  arXiv  Detail & Related papers  (2026-09-01T08:01:57Z)
• SycoPhantasy: Quantifying Sycophancy and Hallucination in Small Open Weight VLMs for Vision-Language Scoring of Fantasy Characters [2.4851820343103035]
  Vision-language models (VLMs) are increasingly deployed as evaluators in tasks requiring nuanced image understanding.<n>We investigate whether small, open-weight VLMs exhibit emphsycophantic behavior when evaluating image-text alignment.
  arXiv  Detail & Related papers  (2026-04-27T11:39:18Z)
• VLM-RobustBench: A Comprehensive Benchmark for Robustness of Vision-Language Models [64.56065206447788]
  Vision-language models (VLMs) achieve strong performance on standard, high-quality datasets.<n>We present VLM-RobustBench, a benchmark spanning 49 augmentation types across noise, blur, weather, digital, and geometrics.<n>We show that low-severity spatial perturbations often degrade performance more than visually severe photometric corruptions.
  arXiv  Detail & Related papers  (2026-03-06T10:58:02Z)
• Investigate the Low-level Visual Perception in Vision-Language based Image Quality Assessment [7.969076042774561]
  We introduce a low-level distortion perception task that requires models to classify specific distortion types.<n>Our analysis shows that although MLLMs are structurally capable of representing such distortions, they tend to overfit training templates.<n>We show that improving the alignment of the vision encoder dramatically enhances distortion recognition accuracy, increasing it from 14.92% to 84.43%.
  arXiv  Detail & Related papers  (2025-12-10T12:06:47Z)
• DIQ-H: Evaluating Hallucination Persistence in VLMs Under Temporal Visual Degradation [0.7874708385247353]
  We introduce DIQ-H, the first benchmark for evaluating VLM robustness under dynamic visual degradation in temporal sequences.<n>DIQ-H applies physics-based corruptions including motion blur, sensor noise, and compression artifacts, and measures hallucination persistence, error recovery, and temporal consistency.<n>To enable scalable annotation, we propose Uncertainty-Guided Iterative Refinement (UIR), which generates reliable pseudo-ground-truth.
  arXiv  Detail & Related papers  (2025-12-03T17:22:29Z)
• VLA Models Are More Generalizable Than You Think: Revisiting Physical and Spatial Modeling [60.341503853471494]
  We show that vision-language-action models degrade sharply under novel camera viewpoints and visual perturbations.<n>We propose a one-shot adaptation framework that recalibrates visual representations through lightweight, learnable updates.
  arXiv  Detail & Related papers  (2025-12-02T16:16:13Z)
• COREVQA: A Crowd Observation and Reasoning Entailment Visual Question Answering Benchmark [3.5018278981067685]
  COREVQA (Crowd Observations and Reasoning Entailment) is a benchmark of 5608 image and synthetically generated true/false statement pairs.<n>Our results show that even the top-performing VLMs achieve accuracy below 80%, with other models performing substantially worse.
  arXiv  Detail & Related papers  (2025-07-17T04:47:47Z)
• SoTA with Less: MCTS-Guided Sample Selection for Data-Efficient Visual Reasoning Self-Improvement [100.85923086072204]
  We introduce ThinkLite-VL, a family of visual reasoning models that achieve state-of-the-art (SoTA) performance using an order of magnitude fewer training samples.<n>We use Monte Carlo Tree Search (MCTS) to measure sample difficulty via the number of reasoning iterations a vision-language model (VLM) requires to solve each instance.<n>ThinkLite-VL-7B and ThinkLite-VL-72B significantly outperform their respective base models across eight visual reasoning benchmarks.
  arXiv  Detail & Related papers  (2025-04-10T17:49:05Z)
• Benchmark Granularity and Model Robustness for Image-Text Retrieval [44.045767657945895]
  We show how dataset granularity and query perturbations affect retrieval performance and robustness.<n>We show that richer captions consistently enhance retrieval, especially in text-to-image tasks.<n>Our results highlight variation in model robustness and a dataset-dependent relationship between caption granularity and sensitivity perturbation.
  arXiv  Detail & Related papers  (2024-07-21T18:08:44Z)
• HallusionBench: An Advanced Diagnostic Suite for Entangled Language Hallucination and Visual Illusion in Large Vision-Language Models [69.52245481329899]
  We introduce HallusionBench, a benchmark for the evaluation of image-context reasoning. The benchmark comprises 346 images paired with 1129 questions, all meticulously crafted by human experts. In our evaluation on HallusionBench, we benchmarked 15 different models, highlighting a 31.42% question-pair accuracy achieved by the state-of-the-art GPT-4V.
  arXiv  Detail & Related papers  (2023-10-23T04:49:09Z)
• Improving Visual Grounding by Encouraging Consistent Gradient-based Explanations [58.442103936918805]
  We show that Attention Mask Consistency produces superior visual grounding results than previous methods. AMC is effective, easy to implement, and is general as it can be adopted by any vision-language model.
  arXiv  Detail & Related papers  (2022-06-30T17:55:12Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.