OrdinalBench: A Benchmark Dataset for Diagnosing Generalization Limits in Ordinal Number Understanding of Vision-Language Models

• URL: http://arxiv.org/abs/2603.07786v1
• Date: Sun, 08 Mar 2026 20:06:45 GMT
• Title: OrdinalBench: A Benchmark Dataset for Diagnosing Generalization Limits in Ordinal Number Understanding of Vision-Language Models
• Abstract summary: Vision-Language Models (VLMs) have advanced across multimodal benchmarks but still show clear gaps in ordinal number understanding.<n>We present OrdinalBench, a diagnostic benchmark that standardizes ordinal number understanding as an evaluation task.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Vision-Language Models (VLMs) have advanced across multimodal benchmarks but still show clear gaps in ordinal number understanding, i.e., the ability to track relative positions and generalize to large indices. We present OrdinalBench, a diagnostic benchmark that standardizes ordinal number understanding as an evaluation task for VLMs. The core task is N-th object identification, defined by a starting reference and traversal rule. Task difficulty is controlled along three axes: (i) ordinal magnitude, from small numbers to extreme cases up to 300; (ii) arrangement complexity, from single loops to maze-like paths; and (iii) object count. The benchmark provides 39,000 question-answer pairs, each annotated with a ground-truth reasoning trajectory and balanced across difficulty levels for controlled large-scale testing. Beyond answer-only evaluation, our framework requires models to generate structured stepwise traces of the counting process and provides an open evaluation toolkit that measures both final accuracy and step-level path consistency. Zero-shot evaluations of GPT-5, Gemini 2.5 Flash Lite, Qwen2.5-VL, InternVL3.5, and Molmo reveal sharp degradation under large-ordinal and complex-path conditions, highlighting weak generalization despite strong scores on standard multimodal tasks. By framing ordinal number understanding as a core target, OrdinalBench provides a reproducible benchmark and diagnostic framework for developing VLMs with stronger sequential reasoning. All data and code are available at https://ordinalbench.github.io/

Related papers

• HoloCount: A Holistic Visual Counting Benchmark for MLLMs [17.08907754405721]
  HoloCount is a diagnostically rich benchmark structured around a three-level hierarchical taxonomy.<n>We evaluate over 20 state-of-the-art Multimodal Large Language Models (MLLMs)<n>Our findings offer a roadmap for developing more grounded and reliable multimodal systems.
  arXiv  Detail & Related papers  (2026-07-07T15:48:57Z)
• Count Anything at Any Granularity [55.70950689938894]
  KubriCount is the largest and most comprehensively annotated counting dataset to date.<n>We train a multi-grained counting model that jointly leverages text and visual exemplars as complementary target specifications.<n>HieraCount substantially improves multi-grained counting accuracy and generalizes robustly to challenging real-world scenarios.
  arXiv  Detail & Related papers  (2026-05-11T17:32:37Z)
• Bootstrapping MLLM for Weakly-Supervised Class-Agnostic Object Counting [59.37613121962146]
  We propose WS-COC, the first MLLM-driven weakly-supervised framework for class-agnostic object counting.<n> WS-COC matches or even surpasses many state-of-art fully-supervised methods while significantly reducing annotation costs.
  arXiv  Detail & Related papers  (2026-02-13T09:58:35Z)
• RankLLM: Weighted Ranking of LLMs by Quantifying Question Difficulty [102.02839046225468]
  RankLLM is a novel framework designed to quantify both question difficulty and model competency.<n>We evaluate 30 models on 35,550 questions across multiple domains.
  arXiv  Detail & Related papers  (2026-02-12T21:28:46Z)
• GamiBench: Evaluating Spatial Reasoning and 2D-to-3D Planning Capabilities of MLLMs with Origami Folding Tasks [2.037026133182214]
  We introduce GamiBench, a benchmark to evaluate spatial reasoning in large language models (MLLMs) through origami-inspired folding tasks.<n>GamiBench includes 186 regular and impossible 2D crease patterns paired with their corresponding 3D folded shapes.<n>GamiBench holistically evaluates the entire reasoning process--measuring cross-view consistency, physical feasibility through impossible-fold detection, and interpretation of intermediate folding steps.
  arXiv  Detail & Related papers  (2025-12-22T01:07:59Z)
• seqBench: A Tunable Benchmark to Quantify Sequential Reasoning Limits of LLMs [1.0519693622157462]
  We introduce seqBench, a benchmark for probing sequential reasoning limits in Large Language Models (LLMs)<n>We find that even top-performing models systematically fail on seqBench's structured reasoning tasks despite minimal search complexity.
  arXiv  Detail & Related papers  (2025-09-21T01:32:13Z)
• CompassVerifier: A Unified and Robust Verifier for LLMs Evaluation and Outcome Reward [50.97588334916863]
  We develop CompassVerifier, an accurate and robust lightweight verifier model for evaluation and outcome reward.<n>It demonstrates multi-domain competency spanning math, knowledge, and diverse reasoning tasks, with the capability to process various answer types.<n>We introduce VerifierBench benchmark comprising model outputs collected from multiple data sources, augmented through manual analysis of metaerror patterns to enhance CompassVerifier.
  arXiv  Detail & Related papers  (2025-08-05T17:55:24Z)
• Stronger Than You Think: Benchmarking Weak Supervision on Realistic Tasks [19.49705185032905]
  Weak supervision (WS) is a popular approach for label-efficient learning, leveraging diverse sources of noisy but inexpensive weak labels to automatically annotate training data.<n>Despite its wide usage, WS and its practical value are challenging to benchmark due to the many knobs in its setup.<n>We introduce a new benchmark, BOXWRENCH, designed to more accurately reflect real-world usages of WS.
  arXiv  Detail & Related papers  (2025-01-13T22:29:31Z)
• Benchmarking Large Language Models with Integer Sequence Generation Tasks [2.204499020600093]
  We present a benchmark designed to rigorously evaluate the capabilities of large language models (LLMs) in mathematical reasoning tasks.<n>The benchmark comprises integer sequence generation tasks sourced from the Online Encyclopedia of Sequences (OEIS)<n>Our evaluation includes leading models from OpenAI (including the specialized reasoning-focused o-series), Anthropic, Meta, and Google.
  arXiv  Detail & Related papers  (2024-11-07T02:05:43Z)
• BabelBench: An Omni Benchmark for Code-Driven Analysis of Multimodal and Multistructured Data [61.936320820180875]
  Large language models (LLMs) have become increasingly pivotal across various domains. BabelBench is an innovative benchmark framework that evaluates the proficiency of LLMs in managing multimodal multistructured data with code execution. Our experimental findings on BabelBench indicate that even cutting-edge models like ChatGPT 4 exhibit substantial room for improvement.
  arXiv  Detail & Related papers  (2024-10-01T15:11:24Z)
• MR-Ben: A Meta-Reasoning Benchmark for Evaluating System-2 Thinking in LLMs [55.20845457594977]
  Large language models (LLMs) have shown increasing capability in problem-solving and decision-making.<n>We present a process-based benchmark MR-Ben that demands a meta-reasoning skill.<n>Our meta-reasoning paradigm is especially suited for system-2 slow thinking.
  arXiv  Detail & Related papers  (2024-06-20T03:50:23Z)
• InfiMM-Eval: Complex Open-Ended Reasoning Evaluation For Multi-Modal Large Language Models [50.03163753638256]
  Multi-modal Large Language Models (MLLMs) are increasingly prominent in the field of artificial intelligence. Our benchmark comprises three key reasoning categories: deductive, abductive, and analogical reasoning. We evaluate a selection of representative MLLMs using this rigorously developed open-ended multi-step elaborate reasoning benchmark.
  arXiv  Detail & Related papers  (2023-11-20T07:06:31Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.