Enginuity: A Dataset and Benchmark for Vision-Language Understanding of Engineering Diagrams

• URL: http://arxiv.org/abs/2606.03410v1
• Date: Tue, 02 Jun 2026 09:54:03 GMT
• Title: Enginuity: A Dataset and Benchmark for Vision-Language Understanding of Engineering Diagrams
• Abstract summary: Enginuity is the first open dataset and benchmark for evaluating visual language models on engineering diagrams.<n>We define two tasks over a corpus of U.S. military service and repair manuals: structured parts-table extraction and free-form visual diagram question answering.<n>We evaluate four frontier VLMs under zero-shot and chain-of-thought prompting.
• Score: 8.001015579487921
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Engineering diagrams pose a distinct challenge for vision-language models: unlike natural images or general documents, they encode information through dense spatial layouts, domain-specific symbols, and cross-references between visual callouts and structured parts tables. Despite their centrality to service, repair, and design workflows, there is no public benchmark for measuring VLM capabilities in this domain; existing datasets primarily focus on flowcharts, scientific figures, or business documents. To address this gap, we introduce Enginuity, the first open dataset and benchmark for evaluating VLMs on complex engineering diagrams. We define two tasks over a corpus of U.S. military service and repair manuals: structured parts-table extraction (Task 1) and free-form visual diagram question answering (VQA)(Task 2) for benchmarking. We evaluate four frontier VLMs (GPT-5.2 Chat, Claude Opus 4.7, Gemma 4, Qwen3-VL-32B-Instruct) under zero-shot and chain-of-thought prompting. On Task 1, models reach Recall@all of 0.61-0.87 but Token F1pen of only 0.03-0.18, exposing a systematic gap between part identification and description fidelity. Task 2 reveals a consistent factual-reasoning gap across all models. A supporting analysis shows that token-overlap metrics under-report model capability on technical descriptions by 2-6x relative to semantic similarity, motivating LLM-as-judge calibration for domain-specific evaluation. We release the dataset, annotations, evaluation harness, and per-sample model outputs to support a reproducible study of VLM capability on engineering content.

Related papers

• Visual Information Extraction from Documents via Classification-Guided Large Vision-Language Models [7.176656402971074]
  We propose a classification-guided large vision-language model (LVLM) framework for multi-type VIE that achieves high accuracy with minimal supervision.<n>The framework offers an efficient, scalable solution for complex document understanding in office automation.
  arXiv  Detail & Related papers  (2026-07-22T03:27:50Z)
• Dual-Pathway Geometry-Aware MLLM for Spatial Intelligence [63.11502936651325]
  We propose GAMSI, a dual-pathway Geometry-Aware MLLM for Spatial Intelligence.<n>It takes only RGB images as input while internalizing both forms of geometric prior within a unified autoregressive backbone.<n>GAMSI achieves state-of-the-art performance on seven spatial intelligence benchmarks.
  arXiv  Detail & Related papers  (2026-05-25T01:33:19Z)
• Can Vision Language Models Assess Graphic Design Aesthetics? A Benchmark, Evaluation, and Dataset Perspective [43.433154981521234]
  Assessing the aesthetic quality of graphic design is central to visual communication, yet remains underexplored in vision language models (VLMs)<n>In this work, we introduce AesEval-Bench, a comprehensive benchmark spanning four dimensions, twelve indicators, and three fully quantifiable tasks.
  arXiv  Detail & Related papers  (2026-03-01T12:55:39Z)
• A Multi-Stage Hybrid Framework for Automated Interpretation of Multi-View Engineering Drawings Using Vision Language Model [0.044780965967547055]
  This paper proposes a three-stage hybrid framework for the automated interpretation of 2D multi-view engineering drawings.<n>YOLOv11-det performs layout segmentation to localize key regions such as views, title blocks, and notes.<n>The second stage uses YOLOv11-obb for CAD-aware, fine-grained detection of annotations, including measures, GD&T symbols, and surface roughness indicators.<n>The third stage employs two Donut-based, OCR-free VLMs for semantic content parsing.
  arXiv  Detail & Related papers  (2025-10-23T09:07:31Z)
• Vision-G1: Towards General Vision Language Reasoning with Multi-Domain Data Curation [64.23194519770897]
  We build a comprehensive RL-ready visual reasoning dataset from 46 data sources across 8 dimensions.<n>We propose an influence function based data selection and difficulty based filtering strategy to identify high-quality training samples from this dataset.<n>We train the VLM, referred to as Vision-G1, using multi-round RL with a data curriculum to iteratively improve its visual reasoning capabilities.
  arXiv  Detail & Related papers  (2025-08-18T07:24:33Z)
• From Drawings to Decisions: A Hybrid Vision-Language Framework for Parsing 2D Engineering Drawings into Structured Manufacturing Knowledge [0.352650106994433]
  Key information from 2D engineering drawings is essential for advancing digital manufacturing.<n>Manual extraction is slow and labor-intensive, while generic OCR models often fail due to complex layouts, engineering symbols, and rotated text.<n>We propose a hybrid vision-language framework that integrates a rotation-aware object detection model (YOLOv11-obb) with a transformer-based vision-language model (VLM)
  arXiv  Detail & Related papers  (2025-06-20T17:10:01Z)
• Granite Vision: a lightweight, open-source multimodal model for enterprise Intelligence [88.74800617923083]
  We introduce Granite Vision, a lightweight large language model with vision capabilities.<n>Our model is trained on a comprehensive instruction-following dataset.<n> Granite Vision achieves strong results in standard benchmarks related to visual document understanding.
  arXiv  Detail & Related papers  (2025-02-14T05:36:32Z)
• Learning to Rank Pre-trained Vision-Language Models for Downstream Tasks [41.488394198111976]
  Vision language models (VLMs) like CLIP show stellar zero-shot capability on classification benchmarks.<n> selecting the VLM with the highest performance on the unlabeled downstream task is non-trivial.<n>This paper introduces the problem of textbfunsupervised vision-language model selection, where only unsupervised downstream datasets are available.
  arXiv  Detail & Related papers  (2024-12-30T03:26:53Z)
• The All-Seeing Project V2: Towards General Relation Comprehension of the Open World [58.40101895719467]
  We present the All-Seeing Project V2, a new model and dataset designed for understanding object relations in images. We propose the All-Seeing Model V2 that integrates the formulation of text generation, object localization, and relation comprehension into a relation conversation task. Our model excels not only in perceiving and recognizing all objects within the image but also in grasping the intricate relation graph between them.
  arXiv  Detail & Related papers  (2024-02-29T18:59:17Z)
• Prismatic VLMs: Investigating the Design Space of Visually-Conditioned Language Models [73.40350756742231]
  Visually-conditioned language models (VLMs) have seen growing adoption in applications such as visual dialogue, scene understanding, and robotic task planning. Despite the volume of new releases, key design decisions around image preprocessing, architecture, and optimization are under-explored.
  arXiv  Detail & Related papers  (2024-02-12T18:21:14Z)
• Exploiting Contextual Target Attributes for Target Sentiment Classification [53.30511968323911]
  Existing PTLM-based models for TSC can be categorized into two groups: 1) fine-tuning-based models that adopt PTLM as the context encoder; 2) prompting-based models that transfer the classification task to the text/word generation task. We present a new perspective of leveraging PTLM for TSC: simultaneously leveraging the merits of both language modeling and explicit target-context interactions via contextual target attributes.
  arXiv  Detail & Related papers  (2023-12-21T11:45:28Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.