FORGE: Fine-grained Multimodal Evaluation for Manufacturing Scenarios

• URL: http://arxiv.org/abs/2604.07413v2
• Date: Sun, 12 Apr 2026 03:02:53 GMT
• Title: FORGE: Fine-grained Multimodal Evaluation for Manufacturing Scenarios
• Abstract summary: The manufacturing sector is increasingly adopting Multimodal Large Language Models (MLLMs) to transition from simple perception to autonomous execution.<n>Progress is hindered by data scarcity and a lack of fine-grained domain semantics in existing datasets.<n>We first construct a high-quality dataset that combines realworld 2D images and 3D point clouds, annotated with fine-grained domain semantics.<n>We then evaluate 18 state-of-the-art MLLMs across three manufacturing tasks, namely verification, structural surface inspection, and assembly verification, revealing significant performance gaps.
• Score: 58.34124792457706
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The manufacturing sector is increasingly adopting Multimodal Large Language Models (MLLMs) to transition from simple perception to autonomous execution, yet current evaluations fail to reflect the rigorous demands of real-world manufacturing environments. Progress is hindered by data scarcity and a lack of fine-grained domain semantics in existing datasets. To bridge this gap, we introduce FORGE. Wefirst construct a high-quality multimodal dataset that combines real-world 2D images and 3D point clouds, annotated with fine-grained domain semantics (e.g., exact model numbers). We then evaluate 18 state-of-the-art MLLMs across three manufacturing tasks, namely workpiece verification, structural surface inspection, and assembly verification, revealing significant performance gaps. Counter to conventional understanding, the bottleneck analysis shows that visual grounding is not the primary limiting factor. Instead, insufficient domain-specific knowledge is the key bottleneck, setting a clear direction for future research. Beyond evaluation, we show that our structured annotations can serve as an actionable training resource: supervised fine-tuning of a compact 3B-parameter model on our data yields up to 90.8% relative improvement in accuracy on held-out manufacturing scenarios, providing preliminary evidence for a practical pathway toward domain-adapted manufacturing MLLMs. The code and datasets are available at https://ai4manufacturing.github.io/forge-web.

Related papers

• MOON3.0: Reasoning-aware Multimodal Representation Learning for E-commerce Product Understanding [13.985610562698348]
  We argue that leveraging the reasoning capabilities of MLLMs to explicitly model fine-grained product attributes holds significant potential.<n>We propose MOON3.0, the first reasoning-aware MLLM-based model for product representation learning.
  arXiv  Detail & Related papers  (2026-04-01T05:55:25Z)
• LLM-Driven 3D Scene Generation of Agricultural Simulation Environments [1.002902747701998]
  Large Language Models (LLMs) for 3D scene generation show promise but often lack domain-specific reasoning, verification mechanisms, and modular design.<n>This paper investigates the use of LLMs to generate agricultural synthetic simulation environments from natural language prompts.<n>A modular multi-LLM pipeline was developed, integrating 3D asset retrieval, domain knowledge injection, and code generation for the Unreal rendering engine.
  arXiv  Detail & Related papers  (2026-02-12T08:33:01Z)
• Follow-Your-Instruction: A Comprehensive MLLM Agent for World Data Synthesis [44.66179436245703]
  Follow-Your-Instruction is a framework for automatically synthesizing high-quality 2D, 3D, and 4D data.<n>It constructs 3D layouts, and leverages Vision-Language Models (VLMs) for semantic refinement.<n>We evaluate the quality of the generated data through comprehensive experiments on the 2D, 3D, and 4D generative tasks.
  arXiv  Detail & Related papers  (2025-08-07T17:12:54Z)
• Topology-Aware Modeling for Unsupervised Simulation-to-Reality Point Cloud Recognition [63.55828203989405]
  We introduce a novel Topology-Aware Modeling (TAM) framework for Sim2Real UDA on object point clouds.<n>Our approach mitigates the domain gap by leveraging global spatial topology, characterized by low-level, high-frequency 3D structures.<n>We propose an advanced self-training strategy that combines cross-domain contrastive learning with self-training.
  arXiv  Detail & Related papers  (2025-06-26T11:53:59Z)
• Mitigating Forgetting in LLM Fine-Tuning via Low-Perplexity Token Learning [65.23593936798662]
  We show that fine-tuning with LLM-generated data improves target task performance and reduces non-target task degradation.<n>This is the first work to provide an empirical explanation based on token perplexity reduction to mitigate catastrophic forgetting in LLMs after fine-tuning.
  arXiv  Detail & Related papers  (2025-01-24T08:18:56Z)
• Web-Scale Visual Entity Recognition: An LLM-Driven Data Approach [56.55633052479446]
  Web-scale visual entity recognition presents significant challenges due to the lack of clean, large-scale training data. We propose a novel methodology to curate such a dataset, leveraging a multimodal large language model (LLM) for label verification, metadata generation, and rationale explanation. Experiments demonstrate that models trained on this automatically curated data achieve state-of-the-art performance on web-scale visual entity recognition tasks.
  arXiv  Detail & Related papers  (2024-10-31T06:55:24Z)
• 3DBench: A Scalable 3D Benchmark and Instruction-Tuning Dataset [13.808860456901204]
  We introduce a scalable 3D benchmark, accompanied by a large-scale instruction-tuning dataset known as 3DBench. Specifically, we establish the benchmark that spans a wide range of spatial and semantic scales, from object-level to scene-level. We present a rigorous pipeline for automatically constructing scalable 3D instruction-tuning datasets, covering 10 diverse multi-modal tasks with more than 0.23 million QA pairs generated in total.
  arXiv  Detail & Related papers  (2024-04-23T02:06:10Z)
• Self-supervised Feature Adaptation for 3D Industrial Anomaly Detection [59.41026558455904]
  We focus on multi-modal anomaly detection. Specifically, we investigate early multi-modal approaches that attempted to utilize models pre-trained on large-scale visual datasets. We propose a Local-to-global Self-supervised Feature Adaptation (LSFA) method to finetune the adaptors and learn task-oriented representation toward anomaly detection.
  arXiv  Detail & Related papers  (2024-01-06T07:30:41Z)
• A New Benchmark: On the Utility of Synthetic Data with Blender for Bare Supervised Learning and Downstream Domain Adaptation [42.2398858786125]
  Deep learning in computer vision has achieved great success with the price of large-scale labeled training data. The uncontrollable data collection process produces non-IID training and test data, where undesired duplication may exist. To circumvent them, an alternative is to generate synthetic data via 3D rendering with domain randomization.
  arXiv  Detail & Related papers  (2023-03-16T09:03:52Z)
• Synthetic-to-Real Domain Generalized Semantic Segmentation for 3D Indoor Point Clouds [69.64240235315864]
  This paper introduces the synthetic-to-real domain generalization setting to this task. The domain gap between synthetic and real-world point cloud data mainly lies in the different layouts and point patterns. Experiments on the synthetic-to-real benchmark demonstrate that both CINMix and multi-prototypes can narrow the distribution gap.
  arXiv  Detail & Related papers  (2022-12-09T05:07:43Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.