A Multimodal Manufacturing Safety Chatbot: Knowledge Base Design, Benchmark Development, and Evaluation of Multiple RAG Approaches

• URL: http://arxiv.org/abs/2511.11847v1
• Date: Fri, 14 Nov 2025 20:10:23 GMT
• Title: A Multimodal Manufacturing Safety Chatbot: Knowledge Base Design, Benchmark Development, and Evaluation of Multiple RAG Approaches
• Abstract summary: Three essential requirements for next-generation safety training systems: high accuracy, low latency, and low cost.<n>We introduce a multimodal chatbots powered by large language models that meets these design requirements.<n>The top configuration achieved an accuracy of 86.66%, an average latency of 10.04 seconds, and an average cost of $0.005 per query.
• Score: 3.2642182031768487
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Ensuring worker safety remains a critical challenge in modern manufacturing environments. Industry 5.0 reorients the prevailing manufacturing paradigm toward more human-centric operations. Using a design science research methodology, we identify three essential requirements for next-generation safety training systems: high accuracy, low latency, and low cost. We introduce a multimodal chatbot powered by large language models that meets these design requirements. The chatbot uses retrieval-augmented generation to ground its responses in curated regulatory and technical documentation. To evaluate our solution, we developed a domain-specific benchmark of expert-validated question and answer pairs for three representative machines: a Bridgeport manual mill, a Haas TL-1 CNC lathe, and a Universal Robots UR5e collaborative robot. We tested 24 RAG configurations using a full-factorial design and assessed them with automated evaluations of correctness, latency, and cost. Our top 2 configurations were then evaluated by ten industry experts and academic researchers. Our results show that retrieval strategy and model configuration have a significant impact on performance. The top configuration (selected for chatbot deployment) achieved an accuracy of 86.66%, an average latency of 10.04 seconds, and an average cost of $0.005 per query. Overall, our work provides three contributions: an open-source, domain-grounded safety training chatbot; a validated benchmark for evaluating AI-assisted safety instruction; and a systematic methodology for designing and assessing AI-enabled instructional and immersive safety training systems for Industry 5.0 environments.

Related papers

• GigaWorld-1: A Roadmap to Build World Models for Robot Policy Evaluation [73.02743604295246]
  WMBench is a benchmark constructed from real-robot teleoperation data and matched policy rollouts.<n>We analyze 7 video world models, 4 action representation schemes, and over 324,000 simulated policy rollouts paired with real robot executions.<n>Our experiments deliver three core insights: evaluator quality is dominated by long-horizon, action-faithful rollout consistency rather than short-term visual realism.
  arXiv  Detail & Related papers  (2026-07-02T17:02:43Z)
• MLReplicate: Benchmarking Autonomous Research Systems for Machine Learning Reproducibility [2.2969445973129123]
  We introduce MLReplicate, an end-to-end benchmark evaluating autonomous research systems on machine learning.<n>The benchmark was constructed from ICML 2025 papers reformulated into standardized input specifications and evaluated across 6 state-of-the-art research systems.<n>In contrast to automated reviews, human reviewers consistently identified methodological flaws, hallucinated experimental results, and failures across all systems.
  arXiv  Detail & Related papers  (2026-05-15T20:35:32Z)
• Benchmarking Generalizable Bimanual Manipulation: RoboTwin Dual-Arm Collaboration Challenge at CVPR 2025 MEIS Workshop [120.2806035123366]
  RoboTwin Dual-Arm Collaboration Challenge was held at the 2nd MEIS Workshop, CVPR 2025.<n>Competitors totally tackled 17 dual-arm manipulation tasks, covering rigid, deformable, and tactile-based scenarios.<n>Report outlines the competition setup, task design, evaluation methodology, key findings and future direction.
  arXiv  Detail & Related papers  (2025-06-29T17:56:41Z)
• Investigating the Potential of Large Language Model-Based Router Multi-Agent Architectures for Foundation Design Automation: A Task Classification and Expert Selection Study [0.0]
  Three approaches were evaluated: single-agent processing, multi-agent designer-checker architecture, and router-based expert selection.<n>Performance assessment utilized baseline models including DeepSeek R1, ChatGPT 4 Turbo, Grok 3, and Gemini 2.5 Pro.<n>Results establish router-based multi-agent systems as optimal for foundation design automation while maintaining professional documentation standards.
  arXiv  Detail & Related papers  (2025-06-13T23:45:24Z)
• From Code Generation to Software Testing: AI Copilot with Context-Based RAG [8.28588489551341]
  We propose a novel perspective on software testing by positing bug detection and coding with fewer bugs as two interconnected problems.<n>We introduce Copilot for Testing, an automated testing system that synchronizes bug detection with updates.<n>Our evaluation demonstrates a 31.2% improvement in bug detection accuracy, a 12.6% increase in critical test coverage, and a 10.5% higher user acceptance rate.
  arXiv  Detail & Related papers  (2025-04-02T16:20:05Z)
• AILuminate: Introducing v1.0 of the AI Risk and Reliability Benchmark from MLCommons [62.374792825813394]
  This paper introduces AILuminate v1.0, the first comprehensive industry-standard benchmark for assessing AI-product risk and reliability.<n>The benchmark evaluates an AI system's resistance to prompts designed to elicit dangerous, illegal, or undesirable behavior in 12 hazard categories.
  arXiv  Detail & Related papers  (2025-02-19T05:58:52Z)
• Runtime Verification and Field-based Testing for ROS-based Robotic Systems [8.675312581079039]
  No clear guidance exists for architecting ROS-based systems to enable runtime verification and field-based testing. This paper aims to fill this gap by providing guidelines to help developers and quality assurance (QA) teams develop, verify, or test their robots in the field.
  arXiv  Detail & Related papers  (2024-04-17T15:52:29Z)
• ASSERT: Automated Safety Scenario Red Teaming for Evaluating the Robustness of Large Language Models [65.79770974145983]
  ASSERT, Automated Safety Scenario Red Teaming, consists of three methods -- semantically aligned augmentation, target bootstrapping, and adversarial knowledge injection. We partition our prompts into four safety domains for a fine-grained analysis of how the domain affects model performance. We find statistically significant performance differences of up to 11% in absolute classification accuracy among semantically related scenarios and error rates of up to 19% absolute error in zero-shot adversarial settings.
  arXiv  Detail & Related papers  (2023-10-14T17:10:28Z)
• Comparing Software Developers with ChatGPT: An Empirical Investigation [0.0]
  This paper conducts an empirical investigation, contrasting the performance of software engineers and AI systems, like ChatGPT, across different evaluation metrics. The paper posits that a comprehensive comparison of software engineers and AI-based solutions, considering various evaluation criteria, is pivotal in fostering human-machine collaboration.
  arXiv  Detail & Related papers  (2023-05-19T17:25:54Z)
• Using Sampling to Estimate and Improve Performance of Automated Scoring Systems with Guarantees [63.62448343531963]
  We propose a combination of the existing paradigms, sampling responses to be scored by humans intelligently. We observe significant gains in accuracy (19.80% increase on average) and quadratic weighted kappa (QWK) (25.60% on average) with a relatively small human budget.
  arXiv  Detail & Related papers  (2021-11-17T05:00:51Z)
• RobustART: Benchmarking Robustness on Architecture Design and Training Techniques [170.3297213957074]
  Deep neural networks (DNNs) are vulnerable to adversarial noises. There are no comprehensive studies of how architecture design and training techniques affect robustness. We propose the first comprehensiveness investigation benchmark on ImageNet.
  arXiv  Detail & Related papers  (2021-09-11T08:01:14Z)
• Integrated Benchmarking and Design for Reproducible and Accessible Evaluation of Robotic Agents [61.36681529571202]
  We describe a new concept for reproducible robotics research that integrates development and benchmarking. One of the central components of this setup is the Duckietown Autolab, a standardized setup that is itself relatively low-cost and reproducible. We validate the system by analyzing the repeatability of experiments conducted using the infrastructure and show that there is low variance across different robot hardware and across different remote labs.
  arXiv  Detail & Related papers  (2020-09-09T15:31:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.