Unlocking Crowdsourcing for Ontology Matching Validation

• URL: http://arxiv.org/abs/2605.12226v2
• Date: Tue, 19 May 2026 16:10:15 GMT
• Title: Unlocking Crowdsourcing for Ontology Matching Validation
• Abstract summary: Large language models (LLMs) pose new challenges for ontology matching (OM)<n>Traditional OM validation that relies on domain experts has become overwhelming.<n>We propose three domain-specific mechanisms, namely differential trustworthiness, coherence pre-filling, and time-dependent opinion, to ensure the quality of crowdsourcing for OM validation.<n>We demonstrate that our crowdsourcing system can be integrated with existing OM systems to enable human-in-the-loop validation.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Recent advances in large language models (LLMs) pose new challenges for ontology matching (OM). While OM systems built on LLMs have shown remarkable capabilities in discovering more matching candidates, traditional OM validation that relies on domain experts has become overwhelming. In this study, we explore the use of crowdsourcing for OM validation and introduce a novel crowdsourcing system. We propose three domain-specific mechanisms, namely differential trustworthiness, coherence pre-filling, and time-dependent opinion, to ensure the quality of crowdsourcing for OM validation. We demonstrate that our crowdsourcing system can be integrated with existing OM systems to enable human-in-the-loop validation. The evaluation of the system also shows its effectiveness in handling diverse user groups and different annotation settings. We also discuss two real-world use cases and current limitations for improvement.

Related papers

• FindIt: A Format-Informed Visual Detection Benchmark for Generalist Multimodal LLMs [37.64883536754805]
  We introduce the first comprehensive benchmark specifically designed to assess the promptable localization abilities of generalist MLLMs.<n>Our benchmark spans four core task categories: object detection, referring expression detection, instance-level detection, and video-based detection.<n>We evaluate a diverse set of open-source and proprietary MLLMs, providing an in-depth analysis of their performance and limitations.
  arXiv  Detail & Related papers  (2026-06-02T23:14:46Z)
• CommunityFact: A Dynamic, Multilingual, Multi-domain Benchmark for Misinformation Detection in the Wild [29.218863878839368]
  CommunityFact is a refreshable benchmark for misinformation detection in the wild.<n>This release contains 15,992 standalone claims across five languages and two domains.
  arXiv  Detail & Related papers  (2026-05-28T17:09:19Z)
• MAS-ProVe: Understanding the Process Verification of Multi-Agent Systems [59.20800753428596]
  We present MAS-ProVe, a systematic empirical study of process verification for multi-agent systems (MAS)<n>Our study spans three verification paradigms (LLM-as-a-Judge, reward models, and process reward models)<n>We find that process-level verification does not consistently improve performance and frequently exhibits high variance.
  arXiv  Detail & Related papers  (2026-02-03T03:30:36Z)
• Benchmarking Chinese Commonsense Reasoning with a Multi-hop Reasoning Perspective [53.594353527056775]
  We propose Chinese Commonsense Multi-hop Reasoning ( CCMOR) to evaluate Large Language Models (LLMs)<n> CCMOR is designed to evaluate LLMs' ability to integrate Chinese-specific factual knowledge with multi-step logical reasoning.<n>We implement a human-in-the-loop verification system, where domain experts systematically validate and refine the generated questions.
  arXiv  Detail & Related papers  (2025-10-09T20:29:00Z)
• Towards Unified Multimodal Misinformation Detection in Social Media: A Benchmark Dataset and Baseline [56.790045049514326]
  Two major forms of deception dominate: human-crafted misinformation and AI-generated content.<n>We propose Unified Multimodal Fake Content Detection (UMFDet), a framework designed to handle both forms of deception.<n>UMFDet achieves robust and consistent performance across both misinformation types, outperforming specialized baselines.
  arXiv  Detail & Related papers  (2025-09-30T09:26:32Z)
• Quantization Meets dLLMs: A Systematic Study of Post-training Quantization for Diffusion LLMs [78.09559830840595]
  We present the first systematic study on quantizing diffusion-based language models.<n>We identify the presence of activation outliers, characterized by abnormally large activation values.<n>We implement state-of-the-art PTQ methods and conduct a comprehensive evaluation.
  arXiv  Detail & Related papers  (2025-08-20T17:59:51Z)
• A Comprehensive Review on Harnessing Large Language Models to Overcome Recommender System Challenges [5.750235776275005]
  Large Language Models (LLMs) can be leveraged to tackle key challenges in recommender systems.<n>LLMs enhance personalization, semantic alignment, and interpretability without requiring extensive task-specific supervision.<n>LLMs enable zero- and few-shot reasoning, allowing systems to operate effectively in cold-start and long-tail scenarios.
  arXiv  Detail & Related papers  (2025-07-17T06:03:57Z)
• Graft: Integrating the Domain Knowledge via Efficient Parameter Synergy for MLLMs [56.76586846269894]
  Multimodal Large Language Models (MLLMs) have achieved success across various domains.<n>Despite its importance, the study of knowledge sharing among domain-specific MLLMs remains largely underexplored.<n>We propose a unified parameter integration framework that enables modular composition of expert capabilities.
  arXiv  Detail & Related papers  (2025-06-30T15:07:41Z)
• Benchmarking Retrieval-Augmented Multimodal Generation for Document Question Answering [60.062194349648195]
  Document Visual Question Answering (DocVQA) faces dual challenges in processing lengthy multimodal documents.<n>Current document retrieval-augmented generation (DocRAG) methods remain limited by their text-centric approaches.<n>We introduce MMDocRAG, a comprehensive benchmark featuring 4,055 expert-annotated QA pairs with multi-page, cross-modal evidence chains.
  arXiv  Detail & Related papers  (2025-05-22T09:52:57Z)
• Towards Agentic Recommender Systems in the Era of Multimodal Large Language Models [75.4890331763196]
  Recent breakthroughs in Large Language Models (LLMs) have led to the emergence of agentic AI systems.<n>LLM-based Agentic RS (LLM-ARS) can offer more interactive, context-aware, and proactive recommendations.
  arXiv  Detail & Related papers  (2025-03-20T22:37:15Z)
• Why Do Multi-Agent LLM Systems Fail? [87.90075668488434]
  We introduce MAST-Data, a comprehensive dataset of 1600+ annotated traces collected across 7 popular MAS frameworks.<n>We build the first Multi-Agent System Failure taxonomy (MAST)<n>We leverage MAST and MAST-Data to analyze failure patterns across models (GPT4, Claude 3, Qwen2.5, CodeLlama) and tasks (coding, math, general agent)
  arXiv  Detail & Related papers  (2025-03-17T19:04:38Z)
• MMIE: Massive Multimodal Interleaved Comprehension Benchmark for Large Vision-Language Models [71.36392373876505]
  We introduce MMIE, a large-scale benchmark for evaluating interleaved multimodal comprehension and generation in Large Vision-Language Models (LVLMs)<n>MMIE comprises 20K meticulously curated multimodal queries, spanning 3 categories, 12 fields, and 102 subfields, including mathematics, coding, physics, literature, health, and arts.<n>It supports both interleaved inputs and outputs, offering a mix of multiple-choice and open-ended question formats to evaluate diverse competencies.
  arXiv  Detail & Related papers  (2024-10-14T04:15:00Z)
• LLMs4OM: Matching Ontologies with Large Language Models [0.14999444543328289]
  Ontology Matching (OM) is a critical task in knowledge integration, where aligning heterogeneous data interoperability and knowledge sharing. We present the LLMs4OM framework, a novel approach to evaluate the effectiveness of Large Language Models (LLMs) in OM tasks.
  arXiv  Detail & Related papers  (2024-04-16T06:55:45Z)
• FKA-Owl: Advancing Multimodal Fake News Detection through Knowledge-Augmented LVLMs [48.32113486904612]
  We propose FKA-Owl, a framework that leverages forgery-specific knowledge to augment Large Vision-Language Models (LVLMs) Experiments on the public benchmark demonstrate that FKA-Owl achieves superior cross-domain performance compared to previous methods.
  arXiv  Detail & Related papers  (2024-03-04T12:35:09Z)
• Agent-OM: Leveraging LLM Agents for Ontology Matching [4.222245509121683]
  This study introduces a novel agent-powered design paradigm for Ontology matching systems.<n>We propose a framework, namely Agent-OM (Agent for Ontology Matching), consisting of two Siamese agents for retrieval and matching.<n>Our system can achieve results very close to the long-standing best performance on simple OM tasks.
  arXiv  Detail & Related papers  (2023-12-01T03:44:54Z)
• Myriad: Large Multimodal Model by Applying Vision Experts for Industrial Anomaly Detection [86.24898024621008]
  We present a novel large multimodal model applying vision experts for industrial anomaly detection(abbreviated to Myriad)<n>We utilize the anomaly map generated by the vision experts as guidance for LMMs, such that the vision model is guided to pay more attention to anomalous regions.<n>Our proposed method not only performs favorably against state-of-the-art methods, but also inherits the flexibility and instruction-following ability of LMMs in the field of IAD.
  arXiv  Detail & Related papers  (2023-10-29T16:49:45Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.