Introducing Verification Task of Set Consistency with Set-Consistency Energy Networks

• URL: http://arxiv.org/abs/2503.10695v2
• Date: Wed, 19 Mar 2025 04:07:06 GMT
• Title: Introducing Verification Task of Set Consistency with Set-Consistency Energy Networks
• Abstract summary: We introduce the task of set-consistency verification, an extension of natural language inference (NLI)<n>We present the Set-Consistency Energy Network (SC-Energy), a novel model that employs a contrastive loss framework to learn the compatibility among a collection of statements.<n>Our approach not only efficiently verifies inconsistencies and pinpoints the specific statements responsible for logical contradictions, but also significantly outperforms existing methods.
• Score: 4.545178162750511
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Examining logical inconsistencies among multiple statements (such as collections of sentences or question-answer pairs) is a crucial challenge in machine learning, particularly for ensuring the safety and reliability of models. Traditional methods that rely on pairwise comparisons often fail to capture inconsistencies that only emerge when more than two statements are evaluated collectively. To address this gap, we introduce the task of set-consistency verification, an extension of natural language inference (NLI) that assesses the logical coherence of entire sets rather than isolated pairs. Building on this task, we present the Set-Consistency Energy Network (SC-Energy), a novel model that employs a contrastive loss framework to learn the compatibility among a collection of statements. Our approach not only efficiently verifies inconsistencies and pinpoints the specific statements responsible for logical contradictions, but also significantly outperforms existing methods including prompting-based LLM models. Furthermore, we release two new datasets: Set-LConVQA and Set-SNLI for set-consistency verification task.

Related papers

• CONSISTRE: A Unified Consistency-Aware Framework for Document-Level Relation Extraction with Large Language Models [11.083514956613383]
  Document-level relation extraction (DocRE) aims to extract relations among multiple entities across extended contexts.<n>Large language models (LLMs) show remarkable reasoning capabilities in information extraction.<n>LLMs may violate fundamental relational constraints such as transitivity, symmetry, and functional uniqueness, leading to contradictory and unreliable outputs.<n>We propose a unified consistency-aware framework for DocRE that addresses this limitation through two complementary tracks.
  arXiv  Detail & Related papers  (2026-07-27T11:56:29Z)
• From Knowing to Doing Precisely: A General Self-Correction and Termination Framework for VLA models [42.553888395227766]
  We propose a lightweight, training-free framework for vision-language-action (VLA) models.<n>VLA-SCT operates as a self-correcting control loop, combining data-driven action refinement with conditional logic for termination.<n>Our method achieves consistent improvements across all datasets in the LIBERO benchmark.
  arXiv  Detail & Related papers  (2026-02-02T08:44:40Z)
• Deconstructing Instruction-Following: A New Benchmark for Granular Evaluation of Large Language Model Instruction Compliance Abilities [2.9203730377983654]
  Existing benchmarks fail to reflect real-world use or isolate compliance from task success.<n>We introduce MOSAIC, a modular framework that uses a dynamically generated dataset with up to 20 application-oriented generation constraints.<n>We show that compliance is not a monolithic capability but varies significantly with constraint type, quantity, and position.
  arXiv  Detail & Related papers  (2026-01-26T15:02:15Z)
• CoT-Seg: Rethinking Segmentation with Chain-of-Thought Reasoning and Self-Correction [50.67483317563736]
  This paper aims to explore a system that can think step-by-step, look up information if needed, generate results, self-evaluate its own results, and refine the results.<n>We introduce CoT-Seg, a training-free framework that rethinks reasoning segmentation by combining chain-of-thought reasoning with self-correction.
  arXiv  Detail & Related papers  (2026-01-24T11:41:54Z)
• Foundations of Global Consistency Checking with Noisy LLM Oracles [1.9954282121641704]
  We show that verifying global consistency requires exponentially many oracle queries in the worst case.<n>We propose an adaptive divide-and-conquer algorithm that identifies minimal inconsistent subsets.<n> Experiments with both synthetic and real oracles show that our method efficiently detects and localizes inconsistencies.
  arXiv  Detail & Related papers  (2026-01-20T05:02:35Z)
• CoDiEmb: A Collaborative yet Distinct Framework for Unified Representation Learning in Information Retrieval and Semantic Textual Similarity [20.349897901019574]
  We introduce CoDiEmb, a unified framework for training unified text embeddings.<n>CoDiEmb integrates three key innovations for effective joint optimization.<n>Our results and analysis demonstrate that the framework mitigates cross-task trade-offs.
  arXiv  Detail & Related papers  (2025-08-15T12:46:35Z)
• Test-Time Consistency in Vision Language Models [26.475993408532304]
  Vision-Language Models (VLMs) have achieved impressive performance across a wide range of multimodal tasks.<n>Recent benchmarks, such as MM-R3, highlight that even state-of-the-art VLMs can produce divergent predictions across semantically equivalent inputs.<n>We propose a simple and effective test-time consistency framework that enhances semantic consistency without supervised re-training.
  arXiv  Detail & Related papers  (2025-06-27T17:09:44Z)
• CORG: Generating Answers from Complex, Interrelated Contexts [57.213304718157985]
  In a real-world corpus, knowledge frequently recurs across documents but often contains inconsistencies due to ambiguous naming, outdated information, or errors. Previous research has shown that language models struggle with these complexities, typically focusing on single factors in isolation. We introduce Context Organizer (CORG), a framework that organizes multiple contexts into independently processed groups.
  arXiv  Detail & Related papers  (2025-04-25T02:40:48Z)
• Contradiction Detection in RAG Systems: Evaluating LLMs as Context Validators for Improved Information Consistency [0.6827423171182154]
  Retrieval Augmented Generation (RAG) systems have emerged as a powerful method for enhancing large language models (LLMs) with up-to-date information. RAG can sometimes surface documents containing contradictory information, particularly in rapidly evolving domains such as news. This study presents a novel data generation framework to simulate different types of contradictions that may occur in the retrieval stage of a RAG system.
  arXiv  Detail & Related papers  (2025-03-31T19:41:15Z)
• ReCon: Enhancing True Correspondence Discrimination through Relation Consistency for Robust Noisy Correspondence Learning [42.292388966066135]
  We propose a general Relation Consistency learning framework, namely ReCon, to accurately discriminate the true correspondences among the multimodal data.<n>ReCon significantly enhances its effectiveness for true correspondence discrimination and therefore reliably filters out the mismatched pairs.
  arXiv  Detail & Related papers  (2025-02-27T10:38:03Z)
• A Cooperative Multi-Agent Framework for Zero-Shot Named Entity Recognition [71.61103962200666]
  Zero-shot named entity recognition (NER) aims to develop entity recognition systems from unannotated text corpora.<n>Recent work has adapted large language models (LLMs) for zero-shot NER by crafting specialized prompt templates.<n>We introduce the cooperative multi-agent system (CMAS), a novel framework for zero-shot NER.
  arXiv  Detail & Related papers  (2025-02-25T23:30:43Z)
• Large Language Models Meet Symbolic Provers for Logical Reasoning Evaluation [24.081573908824353]
  First-order logic (FOL) reasoning is pivotal for intelligent systems.<n>Existing benchmarks often rely on extensive human annotation or handcrafted templates.<n>We propose a novel framework called ProverGen that synergizes the generative strengths of Large Language Models with the rigor and precision of symbolic provers.
  arXiv  Detail & Related papers  (2025-02-10T15:31:54Z)
• Localizing Factual Inconsistencies in Attributable Text Generation [91.981439746404]
  We introduce QASemConsistency, a new formalism for localizing factual inconsistencies in attributable text generation. We first demonstrate the effectiveness of the QASemConsistency methodology for human annotation. We then implement several methods for automatically detecting localized factual inconsistencies.
  arXiv  Detail & Related papers  (2024-10-09T22:53:48Z)
• Advancing Semantic Textual Similarity Modeling: A Regression Framework with Translated ReLU and Smooth K2 Loss [3.435381469869212]
  This paper presents an innovative regression framework for Sentence-BERT STS tasks. It proposes two simple yet effective loss functions: Translated ReLU and Smooth K2 Loss. Experimental results demonstrate that our method achieves convincing performance across seven established STS benchmarks.
  arXiv  Detail & Related papers  (2024-06-08T02:52:43Z)
• Two Failures of Self-Consistency in the Multi-Step Reasoning of LLMs [78.31625291513589]
  We argue that self-consistency is an important criteria for valid multi-step reasoning in tasks where the solution is composed of the answers to multiple sub-steps. We propose two types of self-consistency that are particularly important for multi-step reasoning -- hypothetical consistency and compositional consistency. We demonstrate that multiple variants of the GPT-3/-4 models exhibit poor consistency rates across both types of consistency on a variety of tasks.
  arXiv  Detail & Related papers  (2023-05-23T17:25:59Z)
• Exposing and Addressing Cross-Task Inconsistency in Unified Vision-Language Models [80.23791222509644]
  Inconsistent AI models are considered brittle and untrustworthy by human users. We find that state-of-the-art vision-language models suffer from a surprisingly high degree of inconsistent behavior across tasks. We propose a rank correlation-based auxiliary training objective, computed over large automatically created cross-task contrast sets.
  arXiv  Detail & Related papers  (2023-03-28T16:57:12Z)
• Robust Learning Through Cross-Task Consistency [92.42534246652062]
  We propose a broadly applicable and fully computational method for augmenting learning with Cross-Task Consistency. We observe that learning with cross-task consistency leads to more accurate predictions and better generalization to out-of-distribution inputs.
  arXiv  Detail & Related papers  (2020-06-07T09:24:33Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.