Team Trifecta at Factify5WQA: Setting the Standard in Fact Verification with Fine-Tuning

• URL: http://arxiv.org/abs/2403.10281v1
• Date: Fri, 15 Mar 2024 13:24:28 GMT
• Title: Team Trifecta at Factify5WQA: Setting the Standard in Fact Verification with Fine-Tuning
• Authors: Shang-Hsuan Chiang, Ming-Chih Lo, Lin-Wei Chao, Wen-Chih Peng,
• Abstract summary: We present Pre-CoFactv3, a comprehensive framework comprised of Question Answering and Text Classification components for fact verification. Our team secured first place in the AAAI-24 Factify 3.0 Workshop, surpassing the baseline accuracy by 103% and maintaining a 70% lead over the second competitor.
• Score: 8.292728619484636
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: In this paper, we present Pre-CoFactv3, a comprehensive framework comprised of Question Answering and Text Classification components for fact verification. Leveraging In-Context Learning, Fine-tuned Large Language Models (LLMs), and the FakeNet model, we address the challenges of fact verification. Our experiments explore diverse approaches, comparing different Pre-trained LLMs, introducing FakeNet, and implementing various ensemble methods. Notably, our team, Trifecta, secured first place in the AAAI-24 Factify 3.0 Workshop, surpassing the baseline accuracy by 103% and maintaining a 70% lead over the second competitor. This success underscores the efficacy of our approach and its potential contributions to advancing fact verification research.

Related papers

• Networks of Networks: Complexity Class Principles Applied to Compound AI Systems Design [63.24275274981911]
  Compound AI Systems consisting of many language model inference calls are increasingly employed. In this work, we construct systems, which we call Networks of Networks (NoNs) organized around the distinction between generating a proposed answer and verifying its correctness. We introduce a verifier-based judge NoN with K generators, an instantiation of "best-of-K" or "judge-based" compound AI systems.
  arXiv  Detail & Related papers  (2024-07-23T20:40:37Z)
• Software Mention Recognition with a Three-Stage Framework Based on BERTology Models at SOMD 2024 [0.0]
  This paper describes our systems for the sub-task I in the Software Mention Detection in Scholarly Publications shared-task. Our best performing system addresses the named entity recognition problem through a three-stage framework. Our framework based on the XLM-R-based model achieves a weighted F1-score of 67.80%, delivering our team the 3rd rank in Sub-task I for the Software Mention Recognition task.
  arXiv  Detail & Related papers  (2024-04-23T17:06:24Z)
• The Earth is Flat? Unveiling Factual Errors in Large Language Models [89.94270049334479]
  Large Language Models (LLMs) like ChatGPT are in various applications due to their extensive knowledge from pre-training and fine-tuning. Despite this, they are prone to generating factual and commonsense errors, raising concerns in critical areas like healthcare, journalism, and education. We introduce a novel, automatic testing framework, FactChecker, aimed at uncovering factual inaccuracies in LLMs.
  arXiv  Detail & Related papers  (2024-01-01T14:02:27Z)
• L3 Ensembles: Lifelong Learning Approach for Ensemble of Foundational Language Models [15.726224465017596]
  We propose an approach that focuses on extracting meaningful representations from unseen data and constructing a structured knowledge base. We conducted experiments on various NLP tasks to validate its effectiveness, including benchmarks like GLUE and SuperGLUE. The proposed L3 ensemble method increases the model accuracy by 4% 36% compared to the fine-tuned FLM.
  arXiv  Detail & Related papers  (2023-11-11T06:59:50Z)
• Knowledgeable In-Context Tuning: Exploring and Exploiting Factual Knowledge for In-Context Learning [37.22349652230841]
  Large language models (LLMs) enable in-context learning (ICL) by conditioning on a few labeled training examples as a text-based prompt. In this paper, we demonstrate that factual knowledge is imperative for the performance of ICL in three core facets. We introduce a novel Knowledgeable In-Context Tuning (KICT) framework to further improve the performance of ICL.
  arXiv  Detail & Related papers  (2023-09-26T09:06:39Z)
• Improving Open Information Extraction with Large Language Models: A Study on Demonstration Uncertainty [52.72790059506241]
  Open Information Extraction (OIE) task aims at extracting structured facts from unstructured text. Despite the potential of large language models (LLMs) like ChatGPT as a general task solver, they lag behind state-of-the-art (supervised) methods in OIE tasks.
  arXiv  Detail & Related papers  (2023-09-07T01:35:24Z)
• DiversiGATE: A Comprehensive Framework for Reliable Large Language Models [2.616506436169964]
  We introduce DiversiGATE, a unified framework that consolidates diverse methodologies for LLM verification. We propose a novel SelfLearner' model that conforms to the DiversiGATE framework and refines its performance over time. Our results demonstrate that our approach outperforms traditional LLMs, achieving a considerable 54.8% -> 61.8% improvement on the GSM8K benchmark.
  arXiv  Detail & Related papers  (2023-06-22T22:29:40Z)
• Preserving Knowledge Invariance: Rethinking Robustness Evaluation of Open Information Extraction [50.62245481416744]
  We present the first benchmark that simulates the evaluation of open information extraction models in the real world. We design and annotate a large-scale testbed in which each example is a knowledge-invariant clique. By further elaborating the robustness metric, a model is judged to be robust if its performance is consistently accurate on the overall cliques.
  arXiv  Detail & Related papers  (2023-05-23T12:05:09Z)
• Logically at Factify 2023: A Multi-Modal Fact Checking System Based on Evidence Retrieval techniques and Transformer Encoder Architecture [3.7529756903595963]
  We present the Logically submissions to De-Factify 2 challenge (DE-FACTIFY 2023) on the task 1 of Multi-Modal Fact Checking. We describe our submissions to this challenge including explored evidence retrieval and selection techniques, pre-trained cross-modal and unimodal models, and a cross-modal veracity model based on the well established Transformer (TE) architecture. The final system, a standard two-stage evidence based veracity detection system, yields weighted avg. 0.79 on both val set and final blind test set on the task 1, which achieves 3rd place with a small margin to the top
  arXiv  Detail & Related papers  (2023-01-09T00:19:11Z)
• OPT-IML: Scaling Language Model Instruction Meta Learning through the Lens of Generalization [101.37439352091612]
  We describe the effect of instruction-tuning decisions on downstream task performance when scaling both model and benchmark sizes. We present insights about instruction-tuning decisions as applied to OPT-30B and further exploit these insights to train OPT-IML 30B and 175B, which are instruction-tuned versions of OPT.
  arXiv  Detail & Related papers  (2022-12-22T19:56:09Z)
• UniTE: Unified Translation Evaluation [63.58868113074476]
  UniTE is the first unified framework engaged with abilities to handle all three evaluation tasks. We testify our framework on WMT 2019 Metrics and WMT 2020 Quality Estimation benchmarks.
  arXiv  Detail & Related papers  (2022-04-28T08:35:26Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.