FASTTRACK: Fast and Accurate Fact Tracing for LLMs

• URL: http://arxiv.org/abs/2404.15157v1
• Date: Mon, 22 Apr 2024 00:07:55 GMT
• Title: FASTTRACK: Fast and Accurate Fact Tracing for LLMs
• Authors: Si Chen, Feiyang Kang, Ning Yu, Ruoxi Jia,
• Abstract summary: This paper introduces FASTTRACK, a novel approach that harnesses the capabilities of Large Language Models (LLMs) to validate supportive evidence for queries. Our experiments show that FASTTRACK substantially outperforms existing methods in both accuracy and efficiency.
• Score: 26.476665624884134
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Fact tracing seeks to identify specific training examples that serve as the knowledge source for a given query. Existing approaches to fact tracing rely on assessing the similarity between each training sample and the query along a certain dimension, such as lexical similarity, gradient, or embedding space. However, these methods fall short of effectively distinguishing between samples that are merely relevant and those that actually provide supportive evidence for the information sought by the query. This limitation often results in suboptimal effectiveness. Moreover, these approaches necessitate the examination of the similarity of individual training points for each query, imposing significant computational demands and creating a substantial barrier for practical applications. This paper introduces FASTTRACK, a novel approach that harnesses the capabilities of Large Language Models (LLMs) to validate supportive evidence for queries and at the same time clusters the training database towards a reduced extent for LLMs to trace facts. Our experiments show that FASTTRACK substantially outperforms existing methods in both accuracy and efficiency, achieving more than 100\% improvement in F1 score over the state-of-the-art methods while being X33 faster than \texttt{TracIn}.

Related papers

• SPaRFT: Self-Paced Reinforcement Fine-Tuning for Large Language Models [51.74498855100541]
  Large language models (LLMs) have shown strong reasoning capabilities when fine-tuned with reinforcement learning (RL)<n>We propose textbfSPaRFT, a self-paced learning framework that enables efficient learning based on the capability of the model being trained.
  arXiv  Detail & Related papers  (2025-08-07T03:50:48Z)
• Improving Task Diversity in Label Efficient Supervised Finetuning of LLMs [14.531280062127442]
  Large Language Models (LLMs) have demonstrated remarkable capabilities across diverse domains, but developing high-performing models for specialized applications often requires substantial human annotation.<n>We address the label-efficient learning problem for supervised finetuning (SFT) by leveraging task-diversity as a fundamental principle for effective data selection.<n>Our approach is based on two key observations: 1) task labels for different prompts are often readily available; 2) pre-trained models have significantly varying levels of confidence across tasks.
  arXiv  Detail & Related papers  (2025-07-29T03:51:00Z)
• Curriculum-style Data Augmentation for LLM-based Metaphor Detection [7.4594050203808395]
  We propose a method for metaphor detection by fine-tuning open-source LLMs. Our method achieves state-of-the-art performance across all baselines.
  arXiv  Detail & Related papers  (2024-12-04T02:05:21Z)
• Words Matter: Leveraging Individual Text Embeddings for Code Generation in CLIP Test-Time Adaptation [21.20806568508201]
  We show how to leverage class text information to mitigate distribution drifts encountered by vision-language models (VLMs) during test-time inference. We propose to generate pseudo-labels for the test-time samples by exploiting generic class text embeddings as fixed centroids of a label assignment problem. Experiments on multiple popular test-time adaptation benchmarks presenting diverse complexity empirically show the superiority of CLIP-OT.
  arXiv  Detail & Related papers  (2024-11-26T00:15:37Z)
• Scalable Influence and Fact Tracing for Large Language Model Pretraining [14.598556308631018]
  Training data attribution (TDA) methods aim to attribute model outputs back to specific training examples. This paper refines existing gradient-based methods to work effectively at scale.
  arXiv  Detail & Related papers  (2024-10-22T20:39:21Z)
• BoostAdapter: Improving Vision-Language Test-Time Adaptation via Regional Bootstrapping [64.8477128397529]
  We propose a training-required and training-free test-time adaptation framework. We maintain a light-weight key-value memory for feature retrieval from instance-agnostic historical samples and instance-aware boosting samples. We theoretically justify the rationality behind our method and empirically verify its effectiveness on both the out-of-distribution and the cross-domain datasets.
  arXiv  Detail & Related papers  (2024-10-20T15:58:43Z)
• SoftDedup: an Efficient Data Reweighting Method for Speeding Up Language Model Pre-training [12.745160748376794]
  We propose a soft deduplication method that maintains dataset integrity while selectively reducing the sampling weight of data with high commonness. Central to our approach is the concept of "data commonness", a metric we introduce to quantify the degree of duplication. Empirical analysis shows that this method significantly improves training efficiency, achieving comparable perplexity scores with at least a 26% reduction in required training steps.
  arXiv  Detail & Related papers  (2024-07-09T08:26:39Z)
• A Fixed-Point Approach to Unified Prompt-Based Counting [51.20608895374113]
  This paper aims to establish a comprehensive prompt-based counting framework capable of generating density maps for objects indicated by various prompt types, such as box, point, and text. Our model excels in prominent class-agnostic datasets and exhibits superior performance in cross-dataset adaptation tasks.
  arXiv  Detail & Related papers  (2024-03-15T12:05:44Z)
• Rethinking Classifier Re-Training in Long-Tailed Recognition: A Simple Logits Retargeting Approach [102.0769560460338]
  We develop a simple logits approach (LORT) without the requirement of prior knowledge of the number of samples per class. Our method achieves state-of-the-art performance on various imbalanced datasets, including CIFAR100-LT, ImageNet-LT, and iNaturalist 2018.
  arXiv  Detail & Related papers  (2024-03-01T03:27:08Z)
• Evaluation of Test-Time Adaptation Under Computational Time Constraints [80.40939405129102]
  Test Time Adaptation (TTA) methods leverage unlabeled data at test time to adapt to distribution shifts. Current evaluation protocols overlook the effect of this extra cost, affecting their real-world applicability. We propose a more realistic evaluation protocol for TTA methods, where data is received in an online fashion from a constant-speed data stream.
  arXiv  Detail & Related papers  (2023-04-10T18:01:47Z)
• Tracing Knowledge in Language Models Back to the Training Data [39.02793789536856]
  We introduce a new benchmark for fact tracing: tracing language models' assertions back to the training examples that provided evidence for those predictions. We evaluate influence methods for fact tracing, using well-understood information retrieval metrics.
  arXiv  Detail & Related papers  (2022-05-23T17:34:16Z)
• Combining Feature and Instance Attribution to Detect Artifacts [62.63504976810927]
  We propose methods to facilitate identification of training data artifacts. We show that this proposed training-feature attribution approach can be used to uncover artifacts in training data. We execute a small user study to evaluate whether these methods are useful to NLP researchers in practice.
  arXiv  Detail & Related papers  (2021-07-01T09:26:13Z)
• An Empirical Comparison of Instance Attribution Methods for NLP [62.63504976810927]
  We evaluate the degree to which different potential instance attribution agree with respect to the importance of training samples. We find that simple retrieval methods yield training instances that differ from those identified via gradient-based methods.
  arXiv  Detail & Related papers  (2021-04-09T01:03:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.