Do Localization Methods Actually Localize Memorized Data in LLMs? A Tale of Two Benchmarks

• URL: http://arxiv.org/abs/2311.09060v2
• Date: Tue, 2 Apr 2024 06:32:40 GMT
• Title: Do Localization Methods Actually Localize Memorized Data in LLMs? A Tale of Two Benchmarks
• Authors: Ting-Yun Chang, Jesse Thomason, Robin Jia,
• Abstract summary: Two benchmarks evaluate the ability of localization methods to pinpoint components responsible for memorized data. Despite their different perspectives, our two benchmarks yield consistent rankings of five localization methods.
• Score: 28.46131289972691
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: The concept of localization in LLMs is often mentioned in prior work; however, methods for localization have never been systematically and directly evaluated. We propose two complementary benchmarks that evaluate the ability of localization methods to pinpoint LLM components responsible for memorized data. In our INJ benchmark, we actively inject a piece of new information into a small subset of LLM weights, enabling us to directly evaluate whether localization methods can identify these "ground truth" weights. In our DEL benchmark, we evaluate localization by measuring how much dropping out identified neurons deletes a memorized pretrained sequence. Despite their different perspectives, our two benchmarks yield consistent rankings of five localization methods. Methods adapted from network pruning perform well on both benchmarks, and all evaluated methods show promising localization ability. On the other hand, even successful methods identify neurons that are not specific to a single memorized sequence.

Related papers

• Mitigating Memorization In Language Models [37.899013074095336]
  Language models (LMs) can "memorize" information, encode training data in their weights in such a way that inference-time queries can lead to verbatim regurgitation of that data. We introduce TinyMem, a suite of small, computationally-efficient LMs for the rapid development and evaluation of memorization-mitigation methods. We show, in particular, that our proposed unlearning method BalancedSubnet outperforms other mitigation methods at removing memorized information while preserving performance on target tasks.
  arXiv  Detail & Related papers  (2024-10-03T02:53:51Z)
• Evaluating Human Alignment and Model Faithfulness of LLM Rationale [66.75309523854476]
  We study how well large language models (LLMs) explain their generations through rationales. We show that prompting-based methods are less "faithful" than attribution-based explanations.
  arXiv  Detail & Related papers  (2024-06-28T20:06:30Z)
• SoK: Membership Inference Attacks on LLMs are Rushing Nowhere (and How to Fix It) [16.673210422615348]
  More than 10 new methods have been proposed to perform Membership Inference Attacks (MIAs) against LLMs. Contrary to traditional MIAs which rely on fixed -- but randomized -- records or models, these methods are mostly evaluated on datasets collected post-hoc. This lack of randomization raises concerns of a distribution shift between members and non-members.
  arXiv  Detail & Related papers  (2024-06-25T23:12:07Z)
• RepEval: Effective Text Evaluation with LLM Representation [55.26340302485898]
  RepEval is a metric that leverages the projection of Large Language Models (LLMs) representations for evaluation. Our work underscores the richness of information regarding text quality embedded within LLM representations, offering insights for the development of new metrics.
  arXiv  Detail & Related papers  (2024-04-30T13:50:55Z)
• Reflection-Tuning: Data Recycling Improves LLM Instruction-Tuning [79.32236399694077]
  Low-quality data in the training set are usually detrimental to instruction tuning. We propose a novel method, termed "reflection-tuning" This approach utilizes an oracle LLM to recycle the original training data by introspecting and enhancing the quality of instructions and responses in the data.
  arXiv  Detail & Related papers  (2023-10-18T05:13:47Z)
• Generating Benchmarks for Factuality Evaluation of Language Models [61.69950787311278]
  We propose FACTOR: Factual Assessment via Corpus TransfORmation, a scalable approach for evaluating LM factuality. FACTOR automatically transforms a factual corpus of interest into a benchmark evaluating an LM's propensity to generate true facts from the corpus vs. similar but incorrect statements. We show that: (i) our benchmark scores increase with model size and improve when the LM is augmented with retrieval; (ii) benchmark score and perplexity do not always agree on model ranking; (iii) when perplexity and benchmark score disagree, the latter better reflects factuality in open-ended generation.
  arXiv  Detail & Related papers  (2023-07-13T17:14:38Z)
• Localization Distillation for Object Detection [134.12664548771534]
  Previous knowledge distillation (KD) methods for object detection mostly focus on feature imitation instead of mimicking the classification logits. We present a novel localization distillation (LD) method which can efficiently transfer the localization knowledge from the teacher to the student. We show that logit mimicking can outperform feature imitation and the absence of localization distillation is a critical reason for why logit mimicking underperforms for years.
  arXiv  Detail & Related papers  (2022-04-12T17:14:34Z)
• On the Limits of Pseudo Ground Truth in Visual Camera Re-localisation [83.29404673257328]
  Re-localisation benchmarks measure how well each method replicates the results of a reference algorithm. This begs the question whether the choice of the reference algorithm favours a certain family of re-localisation methods. This paper analyzes two widely used re-localisation datasets and shows that evaluation outcomes indeed vary with the choice of the reference algorithm.
  arXiv  Detail & Related papers  (2021-09-01T12:01:08Z)
• Ground Texture Based Localization Using Compact Binary Descriptors [12.160708336715489]
  Ground texture based localization is a promising approach to achieve high-accuracy positioning of vehicles. We present a self-contained method that can be used for global localization as well as for subsequent local localization updates.
  arXiv  Detail & Related papers  (2020-02-25T17:31:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.