Have You Ever Seen Them? Entity-level Membership Inference through Interrogating Large Language Models

• URL: http://arxiv.org/abs/2606.23030v1
• Date: Mon, 22 Jun 2026 08:40:43 GMT
• Title: Have You Ever Seen Them? Entity-level Membership Inference through Interrogating Large Language Models
• Abstract summary: Large Language Models (LLMs) raise growing concerns about privacy leakage and copyright compliance.<n>We propose entity-level membership inference, which determines whether information related to a target entity is used in LLM training.<n>We construct entity-level datasets and adapt state-of-the-art sample-level label-only methods to the entity-level setting as baselines.
• Score: 6.8669133150322255
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Large Language Models (LLMs) raise growing concerns about privacy leakage and copyright compliance. Membership inference is a key tool for assessing such risks, but existing studies mainly focus on whether specific samples or sample-based data units are used for training. We argue that LLMs exhibit a human-memory-like behavior: an LLM may not memorize a specific sample verbatim, yet it can accumulate and reveal knowledge about a real-world entity from scattered mentions. This analogy motivates us to examine whether an LLM can be interrogated like a human interviewee to reveal its exposure to entity-related information. Motivated by this question, we propose entity-level membership inference, which determines whether information related to a target entity is used in LLM training. We study this task in the practical label-only black-box setting, where only generated texts are observable. We formalize the task under clue, input, and model constraints, establish the necessary and sufficient conditions for its feasibility, and instantiate five interrogation strategies based on this formalization. The strategies use limited entity clues to construct prompts, elicit entity-related responses, and infer membership from semantic features among the generated texts. We construct entity-level datasets and adapt state-of-the-art sample-level label-only methods to the entity-level setting as baselines. Experiments on person entities show that our methods achieve AUC up to 0.97 and bring gains of 6.0%--17.5% in Balanced Accuracy over the best adapted baseline.

Related papers

• EL-MIA: Quantifying Membership Inference Risks of Sensitive Entities in LLMs [10.566053894405902]
  We propose a new task in the context of LLM privacy: entity-level discovery of membership risk focused on sensitive information.<n>Existing methods for MIA can detect the presence of entire prompts or documents in the LLM training data, but they fail to capture risks at a finer granularity.<n>We construct a benchmark dataset for the evaluation of MIA methods on this task.
  arXiv  Detail & Related papers  (2025-10-31T18:50:47Z)
• Large Language Models Enhanced by Plug and Play Syntactic Knowledge for Aspect-based Sentiment Analysis [18.58619664067155]
  Aspect-based sentiment analysis (ABSA) generally requires a deep understanding of the contextual information.<n>We propose a memory module that records syntactic information and is incorporated into language models (LLMs) to instruct the prediction of sentiment polarities.
  arXiv  Detail & Related papers  (2025-06-15T23:16:12Z)
• Contextual Integrity in LLMs via Reasoning and Reinforcement Learning [41.795843170879046]
  We develop a reinforcement learning framework that instills in models the reasoning necessary to achieve contextual integrity.<n>We show that our method substantially reduces inappropriate information disclosure while maintaining task performance.
  arXiv  Detail & Related papers  (2025-05-29T21:26:21Z)
• MoRE-LLM: Mixture of Rule Experts Guided by a Large Language Model [54.14155564592936]
  We propose a Mixture of Rule Experts guided by a Large Language Model (MoRE-LLM)<n>MoRE-LLM steers the discovery of local rule-based surrogates during training and their utilization for the classification task.<n>LLM is responsible for enhancing the domain knowledge alignment of the rules by correcting and contextualizing them.
  arXiv  Detail & Related papers  (2025-03-26T11:09:21Z)
• Latent Factor Models Meets Instructions: Goal-conditioned Latent Factor Discovery without Task Supervision [50.45597801390757]
  Instruct-LF is a goal-oriented latent factor discovery system.<n>It integrates instruction-following ability with statistical models to handle noisy datasets.
  arXiv  Detail & Related papers  (2025-02-21T02:03:08Z)
• Self-Comparison for Dataset-Level Membership Inference in Large (Vision-)Language Models [73.94175015918059]
  We propose a dataset-level membership inference method based on Self-Comparison. Our method does not require access to ground-truth member data or non-member data in identical distribution.
  arXiv  Detail & Related papers  (2024-10-16T23:05:59Z)
• Zero-to-Strong Generalization: Eliciting Strong Capabilities of Large Language Models Iteratively without Gold Labels [75.77877889764073]
  Large Language Models (LLMs) have demonstrated remarkable performance through supervised fine-tuning or in-context learning using gold labels. This study explores whether solely utilizing unlabeled data can elicit strong model capabilities. We propose a new paradigm termed zero-to-strong generalization.
  arXiv  Detail & Related papers  (2024-09-19T02:59:44Z)
• ReCaLL: Membership Inference via Relative Conditional Log-Likelihoods [56.073335779595475]
  We propose ReCaLL (Relative Conditional Log-Likelihood) to detect pretraining data by leveraging conditional language modeling capabilities.<n>Our empirical findings show that conditioning member data on non-member prefixes induces a larger decrease in log-likelihood compared to non-member data.<n>We conduct comprehensive experiments and show that ReCaLL achieves state-of-the-art performance on the WikiMIA dataset.
  arXiv  Detail & Related papers  (2024-06-23T00:23:13Z)
• P-ICL: Point In-Context Learning for Named Entity Recognition with Large Language Models [7.037794031385439]
  In recent years, the rise of large language models (LLMs) has made it possible to directly achieve named entity recognition (NER) without any demonstration samples. Standard ICL only helps LLMs understand task instructions, format and input-label mapping, but neglects the particularity of the NER task itself. We propose a new prompting framework P-ICL to better achieve NER with LLMs, in which some point entities are leveraged as the auxiliary information to recognize each entity type.
  arXiv  Detail & Related papers  (2024-05-08T11:01:21Z)
• CLOMO: Counterfactual Logical Modification with Large Language Models [109.60793869938534]
  We introduce a novel task, Counterfactual Logical Modification (CLOMO), and a high-quality human-annotated benchmark. In this task, LLMs must adeptly alter a given argumentative text to uphold a predetermined logical relationship. We propose an innovative evaluation metric, the Self-Evaluation Score (SES), to directly evaluate the natural language output of LLMs.
  arXiv  Detail & Related papers  (2023-11-29T08:29:54Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.