SemEval-2025 Task 4: Unlearning sensitive content from Large Language Models

• URL: http://arxiv.org/abs/2504.02883v1
• Date: Wed, 02 Apr 2025 07:24:59 GMT
• Title: SemEval-2025 Task 4: Unlearning sensitive content from Large Language Models
• Authors: Anil Ramakrishna, Yixin Wan, Xiaomeng Jin, Kai-Wei Chang, Zhiqi Bu, Bhanukiran Vinzamuri, Volkan Cevher, Mingyi Hong, Rahul Gupta,
• Abstract summary: We introduce SemEval-2025 Task 4: unlearning sensitive content from Large Language Models (LLMs)<n>The task features three subtasks: (1) unlearn long form synthetic creative documents spanning different genres; (2) unlearn short form synthetic biographies containing personally identifiable information (PII); and (3) unlearn real documents sampled from the target model's training dataset.
• Score: 106.83812472773522
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We introduce SemEval-2025 Task 4: unlearning sensitive content from Large Language Models (LLMs). The task features 3 subtasks for LLM unlearning spanning different use cases: (1) unlearn long form synthetic creative documents spanning different genres; (2) unlearn short form synthetic biographies containing personally identifiable information (PII), including fake names, phone number, SSN, email and home addresses, and (3) unlearn real documents sampled from the target model's training dataset. We received over 100 submissions from over 30 institutions and we summarize the key techniques and lessons in this paper.

Related papers

• LUME: LLM Unlearning with Multitask Evaluations [106.83812472773522]
  Unlearning aims to remove copyrighted, sensitive, or private content from large language models (LLMs) without a full retraining.<n>We develop a multi-task unlearning benchmark (LUME) which features three tasks: (1) unlearn synthetically generated creative short novels, (2) unlearn synthetic biographies with sensitive information, and (3) unlearn a collection of public biographies.
  arXiv  Detail & Related papers  (2025-02-20T23:30:45Z)
• Seed-Free Synthetic Data Generation Framework for Instruction-Tuning LLMs: A Case Study in Thai [5.670682861458055]
  We present a synthetic data approach for instruction-tuning large language models (LLMs) for low-resource languages in a data-efficient manner, specifically focusing on Thai. We identify three key properties that contribute to the effectiveness of instruction-tuning datasets: fluency, diversity, and cultural context. Our framework employs an LLM to generate diverse topics, retrieve relevant contexts from Wikipedia, and create instructions for various tasks, such as question answering, summarization, and conversation.
  arXiv  Detail & Related papers  (2024-11-23T07:50:59Z)
• Integrating Planning into Single-Turn Long-Form Text Generation [66.08871753377055]
  We propose to use planning to generate long form content. Our main novelty lies in a single auxiliary task that does not require multiple rounds of prompting or planning. Our experiments demonstrate on two datasets from different domains, that LLMs fine-tuned with the auxiliary task generate higher quality documents.
  arXiv  Detail & Related papers  (2024-10-08T17:02:40Z)
• Prompt Refinement or Fine-tuning? Best Practices for using LLMs in Computational Social Science Tasks [0.0]
  We present an overview of the performance of modern LLM-based classification methods on a benchmark of 23 social knowledge tasks. Our results point to three best practices: select models with larger vocabulary and pre-training corpora; avoid simple zero-shot in favor of AI-enhanced prompting; fine-tune on task-specific data.
  arXiv  Detail & Related papers  (2024-08-02T15:46:36Z)
• Heidelberg-Boston @ SIGTYP 2024 Shared Task: Enhancing Low-Resource Language Analysis With Character-Aware Hierarchical Transformers [2.3020018305241337]
  This work focuses on PoS tagging, morphological tagging, and lemmatization for 13 historical languages. We adapt a hierarchical tokenization method from Sun et al. (2023) and combine it with the advantages of the DeBERTa-V3 architecture. Our models achieved first place in the constrained subtask, nearly reaching the performance levels of the unconstrained task's winner.
  arXiv  Detail & Related papers  (2024-05-30T15:23:34Z)
• Automated Few-shot Classification with Instruction-Finetuned Language Models [76.69064714392165]
  We show that AuT-Few outperforms state-of-the-art few-shot learning methods. We also show that AuT-Few is the best ranking method across datasets on the RAFT few-shot benchmark.
  arXiv  Detail & Related papers  (2023-05-21T21:50:27Z)
• AnnoLLM: Making Large Language Models to Be Better Crowdsourced Annotators [98.11286353828525]
  GPT-3.5 series models have demonstrated remarkable few-shot and zero-shot ability across various NLP tasks. We propose AnnoLLM, which adopts a two-step approach, explain-then-annotate. We build the first conversation-based information retrieval dataset employing AnnoLLM.
  arXiv  Detail & Related papers  (2023-03-29T17:03:21Z)
• PERFECT: Prompt-free and Efficient Few-shot Learning with Language Models [67.3725459417758]
  PERFECT is a simple and efficient method for few-shot fine-tuning of PLMs without relying on any such handcrafting. We show that manually engineered task prompts can be replaced with task-specific adapters that enable sample-efficient fine-tuning. Experiments on a wide range of few-shot NLP tasks demonstrate that PERFECT, while being simple and efficient, also outperforms existing state-of-the-art few-shot learning methods.
  arXiv  Detail & Related papers  (2022-04-03T22:31:25Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.