GlitchMiner: Mining Glitch Tokens in Large Language Models via Gradient-based Discrete Optimization

• URL: http://arxiv.org/abs/2410.15052v4
• Date: Sat, 09 Nov 2024 06:25:27 GMT
• Title: GlitchMiner: Mining Glitch Tokens in Large Language Models via Gradient-based Discrete Optimization
• Authors: Zihui Wu, Haichang Gao, Ping Wang, Shudong Zhang, Zhaoxiang Liu, Shiguo Lian,
• Abstract summary: Glitch tokens in Large Language Models (LLMs) can trigger unpredictable behaviors, threatening model reliability and safety. We propose GlitchMiner, a gradient-based discrete optimization framework that efficiently identifies glitch tokens. Experiments across multiple LLM architectures demonstrate that GlitchMiner outperforms existing methods in detection accuracy and adaptability.
• Score: 5.962706501263955
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Glitch tokens in Large Language Models (LLMs) can trigger unpredictable behaviors, threatening model reliability and safety. Existing detection methods rely on predefined patterns, limiting their adaptability across diverse LLM architectures. We propose GlitchMiner, a gradient-based discrete optimization framework that efficiently identifies glitch tokens by introducing entropy as a measure of prediction uncertainty and employing a local search strategy to explore the token space. Experiments across multiple LLM architectures demonstrate that GlitchMiner outperforms existing methods in detection accuracy and adaptability, achieving over 10% average efficiency improvement. This method enhances vulnerability assessment in LLMs, contributing to the development of more robust and reliable applications. Code is available at https://github.com/wooozihui/GlitchMiner.

Related papers

• MSL: Not All Tokens Are What You Need for Tuning LLM as a Recommender [24.03860153639828]
  We propose a novel Masked Softmax Loss (MSL) tailored for fine-tuning large language models (LLMs) on recommendation. MSL improves LML by identifying and masking invalid tokens that could lead to fictitious item descriptions during loss computation. Extensive experiments conducted on four public datasets further validate the effectiveness of MSL, achieving an average improvement of 42.24% in NDCG@10.
  arXiv  Detail & Related papers  (2025-04-05T13:48:33Z)
• LLM-Guided Evolution: An Autonomous Model Optimization for Object Detection [0.0]
  In machine learning, Neural Architecture Search (NAS) requires domain knowledge of model design and a large amount of trial-and-error to achieve promising performance. The Large Language Model (LLM)-Guided Evolution (GE) framework transformed this approach by incorporating LLMs to directly modify model source code for image classification algorithms on CIFAR data. We show that LLM-GE produced variants with significant performance improvements, such as an increase in Mean Average Precision from 92.5% to 94.5%.
  arXiv  Detail & Related papers  (2025-04-03T05:06:06Z)
• Learning on LLM Output Signatures for gray-box LLM Behavior Analysis [52.81120759532526]
  Large Language Models (LLMs) have achieved widespread adoption, yet our understanding of their behavior remains limited. We develop a transformer-based approach to process that theoretically guarantees approximation of existing techniques. Our approach achieves superior performance on hallucination and data contamination detection in gray-box settings.
  arXiv  Detail & Related papers  (2025-03-18T09:04:37Z)
• CogSteer: Cognition-Inspired Selective Layer Intervention for Efficient Semantic Steering in Large Language Models [22.42235251921268]
  We propose using eye movement measures to interpret large language models (LLMs) behavior across layers. Inspired by these findings, we introduce a steering layer selection and apply it to layer intervention methods via fine-tuning and inference. Our proposed CogSteer methods achieve better results in terms of toxicity scores while efficiently saving 97% of the computational resources and 60% of the training time.
  arXiv  Detail & Related papers  (2024-10-23T09:40:15Z)
• Zeroth-Order Fine-Tuning of LLMs in Random Subspaces [66.27334633749734]
  As language models grow in size, memory demands for backpropagation increase. Zeroth-order (ZOZO) optimization methods offer a memory-efficient alternative. We show that SubZero enhances fine-tuning and achieves faster results compared to standard ZOZO approaches.
  arXiv  Detail & Related papers  (2024-10-11T17:01:43Z)
• EVOLvE: Evaluating and Optimizing LLMs For Exploration [76.66831821738927]
  Large language models (LLMs) remain under-studied in scenarios requiring optimal decision-making under uncertainty. We measure LLMs' (in)ability to make optimal decisions in bandits, a state-less reinforcement learning setting relevant to many applications. Motivated by the existence of optimal exploration algorithms, we propose efficient ways to integrate this algorithmic knowledge into LLMs.
  arXiv  Detail & Related papers  (2024-10-08T17:54:03Z)
• Functional Homotopy: Smoothing Discrete Optimization via Continuous Parameters for LLM Jailbreak Attacks [24.935016443423233]
  This study introduces a novel optimization approach, termed the emphfunctional homotopy method. By constructing a series of easy-to-hard optimization problems, we iteratively solve these problems using principles derived from established homotopy methods. We apply this approach to jailbreak attack synthesis for large language models (LLMs), achieving a $20%-30%$ improvement in success rate over existing methods.
  arXiv  Detail & Related papers  (2024-10-05T17:22:39Z)
• Language Models as Zero-shot Lossless Gradient Compressors: Towards General Neural Parameter Prior Models [56.00251589760559]
  Large language models (LLMs) can act as gradient priors in a zero-shot setting. We introduce LM-GC, a novel method that integrates LLMs with arithmetic coding. Experiments indicate that LM-GC surpasses existing state-of-the-art lossless compression methods.
  arXiv  Detail & Related papers  (2024-09-26T13:38:33Z)
• GlitchProber: Advancing Effective Detection and Mitigation of Glitch Tokens in Large Language Models [17.633722815221983]
  Large language models (LLMs) have achieved unprecedented success in the field of natural language processing. Recent research has discovered a class of abnormal tokens in the model's vocabulary space and named them "glitch tokens" In this work, we aim to enhance the understanding of glitch tokens and propose techniques for their detection and mitigation.
  arXiv  Detail & Related papers  (2024-08-09T07:19:53Z)
• Exploring Automatic Cryptographic API Misuse Detection in the Era of LLMs [60.32717556756674]
  This paper introduces a systematic evaluation framework to assess Large Language Models in detecting cryptographic misuses. Our in-depth analysis of 11,940 LLM-generated reports highlights that the inherent instabilities in LLMs can lead to over half of the reports being false positives. The optimized approach achieves a remarkable detection rate of nearly 90%, surpassing traditional methods and uncovering previously unknown misuses in established benchmarks.
  arXiv  Detail & Related papers  (2024-07-23T15:31:26Z)
• LLM as a Complementary Optimizer to Gradient Descent: A Case Study in Prompt Tuning [69.95292905263393]
  We show that gradient-based and high-level LLMs can effectively collaborate a combined optimization framework. In this paper, we show that these complementary to each other and can effectively collaborate a combined optimization framework.
  arXiv  Detail & Related papers  (2024-05-30T06:24:14Z)
• One Token Can Help! Learning Scalable and Pluggable Virtual Tokens for Retrieval-Augmented Large Language Models [67.49462724595445]
  Retrieval-augmented generation (RAG) is a promising way to improve large language models (LLMs) We propose a novel method that involves learning scalable and pluggable virtual tokens for RAG.
  arXiv  Detail & Related papers  (2024-05-30T03:44:54Z)
• Efficient LLM Jailbreak via Adaptive Dense-to-sparse Constrained Optimization [46.98249466236357]
  Large language models (LLMs) are susceptible to jailbreaking attacks that can generate harmful content. This paper introduces a novel token-level attack method, Adaptive-to-Sparse Constrained Optimization (ADC), which effectively jailbreaks several open-source LLMs.
  arXiv  Detail & Related papers  (2024-05-15T06:11:24Z)
• An Empirical Study of Automated Vulnerability Localization with Large Language Models [21.84971967029474]
  Large Language Models (LLMs) have shown potential in various domains, yet their effectiveness in vulnerability localization remains underexplored. Our investigation encompasses 10+ leading LLMs suitable for code analysis, including ChatGPT and various open-source models. We explore the efficacy of these LLMs using 4 distinct paradigms: zero-shot learning, one-shot learning, discriminative fine-tuning, and generative fine-tuning.
  arXiv  Detail & Related papers  (2024-03-30T08:42:10Z)
• An Empirical Evaluation of Zeroth-Order Optimization Methods on AI-driven Molecule Optimization [78.36413169647408]
  We study the effectiveness of various ZO optimization methods for optimizing molecular objectives. We show the advantages of ZO sign-based gradient descent (ZO-signGD) We demonstrate the potential effectiveness of ZO optimization methods on widely used benchmark tasks from the Guacamol suite.
  arXiv  Detail & Related papers  (2022-10-27T01:58:10Z)
• Reparameterized Variational Divergence Minimization for Stable Imitation [57.06909373038396]
  We study the extent to which variations in the choice of probabilistic divergence may yield more performant ILO algorithms. We contribute a re parameterization trick for adversarial imitation learning to alleviate the challenges of the promising $f$-divergence minimization framework. Empirically, we demonstrate that our design choices allow for ILO algorithms that outperform baseline approaches and more closely match expert performance in low-dimensional continuous-control tasks.
  arXiv  Detail & Related papers  (2020-06-18T19:04:09Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.