Sketch-Guided Constrained Decoding for Boosting Blackbox Large Language Models without Logit Access

• URL: http://arxiv.org/abs/2401.09967v4
• Date: Mon, 22 Jul 2024 01:05:29 GMT
• Title: Sketch-Guided Constrained Decoding for Boosting Blackbox Large Language Models without Logit Access
• Authors: Saibo Geng, Berkay Döner, Chris Wendler, Martin Josifoski, Robert West,
• Abstract summary: We introduce sketch-guided constrained decoding (SGCD), a novel approach to constrained decoding for blackbox large language models (LLMs) SGCD operates without access to the logits of the blackbox LLM. We demonstrate the efficacy of SGCD through experiments in closed information extraction and constituency parsing.
• Score: 14.283269607549892
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Constrained decoding, a technique for enforcing constraints on language model outputs, offers a way to control text generation without retraining or architectural modifications. Its application is, however, typically restricted to models that give users access to next-token distributions (usually via softmax logits), which poses a limitation with blackbox large language models (LLMs). This paper introduces sketch-guided constrained decoding (SGCD), a novel approach to constrained decoding for blackbox LLMs, which operates without access to the logits of the blackbox LLM. SGCD utilizes a locally hosted auxiliary model to refine the output of an unconstrained blackbox LLM, effectively treating this initial output as a "sketch" for further elaboration. This approach is complementary to traditional logit-based techniques and enables the application of constrained decoding in settings where full model transparency is unavailable. We demonstrate the efficacy of SGCD through experiments in closed information extraction and constituency parsing, showing how it enhances the utility and flexibility of blackbox LLMs for complex NLP tasks.

Related papers

• Learning on LLM Output Signatures for gray-box LLM Behavior Analysis [52.81120759532526]
  Large Language Models (LLMs) have achieved widespread adoption, yet our understanding of their behavior remains limited. We develop a transformer-based approach to process that theoretically guarantees approximation of existing techniques. Our approach achieves superior performance on hallucination and data contamination detection in gray-box settings.
  arXiv  Detail & Related papers  (2025-03-18T09:04:37Z)
• The Challenge of Identifying the Origin of Black-Box Large Language Models [34.284190160785336]
  Third parties can customize large language models (LLMs) through fine-tuning and offer only black-box API access. This practice not only exacerbates unfair competition, but also violates licensing agreements. We propose PlugAE, which proactively plugs adversarial token embeddings into the LLM for tracing and identification.
  arXiv  Detail & Related papers  (2025-03-06T11:30:32Z)
• Logits are All We Need to Adapt Closed Models [15.227768874282834]
  Many commercial Large Language Models (LLMs) are often closed-source, limiting developers to prompt tuning for aligning content generation with specific applications. We argue that if such access were available, it would enable more powerful adaptation techniques beyond prompt engineering. We propose a token-level probability reweighting framework that steers black-box LLMs toward application-specific content generation.
  arXiv  Detail & Related papers  (2025-02-03T22:24:22Z)
• SWIFT: On-the-Fly Self-Speculative Decoding for LLM Inference Acceleration [10.970637831760136]
  Speculative decoding (SD) has emerged as a widely used paradigm to accelerate the inference of large language models (LLMs) We introduce SWIFT, an on-the-fly self-speculative decoding algorithm that adaptively selects intermediate layers of LLMs to skip during inference. We show that SWIFT can achieve over a 1.3x-1.6x speedup while preserving the original distribution of the generated text.
  arXiv  Detail & Related papers  (2024-10-09T14:15:30Z)
• Open-domain Implicit Format Control for Large Language Model Generation [52.83173553689678]
  We introduce a novel framework for controlled generation in large language models (LLMs) This study investigates LLMs' capabilities to follow open-domain, one-shot constraints and replicate the format of the example answers. We also develop a dataset collection methodology for supervised fine-tuning that enhances the open-domain format control of LLMs without degrading output quality.
  arXiv  Detail & Related papers  (2024-08-08T11:51:45Z)
• Unlocking Tokens as Data Points for Generalization Bounds on Larger Language Models [79.70436109672599]
  We derive non-vacuous generalization bounds for large language models as large as LLaMA2-70B. Our work achieves the first non-vacuous bounds for models that are deployed in practice and generate high-quality text.
  arXiv  Detail & Related papers  (2024-07-25T16:13:58Z)
• Exploring the Role of Large Language Models in Prompt Encoding for Diffusion Models [42.891427362223176]
  Large language models (LLMs) based on decoder-only transformers have demonstrated superior text understanding capabilities. We propose a novel framework to fully harness the capabilities of LLMs. We further design an LLM-Infused Diffusion Transformer (LI-DiT) based on the framework.
  arXiv  Detail & Related papers  (2024-06-17T17:59:43Z)
• DALD: Improving Logits-based Detector without Logits from Black-box LLMs [56.234109491884126]
  Large Language Models (LLMs) have revolutionized text generation, producing outputs that closely mimic human writing. We present Distribution-Aligned LLMs Detection (DALD), an innovative framework that redefines the state-of-the-art performance in black-box text detection. DALD is designed to align the surrogate model's distribution with that of unknown target LLMs, ensuring enhanced detection capability and resilience against rapid model iterations.
  arXiv  Detail & Related papers  (2024-06-07T19:38:05Z)
• Text-like Encoding of Collaborative Information in Large Language Models for Recommendation [58.87865271693269]
  We introduce BinLLM, a novel method to seamlessly integrate collaborative information with Large Language Models for Recommendation (LLMRec) BinLLM converts collaborative embeddings from external models into binary sequences. BinLLM provides options to compress the binary sequence using dot-decimal notation to avoid excessively long lengths.
  arXiv  Detail & Related papers  (2024-06-05T12:45:25Z)
• FFN-SkipLLM: A Hidden Gem for Autoregressive Decoding with Adaptive Feed Forward Skipping [49.66872823080736]
  Autoregressive Large Language Models (e.g., LLaMa, GPTs) are omnipresent achieving remarkable success in language understanding and generation. To mitigate overload incurred during generation, several early-exit and layer-dropping strategies have been proposed. We propose FFN-SkipLLM, which is an input-adaptive feed-forward skipping strategy.
  arXiv  Detail & Related papers  (2024-04-05T02:35:43Z)
• Guiding LLMs The Right Way: Fast, Non-Invasive Constrained Generation [7.687678490751105]
  We present a novel decoding algorithm, DOMINO, that can enforce constraints in a fully subword-aligned fashion, while leveraging pre-computation and speculative decoding to achieve virtually no overhead and in some cases even almost 2$times$ speedup over unconstrained decoding -- thereby outperforming existing approaches by a wide margin.
  arXiv  Detail & Related papers  (2024-02-07T13:36:02Z)
• Simultaneous Machine Translation with Large Language Models [51.470478122113356]
  We investigate the possibility of applying Large Language Models to SimulMT tasks. We conducted experiments using the textttLlama2-7b-chat model on nine different languages from the MUST-C dataset. The results show that LLM outperforms dedicated MT models in terms of BLEU and LAAL metrics.
  arXiv  Detail & Related papers  (2023-09-13T04:06:47Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.