OD-Stega: LLM-Based Near-Imperceptible Steganography via Optimized Distributions

• URL: http://arxiv.org/abs/2410.04328v1
• Date: Sun, 6 Oct 2024 01:30:45 GMT
• Title: OD-Stega: LLM-Based Near-Imperceptible Steganography via Optimized Distributions
• Authors: Yu-Shin Huang, Peter Just, Krishna Narayanan, Chao Tian,
• Abstract summary: We consider coverless steganography where a Large Language Model drives an arithmetic coding decoder to generate stego-texts. An efficient method should embed secret message bits in as few language tokens as possible, while still keeping the stego-text natural and fluent.
• Score: 7.611860976107124
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We consider coverless steganography where a Large Language Model (LLM) drives an arithmetic coding decoder to generate stego-texts. An efficient method should embed secret message bits in as few language tokens as possible, while still keeping the stego-text natural and fluent. We show that on the individual token level, this problem is mathematically equivalent to maximizing the entropy of a replacement probability distribution of the next token generation, subject to a constraint on the KL divergence between the chosen probability distribution and the original distribution given by the LLM. A closed-form solution is provided for the optimization problem, which can be computed efficiently. Several important practical issues are also tackled: 1) An often-overlooked tokenization mismatch issue is resolved with a simple prompt selection approach, 2) The combination of the optimized distribution and the vocabulary truncation technique is considered, and 3) The combination of the optimized distribution with other sequence-level selection heuristics to further enhance the efficiency and reliability is studied.

Related papers

• Minimax and Communication-Efficient Distributed Best Subset Selection with Oracle Property [0.358439716487063]
  The explosion of large-scale data has outstripped the processing capabilities of single-machine systems. Traditional approaches to distributed inference often struggle with achieving true sparsity in high-dimensional datasets. We propose a novel, two-stage, distributed best subset selection algorithm to address these issues.
  arXiv  Detail & Related papers  (2024-08-30T13:22:08Z)
• Graph-Structured Speculative Decoding [52.94367724136063]
  Speculative decoding has emerged as a promising technique to accelerate the inference of Large Language Models. We introduce an innovative approach utilizing a directed acyclic graph (DAG) to manage the drafted hypotheses. We observe a remarkable speedup of 1.73$times$ to 1.96$times$, significantly surpassing standard speculative decoding.
  arXiv  Detail & Related papers  (2024-07-23T06:21:24Z)
• Stabilized Proximal-Point Methods for Federated Optimization [20.30761752651984]
  Best-known communication complexity among non-accelerated algorithms is achieved by DANE, a distributed proximal-point algorithm. Inspired by the hybrid-projection proximal-point method, we propose a novel distributed algorithm S-DANE. We show that S-DANE achieves the best-known communication complexity while still enjoying good local computation efficiency as S-DANE.
  arXiv  Detail & Related papers  (2024-07-09T17:56:29Z)
• Quantization Avoids Saddle Points in Distributed Optimization [1.579622195923387]
  Distributed non optimization underpins key functionalities of numerous distributed systems. The aim of this paper is to prove that it can effectively escape saddle points convergence to a second-order stationary point convergence. With an easily adjustable quantization, the approach allows a user control to aggressively reduce communication overhead.
  arXiv  Detail & Related papers  (2024-03-15T15:58:20Z)
• Synthesizing Tight Privacy and Accuracy Bounds via Weighted Model Counting [5.552645730505715]
  Two core challenges are finding expressive, compact, and efficient encodings of distributions of DP algorithms. We address the first challenge by developing a method for tight privacy and accuracy bound synthesis. We develop a framework for leveraging inherent symmetries in DP algorithms.
  arXiv  Detail & Related papers  (2024-02-26T19:29:46Z)
• Amortizing intractable inference in large language models [56.92471123778389]
  We use amortized Bayesian inference to sample from intractable posterior distributions. We empirically demonstrate that this distribution-matching paradigm of LLM fine-tuning can serve as an effective alternative to maximum-likelihood training. As an important application, we interpret chain-of-thought reasoning as a latent variable modeling problem.
  arXiv  Detail & Related papers  (2023-10-06T16:36:08Z)
• Learning Distributions via Monte-Carlo Marginalization [9.131712404284876]
  We propose a novel method to learn intractable distributions from their samples. The Monte-Carlo Marginalization (MCMarg) is proposed to address this issue. The proposed approach is a powerful tool to learn complex distributions and the entire process is differentiable.
  arXiv  Detail & Related papers  (2023-08-11T19:08:06Z)
• Fast Computation of Optimal Transport via Entropy-Regularized Extragradient Methods [75.34939761152587]
  Efficient computation of the optimal transport distance between two distributions serves as an algorithm that empowers various applications. This paper develops a scalable first-order optimization-based method that computes optimal transport to within $varepsilon$ additive accuracy.
  arXiv  Detail & Related papers  (2023-01-30T15:46:39Z)
• Efficient and Flexible Sublabel-Accurate Energy Minimization [62.50191141358778]
  We address the problem of minimizing a class of energy functions consisting of data and smoothness terms. Existing continuous optimization methods can find sublabel-accurate solutions, but they are not efficient for large label spaces. We propose an efficient sublabel-accurate method that utilizes the best properties of both continuous and discrete models.
  arXiv  Detail & Related papers  (2022-06-20T06:58:55Z)
• Adaptive Sampling for Best Policy Identification in Markov Decision Processes [79.4957965474334]
  We investigate the problem of best-policy identification in discounted Markov Decision (MDPs) when the learner has access to a generative model. The advantages of state-of-the-art algorithms are discussed and illustrated.
  arXiv  Detail & Related papers  (2020-09-28T15:22:24Z)
• Combining Deep Learning and Optimization for Security-Constrained Optimal Power Flow [94.24763814458686]
  Security-constrained optimal power flow (SCOPF) is fundamental in power systems. Modeling of APR within the SCOPF problem results in complex large-scale mixed-integer programs. This paper proposes a novel approach that combines deep learning and robust optimization techniques.
  arXiv  Detail & Related papers  (2020-07-14T12:38:21Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.