An Open-Source End-to-End FHE Implementation for Privacy-Preserving Llama 3 8B Inference

• URL: http://arxiv.org/abs/2609.12378v1
• Date: Fri, 11 Sep 2026 02:51:44 GMT
• Title: An Open-Source End-to-End FHE Implementation for Privacy-Preserving Llama 3 8B Inference
• Abstract summary: Cloud services typically require users to send prompts to a model provider, creating a privacy risk.<n>We present Odin, an FHE inference system that co-designs ciphertext packing and model execution for Llama.
• Score: 1.6799377888527687
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Cloud LLM services typically require users to send prompts to a model provider, creating a privacy risk. Fully homomorphic encryption (FHE) lets a server perform inference without decrypting the input, but representing data as ciphertexts adds storage and computational overhead. In CKKS-based LLM inference, the packing scheme maps logical tensors to ciphertexts and slots. It therefore determines the ciphertext count and the homomorphic cost of linear layers, and it constrains how data pass between linear layers, attention, and nonlinear computation. As models and sequences grow, inefficient layouts accumulate encoding, compute, and layout-conversion overhead. We present Odin, an FHE inference system that co-designs ciphertext packing and model execution for Llama. Starting from a THOR-style baseline whose bottleneck is weight encoding, Odin uses a feature-major cross-layer layout to unify residual connections and layer interfaces, and builds transient intra-operator layouts for linear projections and attention. This reduces redundant plaintext encoding of weights in wide projections. Within attention, QK^T produces scores that Softmax can consume directly, and PV consumes the resulting probabilities, avoiding intermediate repacking. For nonlinear ops, we use minimax polynomial approximation with input-range control and joint error allocation guided by model quality, reducing polynomial degree and multiplicative depth. To our knowledge, Odin is the first open-source end-to-end GPU CKKS implementation of Llama-3. With Llama-3-8B weights and a 128-token input, Odin evaluates all 32 Transformer layers on a single NVIDIA H100 80 GB GPU. Server-side end-to-end FHE evaluation takes 366.4 s and 58.9 GiB peak device memory. Under the same model, input, CKKS parameters, and hardware, THOR takes 1651.9 s, a 4.51x speedup.

Related papers

• HEAT: Faster Fully Homomorphic Inference via Approximations-Weights Co-Adaptation [56.14416807766864]
  Homomorphic encryption allows a server to run a language model directly on encrypted user prompts.<n>Existing approaches fix the iteration counts uniformly across the model rather than tailoring them to each site's error tolerance.<n>We introduce Homomorphic Encryption-Aware Training (HEAT), a fine-tuning method that makes the per-nonline iteration counts learnable.
  arXiv  Detail & Related papers  (2026-09-01T18:02:02Z)
• Hidden Decoding at Scale: Latent Computation Scaling for Large Language Models [75.80275843320511]
  We study whether an existing backbone can keep improving by allocating more computation to each token while leaving the Transformer backbone fixed.<n>We propose Hidden Decoding, a sequence-length scaling method applied during continued pretraining.
  arXiv  Detail & Related papers  (2026-07-09T07:37:59Z)
• Constrained Decoding for Diffusion Language Models via Efficient Inference over Finite Automata [57.27430779838529]
  We present an exact and tractable algorithm for sampling from the constrained mean-field posterior under any constraint expressible as a finite automaton.<n>The approach guarantees constraint satisfaction by construction, supports both greedy and sampling-based decoding, and is compatible with parallel and block-wise decoding.<n> Empirical evaluations on Dream-7B and LLaDA-8 show substantial accuracy gains across various tasks.
  arXiv  Detail & Related papers  (2026-07-08T05:48:57Z)
• FEnc$^2$: Unifying Data Packing for Efficient Private Inference via Convolution and Architecture-Aware Fragment Encoding [26.667496203770074]
  Homomorphic Encryption (FHE) enables privacy-preserving machine learning but incurs extreme computational and memory overhead.<n>FEnc2 is a unified and principled fragment-based encoding framework for CKKS-based private convolutional neural network inference.<n>It reduces ciphertext count and workload pressure on hardware, primitive-level optimizations such as NTT and keyswitch.
  arXiv  Detail & Related papers  (2026-06-15T07:55:55Z)
• Encrypted Neural Networks without Overflows [4.348418382807017]
  homomorphic encryption (FHE) enables private inference by evaluating neural networks on encrypted data.<n> CKKS scheme is the backbone of most efficient FHE implementations.<n>We propose a formal verification technique that computes certified bounds on the ranges of all neurons in the network.
  arXiv  Detail & Related papers  (2026-05-21T23:08:23Z)
• Resource-Efficient Iterative LLM-Based NAS with Feedback Memory [49.44875022114861]
  Neural Architecture Search (NAS) automates network design, but conventional methods demand substantial computational resources.<n>We propose a closed-loop pipeline leveraging large language models (LLMs) to iteratively generate, evaluate, and refine convolutional neural network architectures.
  arXiv  Detail & Related papers  (2026-03-12T16:00:22Z)
• Scaling up Privacy-Preserving ML: A CKKS Implementation of Llama-2-7B [20.74505614207065]
  homomorphic encryption (FHE) has emerged as a primary solution to provide non-interactive confidential language models (LLMs)<n>We propose an FHE-based private LLM inference solution that allows thousands of input tokens with only a part of them being encrypted.<n>We describe a CKKS-based end-to-end implementation of Llama-2-7B private inference for up to 4096 input tokens, of which the last 128 are encrypted.
  arXiv  Detail & Related papers  (2026-01-26T14:17:23Z)
• Network and Compiler Optimizations for Efficient Linear Algebra Kernels in Private Transformer Inference [2.725051134664174]
  Homomorphic Encryption (FHE) enables computations directly upon encrypted queries.<n>Running encrypted transformer inference is challenging as programmers must map standard kernels to the constrained instruction set provided by FHE.
  arXiv  Detail & Related papers  (2025-12-11T21:56:47Z)
• Efficient Decoding Methods for Language Models on Encrypted Data [32.58944595512403]
  Homomorphic encryption (HE) enables computation on encrypted data for secure inference.<n>Neural text generation requires decoding methods like argmax and sampling, which are non-polynomial and thus computationally expensive under encryption.<n>We introduce cutmax, an HE-friendly argmax algorithm that reduces cipher operations compared to prior methods, enabling practical greedy decoding under encryption.
  arXiv  Detail & Related papers  (2025-09-10T08:23:14Z)
• 70% Size, 100% Accuracy: Lossless LLM Compression for Efficient GPU Inference via Dynamic-Length Float [52.079202872069835]
  Large-scale AI models, such as Large Language Models (LLMs) and Diffusion Models (DMs) have grown rapidly in size.<n>We introduce Dynamic-Length Float (DFloat11), a compression framework that reduces LLM and DM size by 30% while preserving outputs that are bit-for-bit identical to the original model.
  arXiv  Detail & Related papers  (2025-04-15T22:38:38Z)
• Democratizing AI: Open-source Scalable LLM Training on GPU-based Supercomputers [65.35142508909892]
  We present a novel four-dimensional hybrid parallel algorithm implemented in a highly scalable, portable, open-source framework called AxoNN.<n>We demonstrate fine-tuning of a 405-billion parameter LLM using AxoNN on Frontier.
  arXiv  Detail & Related papers  (2025-02-12T06:05:52Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.