SpliTEE: Improving LLM Inference on Trusted Hardware with Differentially Private GPU Outsourcing

• URL: http://arxiv.org/abs/2609.15039v1
• Date: Mon, 14 Sep 2026 04:55:17 GMT
• Title: SpliTEE: Improving LLM Inference on Trusted Hardware with Differentially Private GPU Outsourcing
• Abstract summary: User prompts provided to large language models (LLMs) may contain sensitive or private information that can be misused by remotely deployed models.<n>One way to protect user prompts is to execute the LLM inside a trusted execution environment (TEE)<n>We implement our architecture using Intel TDX and evaluate it with two LLMs: Llama-3.2-3B and Qwen3-4B.
• Score: 0.8446353882843086
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: User prompts provided to large language models (LLMs) may contain sensitive or private information that can be misused by remotely deployed models, such as through inadvertent memorization during retraining. One way to protect user prompts is to execute the LLM inside a trusted execution environment (TEE), with the guarantee that the service provider has no access to computations performed within or information exchanged with the TEE. However, current TEEs are primarily CPU-based and significantly slower than GPUs optimized for LLM inference. To circumvent this, Tramer and Boneh (2019) proposed Slalom, which splits neural network inference between a TEE and an untrusted GPU and encrypts intermediate inputs sent to the GPU. We extend this split-inference architecture to LLM inference and instead protect intermediate inputs using differential privacy. We show that masking intermediate representations is necessary by showing that a prompt-reconstruction attack can recover prompts from these representations with nearly 80% accuracy. Our main contribution is a global sensitivity analysis of key LLM functions, which bounds the required scale of differentially private noise. Unlike encryption, differential privacy avoids quantization, allowing the LLM to remain in the floating-point domain. We also derive an upper bound on floating-point error from masking and noise cancellation in the TEE as a function of the privacy parameter epsilon. We implement our architecture using Intel TDX and evaluate it with two LLMs: Llama-3.2-3B and Qwen3-4B. Our split execution is nearly twice as fast as fully CPU-based inference inside TDX and 5-15 seconds faster than encryption-based Slalom while achieving higher accuracy. Finally, we demonstrate that prompt reconstruction, even with knowledge of the differential privacy mechanism, cannot recover more information than is contained in an unrelated prompt.

Related papers

• LLM Inference in a Flash! [66.43896490006965]
  Large Language Models (LLMs) have shown impressive capabilities across a range of natural language processing tasks.<n>We present an end-to-end integer-only quantization approach to eliminate expensive floating-point computations.
  arXiv  Detail & Related papers  (2026-09-14T18:04:30Z)
• Communication-Efficient Verifiable Attention for LLM Inference [10.639156521036965]
  This paper presents Communication-efficient TEE-GPU Attention (textscVeriAttn) for accelerating verifiable LLM inference.<n>textscVeriAttn offloads both linear and non-linear computations of attention to the GPU, while TEE performs verification.<n>For decoding, when the key-value cache exceeds available GPU memory, textscVeriAttn partitions attention across TEE and GPU to reduce repeated key-value transfers.
  arXiv  Detail & Related papers  (2026-06-15T07:50:15Z)
• TensorCommitments: A Lightweight Verifiable Inference for Language Models [8.140218773902044]
  We propose a tensor-native proof-of-inference scheme for large language models (LLMs)<n>Commitments (TCs) bind the inference to a commitment, an irreversible tag that breaks under tampering.<n>For LLaMA2, TC adds only 0.97% prover and 0.12% verifier time over while improving robustness to tailored LLM attacks by up to 48% over the best prior work requiring a verifier GPU.
  arXiv  Detail & Related papers  (2026-02-13T05:23:31Z)
• Prism: Efficient Test-Time Scaling via Hierarchical Search and Self-Verification for Discrete Diffusion Language Models [96.0074341403456]
  Inference-time compute has re-emerged as a practical way to improve LLM reasoning.<n>Most test-time scaling (TTS) algorithms rely on autoregressive decoding.<n>We propose Prism, an efficient TTS framework for dLLMs.
  arXiv  Detail & Related papers  (2026-02-02T09:14:51Z)
• 1 + 1 > 2: Detector-Empowered Video Large Language Model for Spatio-Temporal Grounding and Reasoning [53.28271278708241]
  We present a Detector-Empowered Video LLM, short for DEViL.<n> DEViL couples a Video LLM with an open-vocabulary detector (OVD)<n>Unlike tokens that merely serve as spatial prompts or segmentor switches, the RST functions as both a control signal and a replacement for the OVD's text embedding.
  arXiv  Detail & Related papers  (2025-12-07T06:11:15Z)
• TZ-LLM: Protecting On-Device Large Language Models with Arm TrustZone [8.538298365840877]
  Large Language Models (LLMs) deployed on mobile devices offer benefits like user privacy and reduced network latency, but introduce a significant security risk.<n>We propose a system design for protecting on-device LLMs using Arm Trusted Execution Environment (TEE), TrustZone.
  arXiv  Detail & Related papers  (2025-11-17T18:59:20Z)
• Confidential LLM Inference: Performance and Cost Across CPU and GPU TEEs [16.49726695421423]
  Large Language Models (LLMs) are increasingly deployed on converged Cloud and High-Performance Computing infrastructure.<n>As LLMs handle confidential inputs, their heightened security requirements slow adoption in privacy-sensitive sectors such as healthcare and finance.<n>We propose Trusted Execution Environments (TEEs) as a solution for securing end-to-end LLM inference.
  arXiv  Detail & Related papers  (2025-09-23T10:36:47Z)
• Towards Confidential and Efficient LLM Inference with Dual Privacy Protection [11.22744810136105]
  This paper proposes CMIF, a Confidential and efficient Model Inference Framework.<n> CMIF confidentially deploys the embedding layer in the client-side TEE and subsequent layers on GPU servers.<n>It optimize the Report-Noisy-Max mechanism to protect sensitive inputs with a slight decrease in model performance.
  arXiv  Detail & Related papers  (2025-09-11T01:54:13Z)
• DP-Fusion: Token-Level Differentially Private Inference for Large Language Models [51.71591819896191]
  Large language models (LLMs) do not preserve privacy at inference-time.<n> DP-Fusion provably bounds the influence a set of tokens in the context can have on the LLM's output.<n>We show that our method creates token-level provably privatized documents with substantially improved theoretical and empirical privacy.
  arXiv  Detail & Related papers  (2025-07-06T20:49:39Z)
• InfiniteHiP: Extending Language Model Context Up to 3 Million Tokens on a Single GPU [48.105361428245736]
  We introduce InfiniteHiP, an inference framework for large language models (LLMs)<n>We dynamically eliminate irrelevant context tokens through a modular hierarchical token pruning algorithm.<n>Our framework achieves an 18.95x speedup in attention decoding for a 1 million token context without requiring additional training.
  arXiv  Detail & Related papers  (2025-02-13T02:52:01Z)
• FusionAI: Decentralized Training and Deploying LLMs with Massive Consumer-Level GPUs [57.12856172329322]
  We envision a decentralized system unlocking the potential vast untapped consumer-level GPU. This system faces critical challenges, including limited CPU and GPU memory, low network bandwidth, the variability of peer and device heterogeneity.
  arXiv  Detail & Related papers  (2023-09-03T13:27:56Z)
• Compress, Then Prompt: Improving Accuracy-Efficiency Trade-off of LLM Inference with Transferable Prompt [96.24800696597707]
  We introduce a new perspective to optimize this trade-off by prompting compressed models. We propose a soft prompt learning method where we expose the compressed model to the prompt learning process. Our experimental analysis suggests our soft prompt strategy greatly improves the performance of the 8x compressed LLaMA-7B model.
  arXiv  Detail & Related papers  (2023-05-17T20:45:13Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.