Peregrine: One-Shot Fine-Tuning for FHE Inference of General Deep CNNs

• URL: http://arxiv.org/abs/2511.18976v1
• Date: Mon, 24 Nov 2025 10:47:39 GMT
• Title: Peregrine: One-Shot Fine-Tuning for FHE Inference of General Deep CNNs
• Authors: Huaming Ling, Ying Wang, Si Chen, Junfeng Fan,
• Abstract summary: We address two fundamental challenges in adapting general deep CNNs for FHE-based inference.<n>The first is approximating non-linear activations such as ReLU with low-degrees while minimizing accuracy degradation.<n>The second is overcoming the cipher capacity barrier that constrains high-resolution image processing on FHE inference.
• Score: 5.719717928243504
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We address two fundamental challenges in adapting general deep CNNs for FHE-based inference: approximating non-linear activations such as ReLU with low-degree polynomials while minimizing accuracy degradation, and overcoming the ciphertext capacity barrier that constrains high-resolution image processing on FHE inference. Our contributions are twofold: (1) a single-stage fine-tuning (SFT) strategy that directly converts pre-trained CNNs into FHE-friendly forms using low-degree polynomials, achieving competitive accuracy with minimal training overhead; and (2) a generalized interleaved packing (GIP) scheme that is compatible with feature maps of virtually arbitrary spatial resolutions, accompanied by a suite of carefully designed homomorphic operators that preserve the GIP-form encryption throughout computation. These advances enable efficient, end-to-end FHE inference across diverse CNN architectures. Experiments on CIFAR-10, ImageNet, and MS COCO demonstrate that the FHE-friendly CNNs obtained via our SFT strategy achieve accuracy comparable to baselines using ReLU or SiLU activations. Moreover, this work presents the first demonstration of FHE-based inference for YOLO architectures in object detection leveraging low-degree polynomial activations.

Related papers

• Knowledge-Informed Neural Network for Complex-Valued SAR Image Recognition [51.03674130115878]
  We introduce the Knowledge-Informed Neural Network (KINN), a lightweight framework built upon a novel "compression-aggregation-compression" architecture.<n>KINN establishes a state-of-the-art in parameter-efficient recognition, offering exceptional generalization in data-scarce and out-of-distribution scenarios.
  arXiv  Detail & Related papers  (2025-10-23T07:12:26Z)
• PAPER: Privacy-Preserving ResNet Models using Low-Degree Polynomial Approximations and Structural Optimizations on Leveled FHE [5.819818547073678]
  Recent work has made non-interactive privacy-preserving inference more practical by running deep Convolution Neural Network (CNN) with Fully Homomorphic Encryption (FHE)<n>They also depend on high-degree approximations of non-linear activations, which increase multiplicative depth and reduce accuracy by 2-5% compared to plaintext ReLU models.<n>In this work, we focus on ResNets, a widely adopted benchmark architecture in privacy-preserving inference, and close the accuracy gap between their FHE non-interactive models and counterparts.
  arXiv  Detail & Related papers  (2025-09-26T19:10:23Z)
• Double-Shot 3D Shape Measurement with a Dual-Branch Network for Structured Light Projection Profilometry [14.749887303860717]
  We propose a dual-branch Convolutional Neural Network (CNN)-Transformer network (PDCNet) to process different structured light (SL) modalities.<n>Within PDCNet, a Transformer branch is used to capture global perception in the fringe images, while a CNN branch is designed to collect local details in the speckle images.<n>Our method can reduce fringe order ambiguity while producing high-accuracy results on self-made datasets.
  arXiv  Detail & Related papers  (2024-07-19T10:49:26Z)
• Characterizing the Training Dynamics of Private Fine-tuning with Langevin diffusion [37.98959061338993]
  We show that differentially private full fine-tuning (DP-FFT) can distort pre-trained backbone features based on both theoretical and empirical results.<n>We prove that a sequential fine-tuning strategy can mitigate the feature distortion.
  arXiv  Detail & Related papers  (2024-02-29T07:01:48Z)
• Efficient and Flexible Neural Network Training through Layer-wise Feedback Propagation [49.44309457870649]
  Layer-wise Feedback feedback (LFP) is a novel training principle for neural network-like predictors.<n>LFP decomposes a reward to individual neurons based on their respective contributions.<n>Our method then implements a greedy reinforcing approach helpful parts of the network and weakening harmful ones.
  arXiv  Detail & Related papers  (2023-08-23T10:48:28Z)
• Training Large Scale Polynomial CNNs for E2E Inference over Homomorphic Encryption [33.35896071292604]
  Training large-scale CNNs that during inference can be run under Homomorphic Encryption (HE) is challenging. We provide a novel training method for large CNNs such as ResNet-152 and ConvNeXt models.
  arXiv  Detail & Related papers  (2023-04-26T20:41:37Z)
• Iterative Soft Shrinkage Learning for Efficient Image Super-Resolution [91.3781512926942]
  Image super-resolution (SR) has witnessed extensive neural network designs from CNN to transformer architectures. This work investigates the potential of network pruning for super-resolution iteration to take advantage of off-the-shelf network designs and reduce the underlying computational overhead. We propose a novel Iterative Soft Shrinkage-Percentage (ISS-P) method by optimizing the sparse structure of a randomly network at each and tweaking unimportant weights with a small amount proportional to the magnitude scale on-the-fly.
  arXiv  Detail & Related papers  (2023-03-16T21:06:13Z)
• Implicit Stochastic Gradient Descent for Training Physics-informed Neural Networks [51.92362217307946]
  Physics-informed neural networks (PINNs) have effectively been demonstrated in solving forward and inverse differential equation problems. PINNs are trapped in training failures when the target functions to be approximated exhibit high-frequency or multi-scale features. In this paper, we propose to employ implicit gradient descent (ISGD) method to train PINNs for improving the stability of training process.
  arXiv  Detail & Related papers  (2023-03-03T08:17:47Z)
• Faster Adaptive Federated Learning [84.38913517122619]
  Federated learning has attracted increasing attention with the emergence of distributed data. In this paper, we propose an efficient adaptive algorithm (i.e., FAFED) based on momentum-based variance reduced technique in cross-silo FL.
  arXiv  Detail & Related papers  (2022-12-02T05:07:50Z)
• An Adaptive and Stability-Promoting Layerwise Training Approach for Sparse Deep Neural Network Architecture [0.0]
  This work presents a two-stage adaptive framework for developing deep neural network (DNN) architectures that generalize well for a given training data set. In the first stage, a layerwise training approach is adopted where a new layer is added each time and trained independently by freezing parameters in the previous layers. We introduce a epsilon-delta stability-promoting concept as a desirable property for a learning algorithm and show that employing manifold regularization yields a epsilon-delta stability-promoting algorithm.
  arXiv  Detail & Related papers  (2022-11-13T09:51:16Z)
• Deep Online Correction for Monocular Visual Odometry [23.124372375670887]
  We propose a novel deep online correction (DOC) framework for monocular visual odometry. depth maps and initial poses are obtained from convolutional neural networks (CNNs) trained in self-supervised manners. Our method achieves outstanding performance with relative transform error (RTE) = 2.0% on KITTI Odometry benchmark for Seq. 09.
  arXiv  Detail & Related papers  (2021-03-18T05:55:51Z)
• Implicit Convex Regularizers of CNN Architectures: Convex Optimization of Two- and Three-Layer Networks in Polynomial Time [70.15611146583068]
  We study training of Convolutional Neural Networks (CNNs) with ReLU activations. We introduce exact convex optimization with a complexity with respect to the number of data samples, the number of neurons, and data dimension.
  arXiv  Detail & Related papers  (2020-06-26T04:47:20Z)
• What Deep CNNs Benefit from Global Covariance Pooling: An Optimization Perspective [102.37204254403038]
  We make an attempt to understand what deep CNNs benefit from GCP in a viewpoint of optimization. We show that GCP can make the optimization landscape more smooth and the gradients more predictive. We conduct extensive experiments using various deep CNN models on diversified tasks, and the results provide strong support to our findings.
  arXiv  Detail & Related papers  (2020-03-25T07:00:45Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.