Frozen CLIP Priors for Robust Self-Supervised Poisson Inverse Problems

• URL: http://arxiv.org/abs/2608.20524v1
• Date: Thu, 20 Aug 2026 19:37:52 GMT
• Title: Frozen CLIP Priors for Robust Self-Supervised Poisson Inverse Problems
• Abstract summary: This paper proposes an ADMM-inspired unrolled plug-and-play solver for Poisson inverse problems that decouples a closed-form data-consistency update from a parameter-efficient prior.<n> Experiments on Poisson CFA demosaicing and deblurring show competitive quality, improved robustness under shifts, and self-supervised performance approaching supervised training.
• Score: 19.52206071875139
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Self-supervised learning for imaging inverse problems is increasingly important in photon-limited settings, where acquiring clean ground truth is impractical and reconstruction must remain stable under dataset and acquisition shifts. This challenge is amplified under Poisson noise, whose signal-dependent statistics interact with sampling operators (e.g., CFA mosaicing). Meanwhile, foundation vision encoders trained at web scale offer distortion-invariant, content-related representations that generalize well across domains, suggesting a promising route to build priors that transfer beyond the training distribution without expensive fine-tuning. This paper proposes an ADMM-inspired unrolled plug-and-play solver for Poisson inverse problems that decouples a closed-form data-consistency update from a parameter-efficient prior. The prior is implemented as a lightweight decoder operating on frozen CLIP RN50 dense multi-scale features, adapting foundation representations with less trainable parameters. For self-supervision, the method integrates GR2R measurement-domain re-corruption with an Equivariant Imaging regularizer via virtual acquisitions. Experiments on Poisson CFA demosaicing and deblurring show competitive quality, improved robustness under shifts, and self-supervised performance approaching supervised training.

Related papers

• Continual Learning with Elastic Regularization and Synthetic Replay for Federated MLLM Fine-Tuning [15.445619998280243]
  Federated Continual Multimodal Learning (FedCMM) is a framework that embeds continual-learning safeguards into the federated optimization loop.<n>FedCMM consistently outperforms recent baselines on accuracy and backward transfer.
  arXiv  Detail & Related papers  (2026-07-13T19:40:30Z)
• A Step Towards Robust Unsupervised Domain Adaptation via Fine-Tuning and Reinforcement Learning [2.436631469537453]
  We propose textbfSFT+RL, a two-stage robust UDA framework that integratesSupervised Fine Tuning and Reinforcement Learning.<n>textbfSFT+RL achieves average improvements of textbf10.2% in clean accuracy and textbf15.8% in adversarial robustness across three datasets.
  arXiv  Detail & Related papers  (2026-07-03T20:56:12Z)
• MoASE++: Mixture of Activation Sparsity Experts with Domain-Adaptive On-policy Distillation for Continual Test Time Adaptation [65.068801413044]
  Continual test-time adaptation adapts a source-pretrained model to non-stationary, unlabeled target streams.<n>We introduce MoASE, a mixture-of-experts that disentangles domain-agnostic structure from domain-specific texture.
  arXiv  Detail & Related papers  (2026-05-18T01:52:12Z)
• Diffusion Controller: Framework, Algorithms and Parameterization [54.82539154511621]
  We present a unified control-theoretic view that casts reverse diffusion sampling as state-only control within (generalized) linearly-solvable Markov Decision Processes.<n>Under this framework, control acts by reweighting the pretrained reverse-time transition kernels, balancing terminal objectives against an $f$-divergence cost.<n>Experiments on Stable Diffusion v1.4 show consistent gains in preference-alignment win rates and improved quality-efficiency trade-offs.
  arXiv  Detail & Related papers  (2026-03-07T01:49:59Z)
• Robust Posterior Diffusion-based Sampling via Adaptive Guidance Scale [39.27744518020771]
  We propose an adaptive likelihood step-size strategy to guide the diffusion process for inverse-problem formulations.<n>The resulting approach, Adaptive Posterior diffusion Sampling (AdaPS), is hyper-free and improves reconstruction quality across diverse imaging tasks.
  arXiv  Detail & Related papers  (2025-11-23T14:37:59Z)
• Self-diffusion for Solving Inverse Problems [3.8870795921263728]
  We propose self-diffusion, a novel framework for solving inverse problems without relying on pretrained generative models.<n>Self-diffusion exploits the spectral bias of neural networks and modulates it through a scheduled noise process.<n>We demonstrate the effectiveness of our approach on a variety of linear inverse problems, showing that self-diffusion achieves competitive or superior performance compared to other methods.
  arXiv  Detail & Related papers  (2025-10-24T12:57:22Z)
• Solving Inverse Problems with FLAIR [68.87167940623318]
  We present FLAIR, a training-free variational framework that leverages flow-based generative models as prior for inverse problems.<n>Results on standard imaging benchmarks demonstrate that FLAIR consistently outperforms existing diffusion- and flow-based methods in terms of reconstruction quality and sample diversity.
  arXiv  Detail & Related papers  (2025-06-03T09:29:47Z)
• ADT: Tuning Diffusion Models with Adversarial Supervision [16.974169058917443]
  Diffusion models have achieved outstanding image generation by reversing a forward noising process to approximate true data distributions.<n>We propose Adrial Diffusion Tuning (ADT) to stimulate the inference process during optimization and align the final outputs with training data.<n>ADT features a siamese-network discriminator with a fixed pre-trained backbone and lightweight trainable parameters.
  arXiv  Detail & Related papers  (2025-04-15T17:37:50Z)
• Generalized Face Forgery Detection via Adaptive Learning for Pre-trained Vision Transformer [54.32283739486781]
  We present a textbfForgery-aware textbfAdaptive textbfVision textbfTransformer (FA-ViT) under the adaptive learning paradigm. FA-ViT achieves 93.83% and 78.32% AUC scores on Celeb-DF and DFDC datasets in the cross-dataset evaluation.
  arXiv  Detail & Related papers  (2023-09-20T06:51:11Z)
• Learning to Mask and Permute Visual Tokens for Vision Transformer Pre-Training [55.12082817901671]
  We propose a new self-supervised pre-training approach, named Masked and Permuted Vision Transformer (MaPeT)<n>MaPeT employs autoregressive and permuted predictions to capture intra-patch dependencies.<n>Our results demonstrate that MaPeT achieves competitive performance on ImageNet, compared to baselines and competitors under the same model setting.
  arXiv  Detail & Related papers  (2023-06-12T18:12:19Z)
• MAPS: A Noise-Robust Progressive Learning Approach for Source-Free Domain Adaptive Keypoint Detection [76.97324120775475]
  Cross-domain keypoint detection methods always require accessing the source data during adaptation. This paper considers source-free domain adaptive keypoint detection, where only the well-trained source model is provided to the target domain.
  arXiv  Detail & Related papers  (2023-02-09T12:06:08Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.