ZOTTA: Test-Time Adaptation with Gradient-Free Zeroth-Order Optimization

• URL: http://arxiv.org/abs/2603.14254v1
• Date: Sun, 15 Mar 2026 07:21:01 GMT
• Title: ZOTTA: Test-Time Adaptation with Gradient-Free Zeroth-Order Optimization
• Abstract summary: Test-time adaptation aims to improve model robustness under distribution shifts.<n>Most existing methods rely on backpropagation (BP), which is computationally costly and incompatible with non-differentiable models.<n>We propose ZOTTA, a fully BP-free TTA framework that performs efficient adaptation using only forward passes via Zeroth-Order Optimization.
• Score: 29.670784134543027
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Test-time adaptation (TTA) aims to improve model robustness under distribution shifts by adapting to unlabeled test data, but most existing methods rely on backpropagation (BP), which is computationally costly and incompatible with non-differentiable models such as quantized models, limiting practical deployment on numerous edge devices. Recent BP-free approaches alleviate overhead but remain either architecture-specific or limited in optimization capacity to handle high-dimensional models. We propose ZOTTA, a fully BP-free TTA framework that performs efficient adaptation using only forward passes via Zeroth-Order Optimization (ZOO). While ZOO is theoretically appealing, naive application leads to slow convergence under high-dimensional parameter spaces and unstable optimization due to the lack of labels. ZOTTA overcomes these challenges through 1) Distribution-Robust Layer Selection, which automatically identifies and freezes layers that already extract distribution-invariant features, updating only domain-sensitive layers to reduce the optimization dimensionality and accelerate convergence; 2) Spatial Feature Aggregation Alignment, which stabilizes ZOO by aligning globally aggregated spatial features between source and target to reduce gradient variance. Together, these components enable architecture-agnostic and stable BP-free adaptation. Extensive experiments on ImageNet-C/R/Sketch/A show that ZOTTA outperforms or matches BP-based methods, e.g., it reduces memory usage by 84% and improves accuracy by 3.9% over SAR on ImageNet-C.

Related papers

• Curvature-Aware Zeroth-Order Optimization for Memory-Efficient Test-Time Adaptation [15.216413521560705]
  Test-time adaptation (TTA) aims to enhance the cross-domain performance of pre-trained models by adapting to unlabeled test data.<n>ZO methods rely only on forward computation, which can largely reduce the complexity and memory overhead of on-device deployment.<n>We propose an improved ZO method to substantially boost the performance of ZO optimization based TTA.
  arXiv  Detail & Related papers  (2026-08-12T17:17:40Z)
• DUET: Decentralized Bilevel Optimization without Lower-Level Strong Convexity [77.51981761667302]
  We introduce a new single-loop DBO algorithm called diminishing quadratically-regularized bilevel decentralized optimization (DUET)<n>DUET eliminates the need for LLSC by introducing a diminishing quadratic regularization to the lower-level (LL) objective.<n>To the best of our knowledge, this is the first work to tackle DBO without LLSC under decentralized settings with data heterogeneity.
  arXiv  Detail & Related papers  (2026-06-19T06:45:13Z)
• EVA-0: Test-Time Model Evolution with Only Two Forward Passes per Sample [55.284129005947484]
  We study test-time model evolution under a strict two-forward budget.<n>We reveal three key obstacles in zeroth-order test-time optimization.<n>We propose EVA-0, a minimal zeroth-order adaptation framework.
  arXiv  Detail & Related papers  (2026-05-15T09:26:02Z)
• Universally Empowering Zeroth-Order Optimization via Adaptive Layer-wise Sampling [43.822941944402544]
  Zeroth-Order optimization presents a promising memory-efficient paradigm for fine-tuning Large Language Models.<n>However, its practical adoption is severely constrained by slow wall-clock convergence and high estimation variance.<n>We propose AdaLeZO, an Adaptive Layer-wise ZO optimization framework.
  arXiv  Detail & Related papers  (2026-04-20T13:37:31Z)
• CurvZO: Adaptive Curvature-Guided Sparse Zeroth-Order Optimization for Efficient LLM Fine-Tuning [15.930478833291827]
  Fine-tuning large language models (LLMs) with backpropagation achieves high performance but incurs substantial memory overhead.<n>We propose CurvZO, which tracks curvature signals online from scalar ZO feedback.<n>We show that CurvZO consistently improves fine-tuning performance and reduces training time over ZO baselines.
  arXiv  Detail & Related papers  (2026-03-23T09:13:45Z)
• DRAFTO: Decoupled Reduced-space and Adaptive Feasibility-repair Trajectory Optimization for Robotic Manipulators [4.0407133618465005]
  This paper introduces a new algorithm for trajectory optimization, Decoupled Reduced-space and Adaptive Feasibility-repair Trajectory Optimization (DRAFTO)<n>To reduce the number of repeated constrained optimizations while handling joint-limit feasibility, the optimization is decoupled into a reduced-space Gauss-Newton (GN) descent.<n>The results of our benchmark tests against optimization-based planners, such as CHOMP, TrajOpt, GPMP2, and FACTO, validate the high efficiency and reliability across diverse scenarios and tasks.
  arXiv  Detail & Related papers  (2026-03-10T20:24:42Z)
• FOZO: Forward-Only Zeroth-Order Prompt Optimization for Test-Time Adaptation [9.28697795097814]
  Test-Time Adaptation is essential for enabling deep learning models to handle real-world data distribution shifts.<n>Backpropagation-based methods are not suitable for low-end deployment devices.<n>We propose Forward-Only Zeroth-Order Optimization (FOZO), a novel and practical backpropagation-free paradigm for TTA.
  arXiv  Detail & Related papers  (2026-03-05T02:12:48Z)
• Parallel Diffusion Solver via Residual Dirichlet Policy Optimization [88.7827307535107]
  Diffusion models (DMs) have achieved state-of-the-art generative performance but suffer from high sampling latency due to their sequential denoising nature.<n>Existing solver-based acceleration methods often face significant image quality degradation under a low-dimensional budget.<n>We propose the Ensemble Parallel Direction solver (dubbed as EPD-EPr), a novel ODE solver that mitigates these errors by incorporating multiple gradient parallel evaluations in each step.
  arXiv  Detail & Related papers  (2025-12-28T05:48:55Z)
• Rethinking Autoregressive Models for Lossless Image Compression via Hierarchical Parallelism and Progressive Adaptation [75.58269386927076]
  Autoregressive (AR) models are often dismissed as impractical due to prohibitive computational cost.<n>This work re-thinks this paradigm, introducing a framework built on hierarchical parallelism and progressive adaptation.<n> Experiments on diverse datasets (natural, satellite, medical) validate that our method achieves new state-of-the-art compression.
  arXiv  Detail & Related papers  (2025-11-14T06:27:58Z)
• HyperTTA: Test-Time Adaptation for Hyperspectral Image Classification under Distribution Shifts [28.21559601586271]
  HyperTTA (Test-Time Adaptable Transformer for Hyperspectral Degradation) is a unified framework that enhances model robustness under diverse degradation conditions.<n>Test-time adaptation strategy, the Confidence-aware Entropy-minimized LayerNorm Adapter (CELA), dynamically updates only the affine parameters of LayerNorm layers.<n>Experiments on two benchmark datasets demonstrate that HyperTTA outperforms state-of-the-art baselines across a wide range of degradation scenarios.
  arXiv  Detail & Related papers  (2025-09-10T09:31:37Z)
• Adaptive Deadline and Batch Layered Synchronized Federated Learning [66.93447103966439]
  Federated learning (FL) enables collaborative model training across distributed edge devices while preserving data privacy, and typically operates in a round-based synchronous manner.<n>We propose ADEL-FL, a novel framework that jointly optimize per-round deadlines and user-specific batch sizes for layer-wise aggregation.
  arXiv  Detail & Related papers  (2025-05-29T19:59:18Z)
• KerZOO: Kernel Function Informed Zeroth-Order Optimization for Accurate and Accelerated LLM Fine-Tuning [15.81250204481401]
  We introduce a kernel-function-based ZO framework aimed at mitigating gradient estimation bias.<n>KerZOO achieves comparable or superior performance to existing ZO baselines.<n>We show that the kernel function is an effective avenue for reducing estimation bias in ZO methods.
  arXiv  Detail & Related papers  (2025-05-24T21:56:03Z)
• APCoTTA: Continual Test-Time Adaptation for Semantic Segmentation of Airborne LiDAR Point Clouds [14.348191795901101]
  Airborne laser scanning (ALS) point cloud segmentation is a fundamental task for large-scale 3D scene understanding.<n> Continuous Test-Time Adaptation (CTTA) offers a solution by adapting a source-pretrained model to evolving, unlabeled target domains.<n>We propose APCoTTA, the first CTTA method tailored for ALS point cloud semantic segmentation.
  arXiv  Detail & Related papers  (2025-05-15T05:21:16Z)
• Sparse Gradient Compression for Fine-Tuning Large Language Models [58.44973963468691]
  Fine-tuning large language models (LLMs) for downstream tasks has become increasingly crucial due to their widespread use and the growing availability of open-source models.<n>High memory costs associated with fine-tuning remain a significant challenge, especially as models increase in size.<n>We propose sparse compression gradient (SGC) to address these limitations.
  arXiv  Detail & Related papers  (2025-02-01T04:18:28Z)
• ALoRE: Efficient Visual Adaptation via Aggregating Low Rank Experts [71.91042186338163]
  ALoRE is a novel PETL method that reuses the hypercomplex parameterized space constructed by Kronecker product to Aggregate Low Rank Experts.<n>Thanks to the artful design, ALoRE maintains negligible extra parameters and can be effortlessly merged into the frozen backbone.
  arXiv  Detail & Related papers  (2024-12-11T12:31:30Z)
• Balancing Rates and Variance via Adaptive Batch-Size for Stochastic Optimization Problems [120.21685755278509]
  In this work, we seek to balance the fact that attenuating step-size is required for exact convergence with the fact that constant step-size learns faster in time up to an error. Rather than fixing the minibatch the step-size at the outset, we propose to allow parameters to evolve adaptively.
  arXiv  Detail & Related papers  (2020-07-02T16:02:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.