ZeroSiam: An Efficient Siamese for Test-Time Entropy Optimization without Collapse

• URL: http://arxiv.org/abs/2509.23183v1
• Date: Sat, 27 Sep 2025 08:37:47 GMT
• Title: ZeroSiam: An Efficient Siamese for Test-Time Entropy Optimization without Collapse
• Abstract summary: We introduce ZeroSiam, an efficient asymmetric Siamese architecture tailored for test-time entropy minimization.<n>ZeroSiam prevents collapse through asymmetric divergence alignment, which is efficiently achieved by a learnable predictor and a stop-gradient operator.<n>We provide empirical and theoretical evidence that ZeroSiam not only prevents collapse solutions, but also absorbs and regularizes biased learning signals, enhancing performance even when no collapse occurs.
• Score: 45.0314640277594
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Test-time entropy minimization helps adapt a model to novel environments and incentivize its reasoning capability, unleashing the model's potential during inference by allowing it to evolve and improve in real-time using its own predictions, achieving promising performance. However, pure entropy minimization can favor non-generalizable shortcuts, such as inflating the logit norm and driving all predictions to a dominant class to reduce entropy, risking collapsed solutions (e.g., constant one-hot outputs) that trivially minimize the objective without meaningful learning. In this paper, we introduce ZeroSiam, an efficient asymmetric Siamese architecture tailored for test-time entropy minimization. ZeroSiam prevents collapse through asymmetric divergence alignment, which is efficiently achieved by a learnable predictor and a stop-gradient operator before the classifier. We provide empirical and theoretical evidence that ZeroSiam not only prevents collapse solutions, but also absorbs and regularizes biased learning signals, enhancing performance even when no collapse occurs. Despite its simplicity, extensive results show that ZeroSiam performs more stably over prior methods using negligible overhead, demonstrating efficacy on both vision adaptation and large language model reasoning tasks across challenging test scenarios and diverse models, including tiny models that are particularly collapse-prone.

Related papers

• The Coverage Principle: How Pre-Training Enables Post-Training [70.25788947586297]
  We study how pre-training shapes the success of the final model.<n>We uncover a mechanism that explains the power of coverage in predicting downstream performance.
  arXiv  Detail & Related papers  (2025-10-16T17:53:50Z)
• Think Just Enough: Sequence-Level Entropy as a Confidence Signal for LLM Reasoning [5.37133760455631]
  We introduce a novel entropy-based framework to drive token efficiency in large language models during reasoning tasks.<n>Our approach uses Shannon entropy from token-level logprobs as a confidence signal to enable early stopping.<n>We show that entropy-based confidence calibration represents an emergent property of advanced post-training optimization.
  arXiv  Detail & Related papers  (2025-10-09T12:33:16Z)
• Ranked Entropy Minimization for Continual Test-Time Adaptation [7.5140668729696145]
  Test-time adaptation aims to adapt to realistic environments in an online manner by learning during test time.<n>Entropy minimization has emerged as a principal strategy for test-time adaptation due to its efficiency and adaptability.<n>We propose ranked entropy minimization to mitigate the stability problem of the entropy minimization method.
  arXiv  Detail & Related papers  (2025-05-22T09:29:38Z)
• From Data to Uncertainty Sets: a Machine Learning Approach [5.877778007271621]
  We leverage robust optimization to protect a constraint against the uncertainty of a machine learning model's output.<n>We derive strong guarantees on the probability of violation.<n>On synthetic computational experiments, our method requires uncertainty sets with radii up to one order of magnitude smaller than those of other approaches.
  arXiv  Detail & Related papers  (2025-03-04T01:30:28Z)
• Improving Pre-trained Self-Supervised Embeddings Through Effective Entropy Maximization [25.854403747410146]
  We motivate an effective entropy criterion (E2MC) for self-supervised learning.<n>We demonstrate that using it to continue training an already-trained SSL model for only a handful of epochs leads to a consistent and, in some cases, significant improvement in downstream performance.<n>We also show that continued pre-training with alternative criteria does not lead to notable improvements, and in some cases, even degrades performance.
  arXiv  Detail & Related papers  (2024-11-24T17:38:23Z)
• COME: Test-time adaption by Conservatively Minimizing Entropy [45.689829178140634]
  Conservatively Minimize the Entropy (COME) is a drop-in replacement of traditional entropy (EM) COME explicitly models the uncertainty by characterizing a Dirichlet prior distribution over model predictions. We show that COME achieves state-of-the-art performance on commonly used benchmarks.
  arXiv  Detail & Related papers  (2024-10-12T09:20:06Z)
• A Universal Class of Sharpness-Aware Minimization Algorithms [57.29207151446387]
  We introduce a new class of sharpness measures, leading to new sharpness-aware objective functions. We prove that these measures are textitly expressive, allowing any function of the training loss Hessian matrix to be represented by appropriate hyper and determinants.
  arXiv  Detail & Related papers  (2024-06-06T01:52:09Z)
• Self-Supervised Dataset Distillation for Transfer Learning [77.4714995131992]
  We propose a novel problem of distilling an unlabeled dataset into a set of small synthetic samples for efficient self-supervised learning (SSL) We first prove that a gradient of synthetic samples with respect to a SSL objective in naive bilevel optimization is textitbiased due to randomness originating from data augmentations or masking. We empirically validate the effectiveness of our method on various applications involving transfer learning.
  arXiv  Detail & Related papers  (2023-10-10T10:48:52Z)
• Predicting Emergent Abilities with Infinite Resolution Evaluation [85.89911520190711]
  We introduce PassUntil, an evaluation strategy with theoretically infinite resolution, through massive sampling in the decoding phase. We predict the performance of the 2.4B model on code generation with merely 0.05% deviation before training starts. We identify a kind of accelerated emergence whose scaling curve cannot be fitted by standard scaling law function.
  arXiv  Detail & Related papers  (2023-10-05T02:35:00Z)
• REALM: Robust Entropy Adaptive Loss Minimization for Improved Single-Sample Test-Time Adaptation [5.749155230209001]
  Fully-test-time adaptation (F-TTA) can mitigate performance loss due to distribution shifts between train and test data. We present a general framework for improving robustness of F-TTA to noisy samples, inspired by self-paced learning and robust loss functions.
  arXiv  Detail & Related papers  (2023-09-07T18:44:58Z)
• Understanding Collapse in Non-Contrastive Learning [122.2499276246997]
  We show that SimSiam representations undergo partial dimensional collapse if the model is too small relative to the dataset size. We propose a metric to measure the degree of this collapse and show that it can be used to forecast the downstream task performance without any fine-tuning or labels.
  arXiv  Detail & Related papers  (2022-09-29T17:59:55Z)
• Reparameterized Variational Divergence Minimization for Stable Imitation [57.06909373038396]
  We study the extent to which variations in the choice of probabilistic divergence may yield more performant ILO algorithms. We contribute a re parameterization trick for adversarial imitation learning to alleviate the challenges of the promising $f$-divergence minimization framework. Empirically, we demonstrate that our design choices allow for ILO algorithms that outperform baseline approaches and more closely match expert performance in low-dimensional continuous-control tasks.
  arXiv  Detail & Related papers  (2020-06-18T19:04:09Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.