ComMem: Complementary Memory Systems for Test-Time Adaptation of Vision-Language Models
Abstract Overview
This paper presents ComMem, a test-time adaptation method for vision-language models that is inspired by complementary memory systems in the brain. The method combines a fast-adapting detailed memory that stores high-confidence visual features with a slow-integrating abstract memory built from textual prototypes. For each test sample, the two memories are jointly optimized to enforce cross-modal consistency while adapting normalization and residual parameters online. The paper evaluates the approach on 15 benchmark datasets under both natural distribution shifts and cross-dataset generalization settings. Across these settings, ComMem is reported to improve robustness and maintain practical efficiency relative to prior adaptation methods.
Novelty
The distinctive aspect of the work is its explicit dual-memory design for VLM test-time adaptation: an HPC-like visual cache for rapid instance-level updates and an NC-like textual memory for gradual semantic integration. The method also introduces a reconsolidation-style joint optimization step that couples the two memories through cross-modal alignment rather than adapting only one modality.
Results
On ImageNet distribution-shift benchmarks, ComMem achieves the best reported average OOD accuracy among compared methods, reaching 48.84% with CLIP-ResNet-50 and 65.36% with CLIP-ViT-B/16. On cross-dataset generalization over 10 datasets, it also attains the top average accuracy, with 64.60% for ResNet-50 and 70.74% for ViT-B/16. The efficiency analysis reports 64.05% accuracy on ImageNet in about 1 hour 32 minutes with 4.08M learnable parameters, making it much faster than prompt-tuning approaches such as TPT and DiffTPT while delivering larger gains over zero-shot CLIP.
Key Points
- ComMem adapts both visual and textual representations through complementary fast and slow memory systems instead of restricting adaptation to a single modality.
- The method consistently ranks first on the reported natural distribution shift and cross-dataset benchmarks across both ResNet-50 and ViT-B/16 CLIP backbones.
- Ablations support the dual-memory design: fast updates work best for the HPC-like cache, slow updates work best for the NC-like textual memory, and moderate cache sizes yield the strongest performance.