Hyperbolic Multimodal Continual Learning: A Closest-Admissible Solution
Abstract Overview
This paper studies continual learning for hyperbolic multimodal models, where sequential updates can preserve task scores yet still distort the Lorentz geometry that encodes within-modality similarity, cross-modal alignment, and semantic hierarchy. The authors show that preserving previously learned multimodal geometry requires all modalities to undergo a shared hyperbolic isometry, which yields a family of admissible first-order parameter updates. Based on this characterization, they propose Hyperbolic Multimodal Continual Learning (HMCL), including a closest-admissible variant (HMCL-CA) and a minimal-rotation variant (HMCL-MR), with post-AdamW correction and task anchoring to control accumulated drift. Experiments on a unified 16-task classification and retrieval stream across three hyperbolic backbones evaluate performance, backward transfer, representation drift, and hierarchy preservation.
Novelty
The distinctive contribution is a geometric continual-learning formulation for hyperbolic multimodal representations that ties old-knowledge preservation to one shared hyperbolic isometry across modalities. The paper further introduces a joint closest-admissible correction, solved via a Lyapunov equation, and analyzes optimizer-consistent post-AdamW correction with task anchoring within the same admissible family.
Results
Across three hyperbolic backbones, both HMCL variants outperform sequential fine-tuning and four continual-learning baselines in final classification, retrieval, and backward transfer, while HMCL-CA attains the best Overall score on every backbone in the main 16-task setting. Representation analyses report 81.2% to 95.5% reductions in radial, angular, cross-modal, and paired-distance drift, and ImageNet-WordNet evaluations show improved semantic ancestry and radial hierarchy preservation. In a separate modality-extended stream with audio and thermal retrieval tasks, HMCL also improves overall performance and reduces forgetting relative to matched Vanilla training.
Key Points
- The method derives admissible continual-learning updates by requiring all modalities to preserve prior hyperbolic geometry through a shared isometry, rather than protecting only parameters or Euclidean subspaces.
- HMCL-CA and HMCL-MR operationalize this theory as corrections to the realized AdamW step, and task anchoring is used to bound within-task accumulation while retaining learning flexibility.
- Empirically, the gains are supported not only by better final task metrics and backward transfer, but also by reduced representation drift and stronger preservation of hierarchical structure on ImageNet-WordNet analyses.