Eigen Memory Tree

• URL: http://arxiv.org/abs/2210.14077v2
• Date: Wed, 26 Oct 2022 20:11:23 GMT
• Title: Eigen Memory Tree
• Authors: Mark Rucker, Jordan T. Ash, John Langford, Paul Mineiro, Ida Momennejad
• Abstract summary: This work introduces the Eigen Memory Tree (EMT), a novel online memory model for sequential learning scenarios. We demonstrate that EMT outperforms existing online memory approaches, and provide a hybridized EMT-parametric algorithm that enjoys drastically improved performance. Our findings are validated using 206 datasets from the OpenML repository in both bounded and infinite memory budget situations.
• Score: 27.33148786536804
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: This work introduces the Eigen Memory Tree (EMT), a novel online memory model for sequential learning scenarios. EMTs store data at the leaves of a binary tree and route new samples through the structure using the principal components of previous experiences, facilitating efficient (logarithmic) access to relevant memories. We demonstrate that EMT outperforms existing online memory approaches, and provide a hybridized EMT-parametric algorithm that enjoys drastically improved performance over purely parametric methods with nearly no downsides. Our findings are validated using 206 datasets from the OpenML repository in both bounded and infinite memory budget situations.

Related papers

• B'MOJO: Hybrid State Space Realizations of Foundation Models with Eidetic and Fading Memory [91.81390121042192]
  We develop a class of models called B'MOJO to seamlessly combine eidetic and fading memory within an composable module. B'MOJO's ability to modulate eidetic and fading memory results in better inference on longer sequences tested up to 32K tokens.
  arXiv  Detail & Related papers  (2024-07-08T18:41:01Z)
• Heterogenous Memory Augmented Neural Networks [84.29338268789684]
  We introduce a novel heterogeneous memory augmentation approach for neural networks. By introducing learnable memory tokens with attention mechanism, we can effectively boost performance without huge computational overhead. We show our approach on various image and graph-based tasks under both in-distribution (ID) and out-of-distribution (OOD) conditions.
  arXiv  Detail & Related papers  (2023-10-17T01:05:28Z)
• Frustratingly Simple Memory Efficiency for Pre-trained Language Models via Dynamic Embedding Pruning [42.652021176354644]
  The memory footprint of pre-trained language models (PLMs) can hinder deployment in memory-constrained settings. We propose a simple yet effective approach that leverages this finding to minimize the memory footprint of the embedding matrix. We show that this approach provides substantial reductions in memory usage across a wide range of models and tasks.
  arXiv  Detail & Related papers  (2023-09-15T19:00:00Z)
• EMO: Episodic Memory Optimization for Few-Shot Meta-Learning [69.50380510879697]
  episodic memory optimization for meta-learning, we call EMO, is inspired by the human ability to recall past learning experiences from the brain's memory. EMO nudges parameter updates in the right direction, even when the gradients provided by a limited number of examples are uninformative. EMO scales well with most few-shot classification benchmarks and improves the performance of optimization-based meta-learning methods.
  arXiv  Detail & Related papers  (2023-06-08T13:39:08Z)
• Dynamic Ensemble Size Adjustment for Memory Constrained Mondrian Forest [0.0]
  In this paper, we show that under memory constraints, increasing the size of a tree-based ensemble classifier can worsen its performance. We experimentally show the existence of an optimal ensemble size for a memory-bounded Mondrian forest on data streams. We conclude that our method can achieve up to 95% of the performance of an optimally-sized Mondrian forest for stable datasets.
  arXiv  Detail & Related papers  (2022-10-11T18:05:58Z)
• A Memory Transformer Network for Incremental Learning [64.0410375349852]
  We study class-incremental learning, a training setup in which new classes of data are observed over time for the model to learn from. Despite the straightforward problem formulation, the naive application of classification models to class-incremental learning results in the "catastrophic forgetting" of previously seen classes. One of the most successful existing methods has been the use of a memory of exemplars, which overcomes the issue of catastrophic forgetting by saving a subset of past data into a memory bank and utilizing it to prevent forgetting when training future tasks.
  arXiv  Detail & Related papers  (2022-10-10T08:27:28Z)
• A Model or 603 Exemplars: Towards Memory-Efficient Class-Incremental Learning [56.450090618578]
  Class-Incremental Learning (CIL) aims to train a model with limited memory size to meet this requirement. We show that when counting the model size into the total budget and comparing methods with aligned memory size, saving models do not consistently work. We propose a simple yet effective baseline, denoted as MEMO for Memory-efficient Expandable MOdel.
  arXiv  Detail & Related papers  (2022-05-26T08:24:01Z)
• Mondrian Forest for Data Stream Classification Under Memory Constraints [0.0]
  We adapt the online Mondrian forest classification algorithm to work with memory constraints on data streams. We design five out-of-memory strategies to update Mondrian trees with new data points when the memory limit is reached. We also design trimming mechanisms to make Mondrian trees more robust to concept drifts under memory constraints.
  arXiv  Detail & Related papers  (2022-05-12T15:35:03Z)
• Semantically Constrained Memory Allocation (SCMA) for Embedding in Efficient Recommendation Systems [27.419109620575313]
  A key challenge for deep learning models is to work with millions of categorical classes or tokens. We propose a novel formulation of memory shared embedding, where memory is shared in proportion to the overlap in semantic information. We demonstrate a significant reduction in the memory footprint while maintaining performance.
  arXiv  Detail & Related papers  (2021-02-24T19:55:49Z)
• PeTra: A Sparsely Supervised Memory Model for People Tracking [50.98911178059019]
  We propose PeTra, a memory-augmented neural network designed to track entities in its memory slots. We empirically compare key modeling choices, finding that we can simplify several aspects of the design of the memory module while retaining strong performance. PeTra is highly effective in both evaluations, demonstrating its ability to track people in its memory despite being trained with limited annotation.
  arXiv  Detail & Related papers  (2020-05-06T17:45:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.