Prometheus Mind: Retrofitting Memory to Frozen Language Models

• URL: http://arxiv.org/abs/2601.15324v2
• Date: Fri, 23 Jan 2026 14:53:00 GMT
• Title: Prometheus Mind: Retrofitting Memory to Frozen Language Models
• Abstract summary: Prometheus Mind retrofits memory to a frozen Qwen3-4B using 11 modular adapters (530MB, 7% overhead)<n>We develop Contrastive Direction Discovery (CDD), which finds semantic directions via minimal pairs without labeled data.<n>On Prometheus-132 (132 cases), the system achieves 94.4% retrieval on clean inputs.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Adding memory to pretrained language models typically requires architectural changes or weight modification. We present Prometheus Mind, which retrofits memory to a frozen Qwen3-4B using 11 modular adapters (530MB, 7% overhead) -- fully reversible by removing the adapters. Building this system required solving four problems: (1) Extraction -- we develop Contrastive Direction Discovery (CDD), which finds semantic directions via minimal pairs without labeled data. (2) Training -- end-to-end optimization collapses; stage-wise training of each adapter on simple proxy tasks succeeds. (3) Injection -- learned encoders fail to generalize; we find that lm_head-weight rows already provide the mapping we need, requiring no training. (4) Hidden state collapse -- transformers make ``wife'' and ``brother'' 0.98+ similar; we train projections to recover distinction (0.98 $\rightarrow$ 0.09). On PrometheusExtract-132 (132 cases), the system achieves 94.4% retrieval on clean inputs (n=54, 95% CI: [84.9%, 98.1%]), degrading to 19.4% on informal inputs with ellipsis, filler words, or implicit subjects (n=36). The primary bottleneck is relation classification (47.3% accuracy), responsible for most extraction errors.

Related papers

• Persistent Recurrent Memory Between Transformer Layers - Improves Language Model Generalization [0.0]
  We introduce a persistent recurrent state that observes hidden representations via cross-attention, updates itself through a GRU, and modulates subsequent processing via gated addition.<n>This module adds only 3.7% additional parameters while reducing evaluation loss from $2.438 pm 0.004$ to $1.743 pm 0.018$, corresponding to a 28.5% reduction on held-out language modeling data.
  arXiv  Detail & Related papers  (2026-09-15T14:31:43Z)
• On the Design of Qwen3.8-Next Architecture: Evaluation, Efficiency, and Training Stability [57.03393882933515]
  We describe the architecture and ablations of Qwen3.8-Flash-Next, a sparse mixture-of-experts model with 125B parameters, 6B activated per token, and 51B parameters of n-gram embedding tables held off the accelerator.<n>On fourteen pre-training benchmarks the model leads the 397B-A17B predecessor on eight and trails it on the rest by at most 2.6 points, at 1/3 the activated parameters, 1/3 the training tokens, and roughly 1/9 the training FLOPs.
  arXiv  Detail & Related papers  (2026-08-31T06:35:07Z)
• When Does Trace-Driven Evaluation Mislead MoE Expert Caching? Replay Semantics, Workload Contamination, and Operating Regimes [3.471896896439183]
  Mixture-of-Experts (MoE) models have outgrown accelerator memory.<n>Policy that raised the hit rate would cut expert traffic per token.<n>We evaluate three evaluation axes that change conclusions, not just numbers.
  arXiv  Detail & Related papers  (2026-08-08T04:29:20Z)
• CARVE: Content-Aware Recurrent with Value Efficiency for Chunk-Parallel Linear Attention [0.0]
  Recurrent models must forget in order to remember, yet the state of the art decides what to erase without consulting what is stored.<n>This memory-blind gating is one of three coupled defects in the leading delta-rule architecture (GDN-2)<n>We introduce CARVE, which resolves all three problems through one principle: erase only on the key axis.
  arXiv  Detail & Related papers  (2026-06-25T16:16:51Z)
• Output Vector Editing for Memorization Mitigation in Large Language Models [68.30351930772788]
  Large language models memorize and reproduce sequences from their training data, creating privacy, copyright, and security risks.<n>Existing neuron-level mitigation methods equate editing with zeroing out neuron activations, but the activation only controls whether a neuron engages.<n>We propose output vector editing, a constrained-optimization edit that locates a small set of neurons responsible for a memorized continuation.
  arXiv  Detail & Related papers  (2026-06-17T07:29:18Z)
• Control-Plane Placement Shapes Forgetting: An Architectural Study of Agent Memory Across Thirteen System Configurations [1.2763567932588586]
  ForgetEval is a 1000-case templated suite plus a 385-case adversarial layer (132 hand-crafted + 253 LLM-drafted oracle-validated)<n>We observe three placement regimes with partly complementary coverage: deterministic primitives suffice for lexical/temporal categories but fail canonicalization.<n>We expose the trade-off via ForgetEval, a 1000-case templated suite plus a 385-case adversarial layer.
  arXiv  Detail & Related papers  (2026-06-14T16:32:15Z)
• Automated Proving of Shannon-Type Entropy Inequalities via Fine-Tuned Language Models and Guided Tree Search [50.16356451328644]
  Proving Shannon-type entropy inequalities is a fundamental task in information theory.<n>We investigate whether small-scale large language models, fine-tuned on atomic proof steps, can automate this process.<n>We show that GPT-5.5 solves 1.7% samples under zero-shot prompting while Psitip solves 33.3% samples.
  arXiv  Detail & Related papers  (2026-06-04T05:43:12Z)
• ImplicitMemBench: Measuring Unconscious Behavioral Adaptation in Large Language Models [60.14219417402433]
  Existing memory benchmarks for LLM agents evaluate explicit recall of facts, yet overlook implicit memory where experience becomes automated behavior without conscious retrieval.<n>We introduce ImplicitMemBench, the first systematic benchmark evaluating implicit memory through three constructs drawn from standard cognitive-science accounts of non-declarative memory.<n>Our 300-item suite employs a unified Learning/Priming-Interfere-Test protocol with first-attempt scoring.
  arXiv  Detail & Related papers  (2026-04-09T10:26:32Z)
• Ouroboros: Dynamic Weight Generation for Recursive Transformers via Input-Conditioned LoRA Modulation [0.0]
  Ouroboros is a system that attaches a compact Controller hypernetwork to a recursive transformer block.<n>The Controller observes the current hidden state, produces a per-step diagonal modulation vector, and applies it to frozen SVD-d LoRA bases.<n>On Qwen2.5-3B split into a Prelude/Recurrent/Coda architecture, Ouroboros reduces training loss by 43.4% over the unmodified 17-layer baseline.<n>The full system adds only 9.2M trainable parameters (Controller, gate, and per-step norms) yet outperforms equivalently-sized static per-step LoRA by 1.44 loss
  arXiv  Detail & Related papers  (2026-04-02T13:52:32Z)
• Transcoder Adapters for Reasoning-Model Diffing [28.090312772463253]
  We introduce transcoder computation, a technique for learning an interpretable approximation.<n>We apply transcoder adapters to characterize the differences between Qwen2.5-Math-7B and its reasoning-distilled variant, DeepSeek-R1-Di-Qwen-7B.
  arXiv  Detail & Related papers  (2026-02-24T13:40:28Z)
• Anatomy of Capability Emergence: Scale-Invariant Representation Collapse and Top-Down Reorganization in Neural Networks [1.5567685129899713]
  We track five geometric measures across five model scales.<n>We find that training begins with a universal representation collapse to task-specific floors.<n>On Pythia, global geometric patterns replicate but per-task precursor signals do not.
  arXiv  Detail & Related papers  (2026-02-17T20:39:02Z)
• SMART SLM: Structured Memory and Reasoning Transformer, A Small Language Model for Accurate Document Assistance [0.34376560669160394]
  SMART (Structured Memory and Reasoning Transformer) structures its processing by using a hierarchical approach.<n>The entire SMART model utilizes 45.51M parameters, which is 64% less than GPT-2 (124M) and 69% less than BERT (133M)
  arXiv  Detail & Related papers  (2025-12-24T16:59:04Z)
• SER-Diff: Synthetic Error Replay Diffusion for Incremental Brain Tumor Segmentation [0.0]
  We propose Synthetic Error Replay Diffusion (SER-Diff), a framework that unifies diffusion-based refinement with incremental learning.<n>SER-Diff mitigates catastrophic forgetting and delivers more accurate and anatomically coherent segmentations across evolving datasets.
  arXiv  Detail & Related papers  (2025-10-06T21:53:45Z)
• Benchmarking Reasoning Robustness in Large Language Models [76.79744000300363]
  We find significant performance degradation on novel or incomplete data.<n>These findings highlight the reliance on recall over rigorous logical inference.<n>This paper introduces a novel benchmark, termed as Math-RoB, that exploits hallucinations triggered by missing information to expose reasoning gaps.
  arXiv  Detail & Related papers  (2025-03-06T15:36:06Z)
• Erasing Without Remembering: Implicit Knowledge Forgetting in Large Language Models [81.62767292169225]
  We investigate knowledge forgetting in large language models with a focus on its generalisation.<n>We propose PerMU, a novel probability perturbation-based unlearning paradigm.<n>Experiments are conducted on a diverse range of datasets, including TOFU, Harry Potter, ZsRE, WMDP, and MUSE.
  arXiv  Detail & Related papers  (2025-02-27T11:03:33Z)
• Memory-Efficient Training for Deep Speaker Embedding Learning in Speaker Verification [50.596077598766975]
  We explore a memory-efficient training strategy for deep speaker embedding learning in resource-constrained scenarios.<n>For activations, we design two types of reversible neural networks which eliminate the need to store intermediate activations.<n>For states, we introduce a dynamic quantization approach that replaces the original 32-bit floating-point values with a dynamic tree-based 8-bit data type.
  arXiv  Detail & Related papers  (2024-12-02T06:57:46Z)
• CLOVER: Cross-Layer Orthogonal Vectors Pruning and Fine-Tuning [24.697865155104644]
  CLOVER (Cross-Layer Orthogonal Vectors) is a novel approach that treats pairs of attention layers as a set of low-rank decompositions.<n>We apply CLOVER to various models, including GPT-2 XL, DeepSeek-V2-Lite, Whisper-Large-v3, Stable Diffusion XL, and LLaMA-3.2-11B-Vision.
  arXiv  Detail & Related papers  (2024-11-26T13:34:02Z)
• GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projection [133.45193150403537]
  Training Large Language Models (LLMs) presents significant memory challenges due to the growing size of weights and GPU states. In this work, we propose Gradient Low-Rank Projection (GaLore) as a memory-efficient training strategy. Our 8-bit GaLore further reduces memory by up to 82.5% and total training memory by 63.3%, compared to a BF16 baseline.
  arXiv  Detail & Related papers  (2024-03-06T07:29:57Z)
• Integral Continual Learning Along the Tangent Vector Field of Tasks [112.02761912526734]
  We propose a lightweight continual learning method which incorporates information from specialized datasets incrementally. It maintains a small fixed-size memory buffer, as low as 0.4% of the source datasets, which is updated by simple resampling. Our method achieves strong performance across various buffer sizes for different datasets.
  arXiv  Detail & Related papers  (2022-11-23T16:49:26Z)
• Compressing 1D Time-Channel Separable Convolutions using Sparse Random Ternary Matrices [65.4388266814055]
  We replace 1x1-convolutions in 1D time-channel separable convolutions with constant, sparse random ternary matrices with weights in $-1,0,+1$. For command recognition on Google Speech Commands v1, we improve the state-of-the-art accuracy from $97.21%$ to $97.41%$ at the same network size. For speech recognition on Librispeech, we half the number of weights to be trained while only sacrificing about $1%$ of the floating-point baseline's word error rate.
  arXiv  Detail & Related papers  (2021-03-31T15:09:20Z)
• Harnessing Geometric Constraints from Auxiliary Labels to Improve Embedding Functions for One-Shot Learning [21.445455835823626]
  We introduce novel geometric constraints on the embedding space learned by a deep model using either manually annotated or automatically detected auxiliary labels. Our methods provide a higher verification accuracy (99.7, 86.2, 99.4, and 79.3% with our proposed TL+PDP+FBV loss, versus 97.5, 72.6, 93.1, and 70.5% using a standard Triplet Loss on the four datasets, respectively)
  arXiv  Detail & Related papers  (2021-03-05T18:27:38Z)
• TinyTL: Reduce Activations, Not Trainable Parameters for Efficient On-Device Learning [78.80707950262214]
  On-device learning enables edge devices to continually adapt the AI models to new data. Existing work solves this problem by reducing the number of trainable parameters. We present Tiny-Transfer-Learning (TinyTL) for memory-efficient on-device learning.
  arXiv  Detail & Related papers  (2020-07-22T18:39:53Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.