What Makes a Redundant Representation Remember? Lineage Isolation, Not Masking

• URL: http://arxiv.org/abs/2609.05304v1
• Date: Fri, 04 Sep 2026 15:57:10 GMT
• Title: What Makes a Redundant Representation Remember? Lineage Isolation, Not Masking
• Abstract summary: Memory-based evolutionary algorithms for dynamic optimization often carry a redundant second copy of the genotype and expose only one copy to the objective.<n>We show this assumption is false as usually implemented, and identify the structural property that actually determines whether the shielded copy retains information.<n>We formalize such methods as a gated dual-copy representation with two independent design axes.
• Score: 5.414649342700987
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Memory-based evolutionary algorithms for dynamic optimization often carry a redundant second copy of the genotype and expose only one copy to the objective, on the assumption that the shielded copy accumulates information about past optima. We show this assumption is false as usually implemented, and identify the structural property that actually determines whether the shielded copy retains information. We formalize such methods as a gated dual-copy representation with two independent design axes: a gating rule deciding which copy is evaluated, and an inheritance rule deciding whether the two copies mix across generations. A ablation shows retained information is governed almost entirely by the inheritance rule (21.4 vs. 1.3 bits) and is nearly invariant to the gating rule. Per-locus independent inheritance reshuffles cross-locus structure every generation, so shielding preserves the variance of the hidden copy while destroying the pattern that constitutes a memory. Under isolated inheritance the memory effect is real: against a single-copy baseline matched for representation budget, the method gains +0.010 AUC when optima recur periodically and loses 0.078 when they drift unidirectionally---a 0.089 separation under otherwise identical settings, which excludes explanations based on added capacity. We show the readout rate is also the corruption rate, predicting and confirming an interior optimum replicated across two implementations. We report one negative result with a mechanism: dual-copy representations lower the mutational error threshold, because gated expression is a selector rather than a joint decoder and therefore provides no coding gain. Finally, we document a benchmarking hazard: on dynamic benchmarks the choice of recombination operator alone shifted our baseline by 0.062 AUC, six times the effect size under study.

Related papers

• Scalable Frequency- and Length-Aware Subdocument Deduplication for Large Language Model Pretraining [17.78093240780209]
  We propose a scalable subdocument deduplication framework that decouples duplicate detection from copy retention.<n>Experiments on FineWeb-Edu and a code-containing web corpus show that models trained on data processed by our method achieve the best overall performance.
  arXiv  Detail & Related papers  (2026-08-04T04:02:28Z)
• Where Does Generative Difficulty Reside? An Empirical Study of Target Representations [62.54876287800644]
  The target representation defines the distribution an image generator must learn, yet it is often treated as an interchangeable interface.<n>We study raw pixels, SD-VAE latents and DINOv2 as well as MAE representation-autoencoder features within a unified masked autoregressive rectified-flow model.<n>Our results show that compression, reconstruction fidelity, token dimensionality, and visible semantic clustering do not individually predict generative behavior.
  arXiv  Detail & Related papers  (2026-08-01T12:32:31Z)
• ScDiVa: Masked Discrete Diffusion for Joint Modeling of Single-Cell Identity and Expression [24.508523704467695]
  Single-cell RNA-seq profiles are high-dimensional, sparse, and unordered, causing autoregressive generation to impose an artificial ordering bias.<n>We propose scDiVa, a masked discrete diffusion foundation model that aligns generation with the dropout-like corruption process.
  arXiv  Detail & Related papers  (2026-02-03T12:50:29Z)
• To Copy or Not to Copy: Copying Is Easier to Induce Than Recall [5.057026826740146]
  Language models must arbitrate between parametric knowledge stored in their weights and contextual information in the prompt.<n>This work presents a mechanistic study of that choice by extracting an empharbitration vector from model activations on a curated dataset.
  arXiv  Detail & Related papers  (2026-01-17T14:46:29Z)
• Seed-Induced Uniqueness in Transformer Models: Subspace Alignment Governs Subliminal Transfer [0.3805935148497361]
  We analyze subliminal transfer in Transformer models, where a teacher embeds hidden traits that can be linearly decoded by a student.<n>We find that transfer strength hinges on alignment within a trait-discriminative subspace.
  arXiv  Detail & Related papers  (2025-11-02T17:34:43Z)
• CopyBench: Measuring Literal and Non-Literal Reproduction of Copyright-Protected Text in Language Model Generation [132.00910067533982]
  We introduce CopyBench, a benchmark designed to measure both literal and non-literal copying in LM generations. We find that, although literal copying is relatively rare, two types of non-literal copying -- event copying and character copying -- occur even in models as small as 7B parameters.
  arXiv  Detail & Related papers  (2024-07-09T17:58:18Z)
• UGMAE: A Unified Framework for Graph Masked Autoencoders [67.75493040186859]
  We propose UGMAE, a unified framework for graph masked autoencoders. We first develop an adaptive feature mask generator to account for the unique significance of nodes. We then design a ranking-based structure reconstruction objective joint with feature reconstruction to capture holistic graph information.
  arXiv  Detail & Related papers  (2024-02-12T19:39:26Z)
• REBAR: Retrieval-Based Reconstruction for Time-series Contrastive Learning [64.08293076551601]
  We propose a novel method of using a learned measure for identifying positive pairs. Our Retrieval-Based Reconstruction measure measures the similarity between two sequences. We show that the REBAR error is a predictor of mutual class membership.
  arXiv  Detail & Related papers  (2023-11-01T13:44:45Z)
• Rethinking Reconstruction Autoencoder-Based Out-of-Distribution Detection [0.0]
  Reconstruction autoencoder-based methods deal with the problem by using input reconstruction error as a metric of novelty vs. normality. We introduce semantic reconstruction, data certainty decomposition and normalized L2 distance to substantially improve original methods. Our method works without any additional data, hard-to-implement structure, time-consuming pipeline, and even harming the classification accuracy of known classes.
  arXiv  Detail & Related papers  (2022-03-04T09:04:55Z)
• Dual Cluster Contrastive learning for Person Re-Identification [78.42770787790532]
  We formulate a unified cluster contrastive framework, named Dual Cluster Contrastive learning (DCC) DCC maintains two types of memory banks: individual and centroid cluster memory banks. It can be easily applied for unsupervised or supervised person ReID.
  arXiv  Detail & Related papers  (2021-12-09T02:43:25Z)
• Self-Supervised Learning Disentangled Group Representation as Feature [82.07737719232972]
  We show that existing Self-Supervised Learning (SSL) only disentangles simple augmentation features such as rotation and colorization. We propose an iterative SSL algorithm: Iterative Partition-based Invariant Risk Minimization (IP-IRM) We prove that IP-IRM converges to a fully disentangled representation and show its effectiveness on various benchmarks.
  arXiv  Detail & Related papers  (2021-10-28T16:12:33Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.