Universally Converging Representations of Matter Across Scientific Foundation Models

• URL: http://arxiv.org/abs/2512.03750v1
• Date: Wed, 03 Dec 2025 12:47:06 GMT
• Title: Universally Converging Representations of Matter Across Scientific Foundation Models
• Authors: Sathya Edamadaka, Soojung Yang, Ju Li, Rafael Gómez-Bombarelli,
• Abstract summary: We show that representations learned by nearly sixty scientific models are highly aligned across a wide range of chemical systems.<n>On inputs similar to those seen during training, high-performing models align closely and weak models diverge into local sub-optima in representation space.<n>Our findings establish representational alignment as a quantitative benchmark for foundation-level generality in scientific models.
• Score: 5.309886698585678
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Machine learning models of vastly different modalities and architectures are being trained to predict the behavior of molecules, materials, and proteins. However, it remains unclear whether they learn similar internal representations of matter. Understanding their latent structure is essential for building scientific foundation models that generalize reliably beyond their training domains. Although representational convergence has been observed in language and vision, its counterpart in the sciences has not been systematically explored. Here, we show that representations learned by nearly sixty scientific models, spanning string-, graph-, 3D atomistic, and protein-based modalities, are highly aligned across a wide range of chemical systems. Models trained on different datasets have highly similar representations of small molecules, and machine learning interatomic potentials converge in representation space as they improve in performance, suggesting that foundation models learn a common underlying representation of physical reality. We then show two distinct regimes of scientific models: on inputs similar to those seen during training, high-performing models align closely and weak models diverge into local sub-optima in representation space; on vastly different structures from those seen during training, nearly all models collapse onto a low-information representation, indicating that today's models remain limited by training data and inductive bias and do not yet encode truly universal structure. Our findings establish representational alignment as a quantitative benchmark for foundation-level generality in scientific models. More broadly, our work can track the emergence of universal representations of matter as models scale, and for selecting and distilling models whose learned representations transfer best across modalities, domains of matter, and scientific tasks.

Related papers

• UniG2U-Bench: Do Unified Models Advance Multimodal Understanding? [50.92401586025528]
  Unified multimodal models have recently demonstrated strong generative capabilities, yet whether and when generation improves understanding remains unclear.<n>We introduce UniG2U-Bench, a comprehensive benchmark categorizing generation-to-understanding (G2U) evaluation into 7 regimes and 30 subtasks.
  arXiv  Detail & Related papers  (2026-03-03T18:36:16Z)
• Visual Generation Unlocks Human-Like Reasoning through Multimodal World Models [60.543714835980325]
  Humans construct internal world models and reason by manipulating the concepts within these models.<n>Recent advances in AI approximate such human cognitive abilities, where world models are believed to be embedded within large language models.<n>This paper presents the first principled study of when and how visual generation benefits reasoning.
  arXiv  Detail & Related papers  (2026-01-27T17:40:07Z)
• Foundation Models for Discovery and Exploration in Chemical Space [57.97784111110166]
  MIST is a family of molecular foundation models trained on large unlabeled datasets.<n>We demonstrate the ability of these models to solve real-world problems across chemical space.
  arXiv  Detail & Related papers  (2025-10-20T17:56:01Z)
• Comparing Computational Pathology Foundation Models using Representational Similarity Analysis [0.6660458629649825]
  We systematically analyze the representational spaces of six CPath foundation models using techniques in computational neuroscience.<n>We find that UNI2 and Virchow2 have the most distinct representational structures, whereas Prov-Gigapath has the highest average similarity across models.
  arXiv  Detail & Related papers  (2025-09-18T23:01:13Z)
• A Survey of Scientific Large Language Models: From Data Foundations to Agent Frontiers [251.23085679210206]
  Scientific Large Language Models (Sci-LLMs) are transforming how knowledge is represented, integrated, and applied in scientific research.<n>This survey reframes the development of Sci-LLMs as a co-evolution between models and their underlying data substrate.<n>We formulate a unified taxonomy of scientific data and a hierarchical model of scientific knowledge.
  arXiv  Detail & Related papers  (2025-08-28T18:30:52Z)
• UNIFORM: Unifying Knowledge from Large-scale and Diverse Pre-trained Models [62.76435672183968]
  We introduce a novel framework, namely UNIFORM, for knowledge transfer from a diverse set of off-the-shelf models into one student model.<n>We propose a dedicated voting mechanism to capture the consensus of knowledge both at the logit level and at the feature level.<n>Experiments demonstrate that UNIFORM effectively enhances unsupervised object recognition performance compared to strong knowledge transfer baselines.
  arXiv  Detail & Related papers  (2025-08-27T00:56:11Z)
• A Markov Categorical Framework for Language Modeling [9.910562011343009]
  Autoregressive language models achieve remarkable performance, yet a unified theory explaining their internal mechanisms, how training shapes their representations, and enables complex behaviors, remains elusive.<n>We introduce a new analytical framework that models the single-step generation process as a composition of information-processing stages using the language of Markov categories.<n>This work presents a powerful new lens for understanding how information flows through a model and how the training objective shapes its internal geometry.
  arXiv  Detail & Related papers  (2025-07-25T13:14:03Z)
• Learning Local Causal World Models with State Space Models and Attention [1.5498250598583487]
  We show that a SSM can model the dynamics of a simple environment and learn a causal model at the same time.<n>We pave the way for further experiments that lean into the strength of SSMs and further enhance them with causal awareness.
  arXiv  Detail & Related papers  (2025-05-04T11:57:02Z)
• Towards Physically Interpretable World Models: Meaningful Weakly Supervised Representations for Visual Trajectory Prediction [0.1534667887016089]
  We propose a novel architecture that aligns learned latent representations with real-world physical quantities.<n>Three case studies demonstrate that our approach achieves physical interpretability and accurate state predictions.
  arXiv  Detail & Related papers  (2024-12-17T12:51:24Z)
• Intriguing Equivalence Structures of the Embedding Space of Vision Transformers [1.7418480517632609]
  Pre-trained large foundation models play a central role in the recent surge of artificial intelligence. Due to their inherent complexity, these models are not well understood. We show via analyses and systematic experiments that the representation space consists of large piecewise linear subspaces.
  arXiv  Detail & Related papers  (2024-01-28T04:59:51Z)
• Dynamic Latent Separation for Deep Learning [67.62190501599176]
  A core problem in machine learning is to learn expressive latent variables for model prediction on complex data. Here, we develop an approach that improves expressiveness, provides partial interpretation, and is not restricted to specific applications.
  arXiv  Detail & Related papers  (2022-10-07T17:56:53Z)
• Geometric and Topological Inference for Deep Representations of Complex Networks [13.173307471333619]
  We present a class of statistics that emphasize the topology as well as the geometry of representations. We evaluate these statistics in terms of the sensitivity and specificity that they afford when used for model selection. These new methods enable brain and computer scientists to visualize the dynamic representational transformations learned by brains and models.
  arXiv  Detail & Related papers  (2022-03-10T17:14:14Z)
• S2RMs: Spatially Structured Recurrent Modules [105.0377129434636]
  We take a step towards exploiting dynamic structure that are capable of simultaneously exploiting both modular andtemporal structures. We find our models to be robust to the number of available views and better capable of generalization to novel tasks without additional training.
  arXiv  Detail & Related papers  (2020-07-13T17:44:30Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.