Grounding Occam's Razor in a Formal Theory of Simplicity

• URL: http://arxiv.org/abs/2004.05269v2
• Date: Thu, 3 Sep 2020 01:34:38 GMT
• Title: Grounding Occam's Razor in a Formal Theory of Simplicity
• Abstract summary: A formal theory of simplicity is introduced, in the context of a "combinational" computation model. A formalization of the cognitive-systems notion of a "coherent dual network" hierarchy and heterarchy is presented.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: A formal theory of simplicity is introduced, in the context of a "combinational" computation model that views computation as comprising the iterated transformational and compositional activity of a population of agents upon each other. Conventional measures of simplicity in terms of algorithmic information etc. are shown to be special cases of a broader understanding of the core "symmetry" properties constituting what is defined here as a Compositional Simplicity Measure (CoSM). This theory of CoSMs is extended to a theory of CoSMOS (Combinational Simplicity Measure Operating Sets) which involve multiple simplicity measures utilized together. Given a vector of simplicity measures, an entity is associated not with an individual simplicity value but with a "simplicity bundles" of Pareto-optimal simplicity-value vectors. CoSMs and CoSMOS are then used as a foundation for a theory of pattern and multipattern, and a theory of hierarchy and heterarchy in systems of patterns. A formalization of the cognitive-systems notion of a "coherent dual network" interweaving hierarchy and heterarchy in a consistent way is presented. The high level end result of this investigation is to re-envision Occam's Razor as something like: When in doubt, prefer hypotheses whose simplicity bundles are Pareto optimal, partly because doing so both permits and benefits from the construction of coherent dual networks comprising coordinated and consistent multipattern hierarchies and heterarchies.

Related papers

• Learning Taxonomic Trees with Hierarchical Representation Regularization for Large Multimodal Models [47.868429337792314]
  Large multimodal models (LMMs) often lack taxonomic knowledge, leading to low hierarchical visual recognition (HVR) consistency.<n>We propose Hierarchical Representation Regularization ($HiR2$), a simple plug-and-play regularizer that improves hierarchical consistency in LMMs.<n>$HiR2$ effectively captures taxonomic structures across diverse LMMs and fine-tuning methods.
  arXiv  Detail & Related papers  (2026-07-03T03:16:41Z)
• T-GINEE: A Tensor-Based Multilayer Graph Representation Learning [50.294562905744215]
  T-GINEE is a statistical regularization framework combining tensor-based generalized estimating equations with task-specific loss to model cross-network correlations explicitly.<n>Our theoretical analysis establishes consistency and normality under mild conditions.<n>Extensive experiments on synthetic and real-world datasets validate T-GINEE's effectiveness for multilayer network analysis.
  arXiv  Detail & Related papers  (2026-05-27T10:51:29Z)
• Hierarchical Concept Geometry in Language Models Emerges from Word Co-occurrence [9.453554184019108]
  We propose a distributional theory of how hypernymy is encoded geometrically in language representations.<n>We characterize theoretically the spectrum of the embedding Gram matrix of word2vec embeddings.<n>Our results indicate that hierarchical concept geometry in LLMs need not reflect a hierarchy-specific functional mechanism.
  arXiv  Detail & Related papers  (2026-05-22T16:24:30Z)
• A Geometrically-Grounded Drive for MDL-Based Optimization in Deep Learning [3.2452107817263003]
  This paper introduces a novel optimization framework that fundamentally integrates the Minimum Description Length (MDL) principle into the training dynamics of deep neural networks.<n>We provide a practical, computationally efficient algorithm with $O(N log N)$ per-iteration complexity (Theoremrefthm:complexity), alongside guarantees for numerical stability (Theoremrefthm:stability) and exponential convergence under convexity assumptions (Theoremrefthm:convergence_rate)<n>This work provides a principled path toward more autonomous, generalizable, and interpretable AI
  arXiv  Detail & Related papers  (2026-03-12T08:31:00Z)
• TopoOR: A Unified Topological Scene Representation for the Operating Room [54.50897207203292]
  Surgical Scene Graphs abstract the complexity of surgical operating rooms (OR) into a structure of entities and their relations.<n>We introduce TopoOR, a new paradigm that models multimodal operating rooms as a higher-order structure.<n>We also propose a higher-order attention mechanism that explicitly preserves manifold structure and modality-specific features.
  arXiv  Detail & Related papers  (2026-03-10T10:19:42Z)
• The Observer-Situation Lattice: A Unified Formal Basis for Perspective-Aware Cognition [2.28438857884398]
  We introduce the Observer-Situation Lattice (OSL), a unified mathematical structure that provides a single, coherent semantic space for perspective-aware cognition.<n>OSL is a finite complete lattice where each element represents a unique observer-situation pair, allowing for a principled and scalable approach to belief management.<n>We present two key algorithms that operate on this lattice: (i) Relativized Belief propagation, an incremental update algorithm that efficiently propagates new information, and (ii) Minimal Contradiction Decomposition, a graph-based procedure that identifies and isolates contradiction components.
  arXiv  Detail & Related papers  (2026-03-02T03:15:36Z)
• A Geometric Unification of Concept Learning with Concept Cones [58.70836885177496]
  Two traditions of interpretability have evolved side by side but seldom spoken to each other: Concept Bottleneck Models (CBMs) and Sparse Autoencoders (SAEs)<n>We show that both paradigms instantiate the same geometric structure.<n>CBMs provide human-defined reference geometries, while SAEs can be evaluated by how well their learned cones approximate or contain those of CBMs.
  arXiv  Detail & Related papers  (2025-12-08T09:51:46Z)
• Native Logical and Hierarchical Representations with Subspace Embeddings [25.274936769664098]
  We introduce a novel paradigm: embedding concepts as linear subspaces.<n>It naturally supports set-theoretic operations like intersection (conjunction) and linear sum (disjunction)<n>Our method achieves state-of-the-art results in reconstruction and link prediction on WordNet.
  arXiv  Detail & Related papers  (2025-08-21T18:29:17Z)
• Emergent Cognitive Convergence via Implementation: A Structured Loop Reflecting Four Theories of Mind (A Position Paper) [0.0]
  We report the discovery of a structural convergence across four influential theories of mind.<n>This convergence occurred within a practical AI agent architecture called Agentic Flow.<n>We introduce PEACE as a descriptive meta-architecture that captures design-level regularities observed in Agentic Flow.
  arXiv  Detail & Related papers  (2025-07-22T02:54:45Z)
• Simplicial methods in the resource theory of contextuality [0.0]
  Building on the theory of simplicial distributions, we introduce event scenarios as a functorial generalization of presheaf-theoretic measurement scenarios.<n>We define symmetric monoidal structures on these categories and extend the distribution functor to a setting, yielding a resource theory that generalizes the presheaf-theoretic notion of simulations.
  arXiv  Detail & Related papers  (2025-05-29T21:14:55Z)
• Provable Benefits of Complex Parameterizations for Structured State Space Models [51.90574950170374]
  Structured state space models (SSMs) are linear dynamical systems adhering to a specified structure. In contrast to typical neural network modules, whose parameterizations are real, SSMs often use complex parameterizations. This paper takes a step towards explaining the benefits of complex parameterizations for SSMs by establishing formal gaps between real and complex diagonal SSMs.
  arXiv  Detail & Related papers  (2024-10-17T22:35:50Z)
• Discovering Abstract Symbolic Relations by Learning Unitary Group Representations [7.303827428956944]
  We investigate a principled approach for symbolic operation completion (SOC) SOC poses a unique challenge in modeling abstract relationships between discrete symbols. We demonstrate that SOC can be efficiently solved by a minimal model - a bilinear map - with a novel factorized architecture.
  arXiv  Detail & Related papers  (2024-02-26T20:18:43Z)
• Complexity in Tame Quantum Theories [0.0]
  We introduce a framework that allows for quantifying the amount of logical information needed to specify a function or set. We derive the complexity of parameter-dependent physical observables and coupling functions appearing in effective Lagrangians.
  arXiv  Detail & Related papers  (2023-10-02T18:00:01Z)
• Understanding and Constructing Latent Modality Structures in Multi-modal Representation Learning [53.68371566336254]
  We argue that the key to better performance lies in meaningful latent modality structures instead of perfect modality alignment. Specifically, we design 1) a deep feature separation loss for intra-modality regularization; 2) a Brownian-bridge loss for inter-modality regularization; and 3) a geometric consistency loss for both intra- and inter-modality regularization.
  arXiv  Detail & Related papers  (2023-03-10T14:38:49Z)
• Synergies between Disentanglement and Sparsity: Generalization and Identifiability in Multi-Task Learning [79.83792914684985]
  We prove a new identifiability result that provides conditions under which maximally sparse base-predictors yield disentangled representations. Motivated by this theoretical result, we propose a practical approach to learn disentangled representations based on a sparsity-promoting bi-level optimization problem.
  arXiv  Detail & Related papers  (2022-11-26T21:02:09Z)
• On the Complexity of Counterfactual Reasoning [9.614694312155795]
  We show that counterfactual reasoning is no harder than associational or interventional reasoning on fully specified SCMs. We extend our results to general counterfactual reasoning that requires contemplating more than two worlds.
  arXiv  Detail & Related papers  (2022-11-24T07:28:17Z)
• On Neural Architecture Inductive Biases for Relational Tasks [76.18938462270503]
  We introduce a simple architecture based on similarity-distribution scores which we name Compositional Network generalization (CoRelNet) We find that simple architectural choices can outperform existing models in out-of-distribution generalizations.
  arXiv  Detail & Related papers  (2022-06-09T16:24:01Z)
• Dist2Cycle: A Simplicial Neural Network for Homology Localization [66.15805004725809]
  Simplicial complexes can be viewed as high dimensional generalizations of graphs that explicitly encode multi-way ordered relations. We propose a graph convolutional model for learning functions parametrized by the $k$-homological features of simplicial complexes.
  arXiv  Detail & Related papers  (2021-10-28T14:59:41Z)
• Contextuality in the Bundle Approach, n-Contextuality, and the Role of Holonomy [0.0]
  Contextuality can be understood as the impossibility to construct a globally consistent description of a model even if there is local agreement. We can describe contextuality with the bundle approach, where the scenario is represented as a simplicial complex. We introduce a hierarchy called n-contextuality to explore the dependence of contextual behavior of a model to the topology of the scenario.
  arXiv  Detail & Related papers  (2021-05-28T22:54:05Z)
• Joint Network Topology Inference via Structured Fusion Regularization [70.30364652829164]
  Joint network topology inference represents a canonical problem of learning multiple graph Laplacian matrices from heterogeneous graph signals. We propose a general graph estimator based on a novel structured fusion regularization. We show that the proposed graph estimator enjoys both high computational efficiency and rigorous theoretical guarantee.
  arXiv  Detail & Related papers  (2021-03-05T04:42:32Z)
• Consistency of Spectral Clustering on Hierarchical Stochastic Block Models [5.983753938303726]
  We study the hierarchy of communities in real-world networks under a generic block model. We prove the strong consistency of this method under a wide range of model parameters. Unlike most of existing work, our theory covers multiscale networks where the connection probabilities may differ by orders of magnitude.
  arXiv  Detail & Related papers  (2020-04-30T01:08:59Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.