Quasi-Equivariant Metanetworks

• URL: http://arxiv.org/abs/2604.23720v1
• Date: Sun, 26 Apr 2026 14:03:58 GMT
• Title: Quasi-Equivariant Metanetworks
• Abstract summary: We introduce the novel concept of quasi-equivariant metanetworks.<n>We show that quasi-equivariant metanetworks achieve good trade-offs between symmetry preservation and representational expressivity.<n>These findings provide a principled foundation for the design of more expressive and functionally robust metanetworks.
• Score: 3.4069385563446684
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Metanetworks are neural architectures designed to operate directly on pretrained weights to perform downstream tasks. However, the parameter space serves only as a proxy for the underlying function class, and the parameter-function mapping is inherently non-injective: distinct parameter configurations may yield identical input-output behaviors. As a result, metanetworks that rely solely on raw parameters risk overlooking the intrinsic symmetries of the architecture. Reasoning about functional identity is therefore essential for effective metanetwork design, motivating the development of equivariant metanetworks, which incorporate equivariance principles to respect architectural symmetries. Existing approaches, however, typically enforce strict equivariance, which imposes rigid constraints and often leads to sparse and less expressive models. To address this limitation, we introduce the novel concept of quasi-equivariance, which allows metanetworks to move beyond the rigidity of strict equivariance while still preserving functional identity. We lay down a principled basis for this framework and demonstrate its broad applicability across diverse neural architectures, including feedforward, convolutional, and transformer networks. Through empirical evaluation, we show that quasi-equivariant metanetworks achieve good trade-offs between symmetry preservation and representational expressivity. These findings advance the theoretical understanding of weight-space learning and provide a principled foundation for the design of more expressive and functionally robust metanetworks.

Related papers

• From Coefficients to Directions: Rethinking Model Merging with Directional Alignment [66.99062575537555]
  We introduce a unified geometric framework, emphMerging with Directional Alignment (method), which aligns directional structures consistently in both the parameter and feature spaces.<n>Our analysis shows that directional alignment improves structural coherence, and extensive experiments across benchmarks, model scales, and task configurations further validate the effectiveness of our approach.
  arXiv  Detail & Related papers  (2025-11-29T08:40:58Z)
• Multi-Modal Manipulation via Multi-Modal Policy Consensus [62.49978559936122]
  We propose a new approach to integrate diverse sensory modalities for robotic manipulation.<n>Our method factorizes the policy into a set of diffusion models, each specialized for a single representation.<n>We evaluate our approach on simulated manipulation tasks in RLBench, as well as real-world tasks such as occluded object picking, in-hand spoon reorientation, and puzzle insertion.
  arXiv  Detail & Related papers  (2025-09-27T19:43:04Z)
• Generalized Linear Mode Connectivity for Transformers [87.32299363530996]
  A striking phenomenon is linear mode connectivity (LMC), where independently trained models can be connected by low- or zero-loss paths.<n>Prior work has predominantly focused on neuron re-ordering through permutations, but such approaches are limited in scope.<n>We introduce a unified framework that captures four symmetry classes: permutations, semi-permutations, transformations, and general invertible maps.<n>This generalization enables, for the first time, the discovery of low- and zero-barrier linear paths between independently trained Vision Transformers and GPT-2 models.
  arXiv  Detail & Related papers  (2025-06-28T01:46:36Z)
• On Universality Classes of Equivariant Networks [9.137637807153464]
  We investigate the approximation power of equivariant neural networks beyond separation constraints.<n>We show that separation power does not fully capture expressivity.<n>We identify settings where shallow equivariant networks do achieve universality.
  arXiv  Detail & Related papers  (2025-06-02T22:07:52Z)
• Spectral Normalization and Voigt-Reuss net: A universal approach to microstructure-property forecasting with physical guarantees [0.0]
  A crucial step in the design process is the rapid evaluation of effective mechanical, thermal, or, in general, elasticity properties.<n>The classical simulation-based approach, which uses, e.g., finite elements and FFT-based solvers, can require substantial computational resources.<n>We propose a novel spectral normalization scheme that a priori enforces these bounds.
  arXiv  Detail & Related papers  (2025-04-01T12:21:57Z)
• Improving Equivariant Networks with Probabilistic Symmetry Breaking [9.164167226137664]
  Equivariant networks encode known symmetries into neural networks, often enhancing generalizations.<n>This poses an important problem, both (1) for prediction tasks on domains where self-symmetries are common, and (2) for generative models, which must break symmetries in order to reconstruct from highly symmetric latent spaces.<n>We present novel theoretical results that establish sufficient conditions for representing such distributions.
  arXiv  Detail & Related papers  (2025-03-27T21:04:49Z)
• Semantic Loss Functions for Neuro-Symbolic Structured Prediction [74.18322585177832]
  We discuss the semantic loss, which injects knowledge about such structure, defined symbolically, into training. It is agnostic to the arrangement of the symbols, and depends only on the semantics expressed thereby. It can be combined with both discriminative and generative neural models.
  arXiv  Detail & Related papers  (2024-05-12T22:18:25Z)
• From Bricks to Bridges: Product of Invariances to Enhance Latent Space Communication [19.336940758147442]
  It has been observed that representations learned by distinct neural networks conceal structural similarities when the models are trained under similar inductive biases. We introduce a versatile method to directly incorporate a set of invariances into the representations, constructing a product space of invariant components on top of the latent representations. We validate our solution on classification and reconstruction tasks, observing consistent latent similarity and downstream performance improvements in a zero-shot stitching setting.
  arXiv  Detail & Related papers  (2023-10-02T13:55:38Z)
• Equivariance with Learned Canonicalization Functions [77.32483958400282]
  We show that learning a small neural network to perform canonicalization is better than using predefineds. Our experiments show that learning the canonicalization function is competitive with existing techniques for learning equivariant functions across many tasks.
  arXiv  Detail & Related papers  (2022-11-11T21:58:15Z)
• A Simple Strategy to Provable Invariance via Orbit Mapping [14.127786615513978]
  We propose a method to make network architectures provably invariant with respect to group actions. In a nutshell, we intend to 'undo' any possible transformation before feeding the data into the actual network.
  arXiv  Detail & Related papers  (2022-09-24T03:40:42Z)
• Equivariance and generalization in neural networks [0.0]
  We focus on the consequences of incorporating translational equivariance among the network properties. The benefits of equivariant networks are exemplified by studying a complex scalar field theory. In most of the tasks our best equivariant architectures can perform and generalize significantly better than their non-equivariant counterparts.
  arXiv  Detail & Related papers  (2021-12-23T12:38:32Z)
• Frame Averaging for Invariant and Equivariant Network Design [50.87023773850824]
  We introduce Frame Averaging (FA), a framework for adapting known (backbone) architectures to become invariant or equivariant to new symmetry types. We show that FA-based models have maximal expressive power in a broad setting. We propose a new class of universal Graph Neural Networks (GNNs), universal Euclidean motion invariant point cloud networks, and Euclidean motion invariant Message Passing (MP) GNNs.
  arXiv  Detail & Related papers  (2021-10-07T11:05:23Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.