Equilibrium Aggregation: Encoding Sets via Optimization

• URL: http://arxiv.org/abs/2202.12795v1
• Date: Fri, 25 Feb 2022 16:13:55 GMT
• Title: Equilibrium Aggregation: Encoding Sets via Optimization
• Abstract summary: We propose an optimization-based method called Equilibrium Aggregation. We show that many existing aggregation methods can be recovered as special cases of Equilibrium Aggregation. We validate its efficiency on three different tasks: median estimation, class counting, and molecular property prediction.
• Score: 4.849550522970841
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Processing sets or other unordered, potentially variable-sized inputs in neural networks is usually handled by \emph{aggregating} a number of input tensors into a single representation. While a number of aggregation methods already exist from simple sum pooling to multi-head attention, they are limited in their representational power both from theoretical and empirical perspectives. On the search of a principally more powerful aggregation strategy, we propose an optimization-based method called Equilibrium Aggregation. We show that many existing aggregation methods can be recovered as special cases of Equilibrium Aggregation and that it is provably more efficient in some important cases. Equilibrium Aggregation can be used as a drop-in replacement in many existing architectures and applications. We validate its efficiency on three different tasks: median estimation, class counting, and molecular property prediction. In all experiments, Equilibrium Aggregation achieves higher performance than the other aggregation techniques we test.

Related papers

• The Interplay Between Interpolation and Aggregation in Regression: Optimal Sample Complexity [21.276981570672064]
  We show that some hypothesis classes are learnable only by aggregating infinitely many hypotheses.<n>We prove that an extremely simple aggregation procedure, combining three interpolating hypotheses via the median, is optimal among all these aggregation procedures.
  arXiv  Detail & Related papers  (2026-05-28T12:02:09Z)
• Balanced Aggregation: Understanding and Fixing Aggregation Bias in GRPO [70.38763678943648]
  Reinforcement learning with verifiable rewards (RLVR) has become a central paradigm for improving reasoning and code generation in large language models.<n>Standard GRPO uses sequence aggregation, while recent work has advocated token aggregation as a better alternative.<n>We show that these two rules induce different optimization biases; token aggregation introduces sign-length coupling, while sequence aggregation implicitly downweights longer responses.
  arXiv  Detail & Related papers  (2026-04-14T09:48:46Z)
• SOLVAR: Fast covariance-based heterogeneity analysis with pose refinement for cryo-EM [1.739627424017212]
  Cryo-electron microscopy (cryo-EM) has emerged as a powerful technique for resolving the three-dimensional structures of macromolecules.<n>A key challenge in cryo-EM is characterizing continuous heterogeneity, where molecules adopt a continuum of conformational states.<n>Covariance-based methods offer a principled approach to modeling structural variability.
  arXiv  Detail & Related papers  (2026-02-19T18:28:46Z)
• Almost Asymptotically Optimal Active Clustering Through Pairwise Observations [59.20614082241528]
  We propose a new analysis framework for clustering $M$ items into an unknown number of $K$ distinct groups using noisy and actively collected responses.<n>We establish a fundamental lower bound on the expected number of queries needed to achieve a desired confidence in the accuracy of the clustering.<n>We develop a computationally feasible variant of the Generalized Likelihood Ratio statistic and show that its performance gap to the lower bound can be accurately empirically estimated.
  arXiv  Detail & Related papers  (2026-02-05T14:16:47Z)
• A Unified Probabilistic Framework for Dictionary Learning with Parsimonious Activation [10.775460285501739]
  We introduce a parsimony promoting regularizer based on the row-wise $L_infty$ norm of the coefficient matrix.<n>This additional penalty encourages entire rows of the coefficient matrix to vanish, thereby reducing the number of dictionary atoms activated across the dataset.
  arXiv  Detail & Related papers  (2025-09-30T02:46:11Z)
• Parameter-free approximate equivariance for tasks with finite group symmetry [15.964726158869777]
  Equivariant neural networks incorporate symmetries through group actions, embedding them as an inductive bias to improve performance on a wide variety of tasks.<n>We propose a simple zero- parameter approach that imposes approximate equivariance for a finite group in the latent representation, as an additional term in the loss function.<n>We benchmark our approach on three datasets and compare it against several existing equivariant methods, showing that it achieves similar or better performance for a fraction of the parameters.
  arXiv  Detail & Related papers  (2025-06-09T21:23:26Z)
• Adaptive Sampled Softmax with Inverted Multi-Index: Methods, Theory and Applications [79.53938312089308]
  The MIDX-Sampler is a novel adaptive sampling strategy based on an inverted multi-index approach. Our method is backed by rigorous theoretical analysis, addressing key concerns such as sampling bias, gradient bias, convergence rates, and generalization error bounds.
  arXiv  Detail & Related papers  (2025-01-15T04:09:21Z)
• BI-EqNO: Generalized Approximate Bayesian Inference with an Equivariant Neural Operator Framework [9.408644291433752]
  We introduce BI-EqNO, an equivariant neural operator framework for generalized approximate Bayesian inference. BI-EqNO transforms priors into posteriors on conditioned observation data through data-driven training. We demonstrate BI-EqNO's utility through two examples: (1) as a generalized Gaussian process (gGP) for regression, and (2) as an ensemble neural filter (EnNF) for sequential data assimilation.
  arXiv  Detail & Related papers  (2024-10-21T18:39:16Z)
• Near-Optimal Resilient Aggregation Rules for Distributed Learning Using 1-Center and 1-Mean Clustering with Outliers [24.88026399458157]
  Byzantine machine learning has garnered considerable attention in light of the unpredictable faults that can occur. The key to secure machines in distributed learning is resilient aggregation mechanisms.
  arXiv  Detail & Related papers  (2023-12-20T08:36:55Z)
• Deep Similarity Learning Loss Functions in Data Transformation for Class Imbalance [2.693342141713236]
  In this paper, we use deep neural networks to train new representations of multi-class data. Our proposal modifies the distribution of features, i.e. the positions of examples in the learned embedded representation, and it does not modify the class sizes. Experiments with popular multi-class imbalanced benchmark data sets and three classifiers showed the advantage of the proposed approach.
  arXiv  Detail & Related papers  (2023-12-16T23:10:09Z)
• Rethinking Clustering-Based Pseudo-Labeling for Unsupervised Meta-Learning [146.11600461034746]
  Method for unsupervised meta-learning, CACTUs, is a clustering-based approach with pseudo-labeling. This approach is model-agnostic and can be combined with supervised algorithms to learn from unlabeled data. We prove that the core reason for this is lack of a clustering-friendly property in the embedding space.
  arXiv  Detail & Related papers  (2022-09-27T19:04:36Z)
• Consensual Aggregation on Random Projected High-dimensional Features for Regression [0.0]
  We present a study of a kernel-based consensual aggregation on randomly projected high-dimensional features of predictions for regression. We numerically illustrate that the aggregation scheme upholds its performance on very large and highly correlated features. The efficiency of the proposed method is illustrated through several experiments evaluated on different types of synthetic and real datasets.
  arXiv  Detail & Related papers  (2022-04-06T06:35:47Z)
• Equivariance Allows Handling Multiple Nuisance Variables When Analyzing Pooled Neuroimaging Datasets [53.34152466646884]
  In this paper, we show how bringing recent results on equivariant representation learning instantiated on structured spaces together with simple use of classical results on causal inference provides an effective practical solution. We demonstrate how our model allows dealing with more than one nuisance variable under some assumptions and can enable analysis of pooled scientific datasets in scenarios that would otherwise entail removing a large portion of the samples.
  arXiv  Detail & Related papers  (2022-03-29T04:54:06Z)
• A Stochastic Newton Algorithm for Distributed Convex Optimization [62.20732134991661]
  We analyze a Newton algorithm for homogeneous distributed convex optimization, where each machine can calculate gradients of the same population objective. We show that our method can reduce the number, and frequency, of required communication rounds compared to existing methods without hurting performance.
  arXiv  Detail & Related papers  (2021-10-07T17:51:10Z)
• Flexible Model Aggregation for Quantile Regression [92.63075261170302]
  Quantile regression is a fundamental problem in statistical learning motivated by a need to quantify uncertainty in predictions. We investigate methods for aggregating any number of conditional quantile models. All of the models we consider in this paper can be fit using modern deep learning toolkits.
  arXiv  Detail & Related papers  (2021-02-26T23:21:16Z)
• Learning from Aggregate Observations [82.44304647051243]
  We study the problem of learning from aggregate observations where supervision signals are given to sets of instances. We present a general probabilistic framework that accommodates a variety of aggregate observations. Simple maximum likelihood solutions can be applied to various differentiable models.
  arXiv  Detail & Related papers  (2020-04-14T06:18:50Z)
• M2m: Imbalanced Classification via Major-to-minor Translation [79.09018382489506]
  In most real-world scenarios, labeled training datasets are highly class-imbalanced, where deep neural networks suffer from generalizing to a balanced testing criterion. In this paper, we explore a novel yet simple way to alleviate this issue by augmenting less-frequent classes via translating samples from more-frequent classes. Our experimental results on a variety of class-imbalanced datasets show that the proposed method improves the generalization on minority classes significantly compared to other existing re-sampling or re-weighting methods.
  arXiv  Detail & Related papers  (2020-04-01T13:21:17Z)
• Statistically Guided Divide-and-Conquer for Sparse Factorization of Large Matrix [2.345015036605934]
  We formulate the statistical problem as a sparse factor regression and tackle it with a divide-conquer approach. In the first stage division, we consider both latent parallel approaches for simplifying the task into a set of co-parsesparserank estimation (CURE) problems. In the second stage division, we innovate a stagewise learning technique, consisting of a sequence simple incremental paths, to efficiently trace out the whole solution of CURE.
  arXiv  Detail & Related papers  (2020-03-17T19:12:21Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.