HyperNet Fields: Efficiently Training Hypernetworks without Ground Truth by Learning Weight Trajectories

• URL: http://arxiv.org/abs/2412.17040v1
• Date: Sun, 22 Dec 2024 14:37:10 GMT
• Title: HyperNet Fields: Efficiently Training Hypernetworks without Ground Truth by Learning Weight Trajectories
• Authors: Eric Hedlin, Munawar Hayat, Fatih Porikli, Kwang Moo Yi, Shweta Mahajan,
• Abstract summary: We propose a method to train hypernetworks, without the need for any per-sample ground truth. Our key idea is to learn a Hypernetwork Field and estimate the entire trajectory of network weight training instead of simply its converged state.
• Score: 62.975803165786324
• License:
• Abstract: To efficiently adapt large models or to train generative models of neural representations, Hypernetworks have drawn interest. While hypernetworks work well, training them is cumbersome, and often requires ground truth optimized weights for each sample. However, obtaining each of these weights is a training problem of its own-one needs to train, e.g., adaptation weights or even an entire neural field for hypernetworks to regress to. In this work, we propose a method to train hypernetworks, without the need for any per-sample ground truth. Our key idea is to learn a Hypernetwork `Field` and estimate the entire trajectory of network weight training instead of simply its converged state. In other words, we introduce an additional input to the Hypernetwork, the convergence state, which then makes it act as a neural field that models the entire convergence pathway of a task network. A critical benefit in doing so is that the gradient of the estimated weights at any convergence state must then match the gradients of the original task -- this constraint alone is sufficient to train the Hypernetwork Field. We demonstrate the effectiveness of our method through the task of personalized image generation and 3D shape reconstruction from images and point clouds, demonstrating competitive results without any per-sample ground truth.

Related papers

• HyperInterval: Hypernetwork approach to training weight interval regions in continual learning [0.0]
  Interval Continual Learning (InterContiNet) relies on enforcing interval constraints on the neural network parameter space. We introduce our, a technique that employs interval arithmetic within the embedding space. our obtains significantly better results than InterContiNet and gives SOTA results on several benchmarks.
  arXiv  Detail & Related papers  (2024-05-24T11:20:41Z)
• Principled Weight Initialization for Hypernetworks [15.728811174027896]
  Hypernetworks are meta neural networks that generate weights for a main neural network in an end-to-end differentiable manner. We show that classical weight-as-a-service methods fail to produce weights for the mainnet in the correct scale. We develop principled techniques for weight-as-a-service in hypernets, and show that they lead to more stable mainnet weights, lower training loss, and faster convergence.
  arXiv  Detail & Related papers  (2023-12-13T04:49:18Z)
• HyperMask: Adaptive Hypernetwork-based Masks for Continual Learning [0.0]
  We propose a method called HyperMask, which dynamically filters a target network depending on the CL task. Due to the lottery ticket hypothesis, we can use a single network with weighted forgettings.
  arXiv  Detail & Related papers  (2023-09-29T20:01:11Z)
• Magnitude Invariant Parametrizations Improve Hypernetwork Learning [0.0]
  Hypernetworks are powerful neural networks that predict the parameters of another neural network. Training typically converges far more slowly than for non-hypernetwork models. We identify a fundamental and previously unidentified problem that contributes to the challenge of training hypernetworks. We present a simple solution to this problem using a revised hypernetwork formulation that we call Magnitude Invariant Parametrizations (MIP)
  arXiv  Detail & Related papers  (2023-04-15T22:18:29Z)
• Random Weights Networks Work as Loss Prior Constraint for Image Restoration [50.80507007507757]
  We present our belief Random Weights Networks can be Acted as Loss Prior Constraint for Image Restoration'' Our belief can be directly inserted into existing networks without any training and testing computational cost. To emphasize, our main focus is to spark the realms of loss function and save their current neglected status.
  arXiv  Detail & Related papers  (2023-03-29T03:43:51Z)
• Dual Lottery Ticket Hypothesis [71.95937879869334]
  Lottery Ticket Hypothesis (LTH) provides a novel view to investigate sparse network training and maintain its capacity. In this work, we regard the winning ticket from LTH as the subnetwork which is in trainable condition and its performance as our benchmark. We propose a simple sparse network training strategy, Random Sparse Network Transformation (RST), to substantiate our DLTH.
  arXiv  Detail & Related papers  (2022-03-08T18:06:26Z)
• Simultaneous Training of Partially Masked Neural Networks [67.19481956584465]
  We show that it is possible to train neural networks in such a way that a predefined 'core' subnetwork can be split-off from the trained full network with remarkable good performance. We show that training a Transformer with a low-rank core gives a low-rank model with superior performance than when training the low-rank model alone.
  arXiv  Detail & Related papers  (2021-06-16T15:57:51Z)
• Learning Neural Network Subspaces [74.44457651546728]
  Recent observations have advanced our understanding of the neural network optimization landscape. With a similar computational cost as training one model, we learn lines, curves, and simplexes of high-accuracy neural networks. With a similar computational cost as training one model, we learn lines, curves, and simplexes of high-accuracy neural networks.
  arXiv  Detail & Related papers  (2021-02-20T23:26:58Z)
• Cream of the Crop: Distilling Prioritized Paths For One-Shot Neural Architecture Search [60.965024145243596]
  One-shot weight sharing methods have recently drawn great attention in neural architecture search due to high efficiency and competitive performance. To alleviate this problem, we present a simple yet effective architecture distillation method. We introduce the concept of prioritized path, which refers to the architecture candidates exhibiting superior performance during training. Since the prioritized paths are changed on the fly depending on their performance and complexity, the final obtained paths are the cream of the crop.
  arXiv  Detail & Related papers  (2020-10-29T17:55:05Z)
• Training highly effective connectivities within neural networks with randomly initialized, fixed weights [4.56877715768796]
  We introduce a novel way of training a network by flipping the signs of the weights. We obtain good results even with weights constant magnitude or even when weights are drawn from highly asymmetric distributions.
  arXiv  Detail & Related papers  (2020-06-30T09:41:18Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.