AdaGrid: Adaptive Grid Search for Link Prediction Training Objective

• URL: http://arxiv.org/abs/2203.16162v1
• Date: Wed, 30 Mar 2022 09:24:17 GMT
• Title: AdaGrid: Adaptive Grid Search for Link Prediction Training Objective
• Authors: Tim Po\v{s}tuvan, Jiaxuan You, Mohammadreza Banaei, R\'emi Lebret, Jure Leskovec
• Abstract summary: Training objective crucially influences the model's performance and generalization capabilities. We propose Adaptive Grid Search (AdaGrid) which dynamically adjusts the edge message ratio during training. We show that AdaGrid can boost the performance of the models up to $1.9%$ while being nine times more time-efficient than a complete search.
• Score: 58.79804082133998
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: One of the most important factors that contribute to the success of a machine learning model is a good training objective. Training objective crucially influences the model's performance and generalization capabilities. This paper specifically focuses on graph neural network training objective for link prediction, which has not been explored in the existing literature. Here, the training objective includes, among others, a negative sampling strategy, and various hyperparameters, such as edge message ratio which controls how training edges are used. Commonly, these hyperparameters are fine-tuned by complete grid search, which is very time-consuming and model-dependent. To mitigate these limitations, we propose Adaptive Grid Search (AdaGrid), which dynamically adjusts the edge message ratio during training. It is model agnostic and highly scalable with a fully customizable computational budget. Through extensive experiments, we show that AdaGrid can boost the performance of the models up to $1.9\%$ while being nine times more time-efficient than a complete search. Overall, AdaGrid represents an effective automated algorithm for designing machine learning training objectives.

Related papers

• Learning the Regularization Strength for Deep Fine-Tuning via a Data-Emphasized Variational Objective [4.453137996095194]
  grid search is computationally expensive, requires carving out a validation set, and requires practitioners to specify candidate values. Our proposed technique overcomes all three disadvantages of grid search. We demonstrate effectiveness on image classification tasks on several datasets, yielding heldout accuracy comparable to existing approaches.
  arXiv  Detail & Related papers  (2024-10-25T16:32:11Z)
• Controllable Unlearning for Image-to-Image Generative Models via $\varepsilon$-Constrained Optimization [12.627103884294476]
  We study the machine unlearning problem in Image-to-Image (I2I) generative models. Previous studies mainly treat it as a single objective optimization problem, offering a solitary solution. We propose a controllable unlearning framework that uses a control coefficient $varepsilon$ to control the trade-off.
  arXiv  Detail & Related papers  (2024-08-03T07:04:55Z)
• GraphFM: A Comprehensive Benchmark for Graph Foundation Model [33.157367455390144]
  Foundation Models (FMs) serve as a general class for the development of artificial intelligence systems. Despite extensive research into self-supervised learning as the cornerstone of FMs, several outstanding issues persist. The extent of generalization capability on downstream tasks remains unclear. It is unknown how effectively these models can scale to large datasets.
  arXiv  Detail & Related papers  (2024-06-12T15:10:44Z)
• Accurate Neural Network Pruning Requires Rethinking Sparse Optimization [87.90654868505518]
  We show the impact of high sparsity on model training using the standard computer vision and natural language processing sparsity benchmarks. We provide new approaches for mitigating this issue for both sparse pre-training of vision models and sparse fine-tuning of language models.
  arXiv  Detail & Related papers  (2023-08-03T21:49:14Z)
• Controlled Sparsity via Constrained Optimization or: How I Learned to Stop Tuning Penalties and Love Constraints [81.46143788046892]
  We focus on the task of controlling the level of sparsity when performing sparse learning. Existing methods based on sparsity-inducing penalties involve expensive trial-and-error tuning of the penalty factor. We propose a constrained formulation where sparsification is guided by the training objective and the desired sparsity target in an end-to-end fashion.
  arXiv  Detail & Related papers  (2022-08-08T21:24:20Z)
• Hybrid quantum ResNet for car classification and its hyperparameter optimization [0.0]
  This paper presents a quantum-inspired hyperparameter optimization technique and a hybrid quantum-classical machine learning model for supervised learning. We test our approaches in a car image classification task and demonstrate a full-scale implementation of the hybrid quantum ResNet model. A classification accuracy of 0.97 was obtained by the hybrid model after 18 iterations, whereas the classical model achieved an accuracy of 0.92 after 75 iterations.
  arXiv  Detail & Related papers  (2022-05-10T13:25:36Z)
• Neural Capacitance: A New Perspective of Neural Network Selection via Edge Dynamics [85.31710759801705]
  Current practice requires expensive computational costs in model training for performance prediction. We propose a novel framework for neural network selection by analyzing the governing dynamics over synaptic connections (edges) during training. Our framework is built on the fact that back-propagation during neural network training is equivalent to the dynamical evolution of synaptic connections.
  arXiv  Detail & Related papers  (2022-01-11T20:53:15Z)
• MT3: Meta Test-Time Training for Self-Supervised Test-Time Adaption [69.76837484008033]
  An unresolved problem in Deep Learning is the ability of neural networks to cope with domain shifts during test-time. We combine meta-learning, self-supervision and test-time training to learn to adapt to unseen test distributions. Our approach significantly improves the state-of-the-art results on the CIFAR-10-Corrupted image classification benchmark.
  arXiv  Detail & Related papers  (2021-03-30T09:33:38Z)
• TrackMPNN: A Message Passing Graph Neural Architecture for Multi-Object Tracking [8.791710193028903]
  This study follows many previous approaches to multi-object tracking (MOT) that model the problem using graph-based data structures. We create a framework based on dynamic undirected graphs that represent the data association problem over multiple timesteps. We also provide solutions and propositions for the computational problems that need to be addressed to create a memory-efficient, real-time, online algorithm.
  arXiv  Detail & Related papers  (2021-01-11T21:52:25Z)
• Graph Ordering: Towards the Optimal by Learning [69.72656588714155]
  Graph representation learning has achieved a remarkable success in many graph-based applications, such as node classification, prediction, and community detection. However, for some kind of graph applications, such as graph compression and edge partition, it is very hard to reduce them to some graph representation learning tasks. In this paper, we propose to attack the graph ordering problem behind such applications by a novel learning approach.
  arXiv  Detail & Related papers  (2020-01-18T09:14:16Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.