TorchDEQ: A Library for Deep Equilibrium Models

• URL: http://arxiv.org/abs/2310.18605v1
• Date: Sat, 28 Oct 2023 06:16:10 GMT
• Title: TorchDEQ: A Library for Deep Equilibrium Models
• Authors: Zhengyang Geng, J. Zico Kolter
• Abstract summary: We present TorchDEQ, an out-of-the-box library that allows users to define, train, and infer using DEQs over multiple domains with minimal code and best practices. We build a DEQ Zoo'' that supports six published implicit models across different domains. We have substantially improved the performance, training stability, and efficiency of DEQs on ten datasets across all six projects in the DEQ Zoo.
• Score: 72.65236284030894
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Deep Equilibrium (DEQ) Models, an emerging class of implicit models that maps inputs to fixed points of neural networks, are of growing interest in the deep learning community. However, training and applying DEQ models is currently done in an ad-hoc fashion, with various techniques spread across the literature. In this work, we systematically revisit DEQs and present TorchDEQ, an out-of-the-box PyTorch-based library that allows users to define, train, and infer using DEQs over multiple domains with minimal code and best practices. Using TorchDEQ, we build a ``DEQ Zoo'' that supports six published implicit models across different domains. By developing a joint framework that incorporates the best practices across all models, we have substantially improved the performance, training stability, and efficiency of DEQs on ten datasets across all six projects in the DEQ Zoo. TorchDEQ and DEQ Zoo are released as \href{https://github.com/locuslab/torchdeq}{open source}.

Related papers

• MALPOLON: A Framework for Deep Species Distribution Modeling [3.1457219084519004]
  MALPOLON aims to facilitate training and inferences of deep species distribution models (deep-SDM) It is written in Python and built upon the PyTorch library. The framework is open-sourced on GitHub and PyPi.
  arXiv  Detail & Related papers  (2024-09-26T17:45:10Z)
• FreeSeg-Diff: Training-Free Open-Vocabulary Segmentation with Diffusion Models [56.71672127740099]
  We focus on the task of image segmentation, which is traditionally solved by training models on closed-vocabulary datasets. We leverage different and relatively small-sized, open-source foundation models for zero-shot open-vocabulary segmentation. Our approach (dubbed FreeSeg-Diff), which does not rely on any training, outperforms many training-based approaches on both Pascal VOC and COCO datasets.
  arXiv  Detail & Related papers  (2024-03-29T10:38:25Z)
• Mixture-Models: a one-stop Python Library for Model-based Clustering using various Mixture Models [4.60168321737677]
  textttMixture-Models is an open-source Python library for fitting Gaussian Mixture Models (GMM) and their variants. It streamlines the implementation and analysis of these models using various first/second order optimization routines. The library provides user-friendly model evaluation tools, such as BIC, AIC, and log-likelihood estimation.
  arXiv  Detail & Related papers  (2024-02-08T19:34:24Z)
• Learning a Diffusion Model Policy from Rewards via Q-Score Matching [93.0191910132874]
  We present a theoretical framework linking the structure of diffusion model policies to a learned Q-function. We propose a new policy update method from this theory, which we denote Q-score matching.
  arXiv  Detail & Related papers  (2023-12-18T23:31:01Z)
• FlexModel: A Framework for Interpretability of Distributed Large Language Models [0.0]
  We present FlexModel, a software package providing a streamlined interface for engaging with models distributed across multi- GPU and multi-node configurations. The library is compatible with existing model distribution libraries and encapsulates PyTorch models. It exposes user-registerable HookFunctions to facilitate straightforward interaction with distributed model internals.
  arXiv  Detail & Related papers  (2023-12-05T21:19:33Z)
• PyRCA: A Library for Metric-based Root Cause Analysis [66.72542200701807]
  PyRCA is an open-source machine learning library of Root Cause Analysis (RCA) for Artificial Intelligence for IT Operations (AIOps) It provides a holistic framework to uncover the complicated metric causal dependencies and automatically locate root causes of incidents.
  arXiv  Detail & Related papers  (2023-06-20T09:55:10Z)
• Chain-of-Skills: A Configurable Model for Open-domain Question Answering [79.8644260578301]
  The retrieval model is an indispensable component for real-world knowledge-intensive tasks. Recent work focuses on customized methods, limiting the model transferability and scalability. We propose a modular retriever where individual modules correspond to key skills that can be reused across datasets.
  arXiv  Detail & Related papers  (2023-05-04T20:19:39Z)
• Pythae: Unifying Generative Autoencoders in Python -- A Benchmarking Use Case [0.0]
  We present Pythae, a versatile open-source Python library providing straightforward, reproducible and reliable use of generative autoencoder models. We present and compare 19 generative autoencoder models representative of some of the main improvements on downstream tasks.
  arXiv  Detail & Related papers  (2022-06-16T17:11:41Z)
• TyXe: Pyro-based Bayesian neural nets for Pytorch [12.343312954353639]
  We introduce TyXe, a Bayesian neural network library built on top of Pytorch and Pyro. Our leading design principle is to cleanly separate architecture, prior, inference and likelihood specification. In contrast to existing packages TyXe does not implement any layer classes, and instead relies on architectures defined in generic Pytorch code.
  arXiv  Detail & Related papers  (2021-10-01T09:04:26Z)
• Multiscale Deep Equilibrium Models [162.15362280927476]
  We propose a new class of implicit networks, the multiscale deep equilibrium model (MDEQ) An MDEQ directly solves for and backpropagates through the equilibrium points of multiple feature resolutions simultaneously. We illustrate the effectiveness of this approach on two large-scale vision tasks: ImageNet classification and semantic segmentation on high-resolution images from the Cityscapes dataset.
  arXiv  Detail & Related papers  (2020-06-15T18:07:44Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.