Content-Based Search for Deep Generative Models

• URL: http://arxiv.org/abs/2210.03116v4
• Date: Tue, 24 Oct 2023 04:26:10 GMT
• Title: Content-Based Search for Deep Generative Models
• Authors: Daohan Lu, Sheng-Yu Wang, Nupur Kumari, Rohan Agarwal, Mia Tang, David Bau, Jun-Yan Zhu
• Abstract summary: We introduce the task of content-based model search: given a query and a large set of generative models, finding the models that best match the query. As each generative model produces a distribution of images, we formulate the search task as an optimization problem to select the model with the highest probability of generating similar content as the query. We demonstrate that our method outperforms several baselines on Generative Model Zoo, a new benchmark we create for the model retrieval task.
• Score: 45.322081206025544
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The growing proliferation of customized and pretrained generative models has made it infeasible for a user to be fully cognizant of every model in existence. To address this need, we introduce the task of content-based model search: given a query and a large set of generative models, finding the models that best match the query. As each generative model produces a distribution of images, we formulate the search task as an optimization problem to select the model with the highest probability of generating similar content as the query. We introduce a formulation to approximate this probability given the query from different modalities, e.g., image, sketch, and text. Furthermore, we propose a contrastive learning framework for model retrieval, which learns to adapt features for various query modalities. We demonstrate that our method outperforms several baselines on Generative Model Zoo, a new benchmark we create for the model retrieval task.

Related papers

• Embedding-based statistical inference on generative models [10.948308354932639]
  We extend results related to embedding-based representations of generative models to classical statistical inference settings. We demonstrate that using the perspective space as the basis of a notion of "similar" is effective for multiple model-level inference tasks.
  arXiv  Detail & Related papers  (2024-10-01T22:28:39Z)
• Consistent estimation of generative model representations in the data kernel perspective space [13.099029073152257]
  Generative models, such as large language models and text-to-image diffusion models, produce relevant information when presented a query. Different models may produce different information when presented the same query. We present novel theoretical results for embedding-based representations of generative models in the context of a set of queries.
  arXiv  Detail & Related papers  (2024-09-25T19:35:58Z)
• Query-oriented Data Augmentation for Session Search [71.84678750612754]
  We propose query-oriented data augmentation to enrich search logs and empower the modeling. We generate supplemental training pairs by altering the most important part of a search context. We develop several strategies to alter the current query, resulting in new training data with varying degrees of difficulty.
  arXiv  Detail & Related papers  (2024-07-04T08:08:33Z)
• FusionBench: A Comprehensive Benchmark of Deep Model Fusion [78.80920533793595]
  Deep model fusion is a technique that unifies the predictions or parameters of several deep neural networks into a single model. FusionBench is the first comprehensive benchmark dedicated to deep model fusion.
  arXiv  Detail & Related papers  (2024-06-05T13:54:28Z)
• PAMI: partition input and aggregate outputs for model interpretation [69.42924964776766]
  In this study, a simple yet effective visualization framework called PAMI is proposed based on the observation that deep learning models often aggregate features from local regions for model predictions. The basic idea is to mask majority of the input and use the corresponding model output as the relative contribution of the preserved input part to the original model prediction. Extensive experiments on multiple tasks confirm the proposed method performs better than existing visualization approaches in more precisely finding class-specific input regions.
  arXiv  Detail & Related papers  (2023-02-07T08:48:34Z)
• Model ensemble instead of prompt fusion: a sample-specific knowledge transfer method for few-shot prompt tuning [85.55727213502402]
  We focus on improving the few-shot performance of prompt tuning by transferring knowledge from soft prompts of source tasks. We propose Sample-specific Ensemble of Source Models (SESoM) SESoM learns to adjust the contribution of each source model for each target sample separately when ensembling source model outputs.
  arXiv  Detail & Related papers  (2022-10-23T01:33:16Z)
• An Ample Approach to Data and Modeling [1.0152838128195467]
  We describe a framework for modeling how models can be built that integrates concepts and methods from a wide range of fields. The reference M* meta model framework is presented, which relies critically in associating whole datasets and respective models in terms of a strict equivalence relation. Several considerations about how the developed framework can provide insights about data clustering, complexity, collaborative research, deep learning, and creativity are then presented.
  arXiv  Detail & Related papers  (2021-10-05T01:26:09Z)
• Model-agnostic multi-objective approach for the evolutionary discovery of mathematical models [55.41644538483948]
  In modern data science, it is more interesting to understand the properties of the model, which parts could be replaced to obtain better results. We use multi-objective evolutionary optimization for composite data-driven model learning to obtain the algorithm's desired properties.
  arXiv  Detail & Related papers  (2021-07-07T11:17:09Z)
• Conditional Generative Models for Counterfactual Explanations [0.0]
  We propose a general framework to generate sparse, in-distribution counterfactual model explanations. The framework is flexible with respect to the type of generative model used as well as the task of the underlying predictive model.
  arXiv  Detail & Related papers  (2021-01-25T14:31:13Z)
• ZSCRGAN: A GAN-based Expectation Maximization Model for Zero-Shot Retrieval of Images from Textual Descriptions [13.15755441853131]
  We propose a novel GAN-based model for zero-shot text to image retrieval. The proposed model is trained using an Expectation-Maximization framework. Experiments on multiple benchmark datasets show that our proposed model comfortably outperforms several state-of-the-art zero-shot text to image retrieval models.
  arXiv  Detail & Related papers  (2020-07-23T18:50:03Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.