On Designing Good Representation Learning Models
- URL: http://arxiv.org/abs/2107.05948v1
- Date: Tue, 13 Jul 2021 09:39:43 GMT
- Title: On Designing Good Representation Learning Models
- Authors: Qinglin Li, Bin Li, Jonathan M Garibaldi, Guoping Qiu
- Abstract summary: The goal of representation learning is different from the ultimate objective of machine learning such as decision making.
It is difficult to establish clear and direct objectives for training representation learning models.
- Score: 22.25835082568234
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: The goal of representation learning is different from the ultimate objective
of machine learning such as decision making, it is therefore very difficult to
establish clear and direct objectives for training representation learning
models. It has been argued that a good representation should disentangle the
underlying variation factors, yet how to translate this into training
objectives remains unknown. This paper presents an attempt to establish direct
training criterions and design principles for developing good representation
learning models. We propose that a good representation learning model should be
maximally expressive, i.e., capable of distinguishing the maximum number of
input configurations. We formally define expressiveness and introduce the
maximum expressiveness (MEXS) theorem of a general learning model. We propose
to train a model by maximizing its expressiveness while at the same time
incorporating general priors such as model smoothness. We present a conscience
competitive learning algorithm which encourages the model to reach its MEXS
whilst at the same time adheres to model smoothness prior. We also introduce a
label consistent training (LCT) technique to boost model smoothness by
encouraging it to assign consistent labels to similar samples. We present
extensive experimental results to show that our method can indeed design
representation learning models capable of developing representations that are
as good as or better than state of the art. We also show that our technique is
computationally efficient, robust against different parameter settings and can
work effectively on a variety of datasets.
Related papers
- Aligning Visual Contrastive learning models via Preference Optimization [0.9438963196770565]
This paper introduces a novel method for training contrastive learning models using Preference Optimization (PO) to break down complex concepts.
Our method systematically aligns model behavior with desired preferences, enhancing performance on the targeted task.
In particular, we focus on enhancing model robustness against typographic attacks, commonly seen in contrastive models like CLIP.
We further apply our method to disentangle gender understanding and mitigate gender biases, offering a more nuanced control over these sensitive attributes.
arXiv Detail & Related papers (2024-11-12T08:14:54Z) - Has Your Pretrained Model Improved? A Multi-head Posterior Based
Approach [25.927323251675386]
We leverage the meta-features associated with each entity as a source of worldly knowledge and employ entity representations from the models.
We propose using the consistency between these representations and the meta-features as a metric for evaluating pre-trained models.
Our method's effectiveness is demonstrated across various domains, including models with relational datasets, large language models and image models.
arXiv Detail & Related papers (2024-01-02T17:08:26Z) - Sequential Modeling Enables Scalable Learning for Large Vision Models [120.91839619284431]
We introduce a novel sequential modeling approach which enables learning a Large Vision Model (LVM) without making use of any linguistic data.
We define a common format, "visual sentences", in which we can represent raw images and videos as well as annotated data sources.
arXiv Detail & Related papers (2023-12-01T18:59:57Z) - Flow Factorized Representation Learning [109.51947536586677]
We introduce a generative model which specifies a distinct set of latent probability paths that define different input transformations.
We show that our model achieves higher likelihoods on standard representation learning benchmarks while simultaneously being closer to approximately equivariant models.
arXiv Detail & Related papers (2023-09-22T20:15:37Z) - GPT4Image: Can Large Pre-trained Models Help Vision Models on Perception
Tasks? [51.22096780511165]
We present a new learning paradigm in which the knowledge extracted from large pre-trained models are utilized to help models like CNN and ViT learn enhanced representations.
We feed detailed descriptions into a pre-trained encoder to extract text embeddings with rich semantic information that encodes the content of images.
arXiv Detail & Related papers (2023-06-01T14:02:45Z) - Evaluating Representations with Readout Model Switching [19.907607374144167]
In this paper, we propose to use the Minimum Description Length (MDL) principle to devise an evaluation metric.
We design a hybrid discrete and continuous-valued model space for the readout models and employ a switching strategy to combine their predictions.
The proposed metric can be efficiently computed with an online method and we present results for pre-trained vision encoders of various architectures.
arXiv Detail & Related papers (2023-02-19T14:08:01Z) - Minimal Value-Equivalent Partial Models for Scalable and Robust Planning
in Lifelong Reinforcement Learning [56.50123642237106]
Common practice in model-based reinforcement learning is to learn models that model every aspect of the agent's environment.
We argue that such models are not particularly well-suited for performing scalable and robust planning in lifelong reinforcement learning scenarios.
We propose new kinds of models that only model the relevant aspects of the environment, which we call "minimal value-minimal partial models"
arXiv Detail & Related papers (2023-01-24T16:40:01Z) - Distilling Interpretable Models into Human-Readable Code [71.11328360614479]
Human-readability is an important and desirable standard for machine-learned model interpretability.
We propose to train interpretable models using conventional methods, and then distill them into concise, human-readable code.
We describe a piecewise-linear curve-fitting algorithm that produces high-quality results efficiently and reliably across a broad range of use cases.
arXiv Detail & Related papers (2021-01-21T01:46:36Z) - Models, Pixels, and Rewards: Evaluating Design Trade-offs in Visual
Model-Based Reinforcement Learning [109.74041512359476]
We study a number of design decisions for the predictive model in visual MBRL algorithms.
We find that a range of design decisions that are often considered crucial, such as the use of latent spaces, have little effect on task performance.
We show how this phenomenon is related to exploration and how some of the lower-scoring models on standard benchmarks will perform the same as the best-performing models when trained on the same training data.
arXiv Detail & Related papers (2020-12-08T18:03:21Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.