GPTFace: Generative Pre-training of Facial-Linguistic Transformer by Span Masking and Weakly Correlated Text-image Data

• URL: http://arxiv.org/abs/2510.18345v1
• Date: Tue, 21 Oct 2025 06:55:44 GMT
• Title: GPTFace: Generative Pre-training of Facial-Linguistic Transformer by Span Masking and Weakly Correlated Text-image Data
• Authors: Yudong Li, Hao Li, Xianxu Hou, Linlin Shen,
• Abstract summary: We present a generative pre-training model for facial knowledge learning that leverages large-scale web-built data for training.<n>Our approach is also applicable to a wide range of face editing tasks, including face attribute editing, expression manipulation, mask removal, and photo inpainting.
• Score: 53.92883885331805
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Compared to the prosperity of pre-training models in natural image understanding, the research on large-scale pre-training models for facial knowledge learning is still limited. Current approaches mainly rely on manually assembled and annotated face datasets for training, but labeling such datasets is labor-intensive and the trained models have limited scalability beyond the training data. To address these limitations, we present a generative pre-training model for facial knowledge learning that leverages large-scale web-built data for training. We use texts and images containing human faces crawled from the internet and conduct pre-training on self-supervised tasks, including masked image/language modeling (MILM) and image-text matching (ITM). During the generation stage, we further utilize the image-text matching loss to pull the generation distribution towards the control signal for controllable image/text generation. Experimental results demonstrate that our model achieves comparable performance to state-of-the-art pre-training models for various facial downstream tasks, such as attribution classification and expression recognition. Furthermore, our approach is also applicable to a wide range of face editing tasks, including face attribute editing, expression manipulation, mask removal, and photo inpainting.

Related papers

• MaskedCLIP: Bridging the Masked and CLIP Space for Semi-Supervised Medical Vision-Language Pre-training [27.35164449801058]
  State-of-the-art methods leverage either paired image-text data via vision-language pre-training or unpaired image data via self-supervised pre-training to learn foundation models.<n>We propose MaskedCLIP, a synergistic masked image modeling and contrastive language-image pre-training framework.
  arXiv  Detail & Related papers  (2025-07-23T06:15:54Z)
• Evolved Hierarchical Masking for Self-Supervised Learning [49.77271430882176]
  Existing Masked Image Modeling methods apply fixed mask patterns to guide the self-supervised training.<n>This paper introduces an evolved hierarchical masking method to pursue general visual cues modeling in self-supervised learning.
  arXiv  Detail & Related papers  (2025-04-12T09:40:14Z)
• FaceChain-FACT: Face Adapter with Decoupled Training for Identity-preserved Personalization [24.600720169589334]
  adapter-based method obtains the ability to customize and generate portraits by text-to-image training on facial data. There is often a significant performance decrease in test following ability, controllability, and diversity of generated faces compared to the base model. We propose the Face Adapter with deCoupled Training (FACT) framework, focusing on both model architecture and training strategy.
  arXiv  Detail & Related papers  (2024-10-16T07:25:24Z)
• Scaling Laws of Synthetic Images for Model Training ... for Now [54.43596959598466]
  We study the scaling laws of synthetic images generated by state of the art text-to-image models. We observe that synthetic images demonstrate a scaling trend similar to, but slightly less effective than, real images in CLIP training.
  arXiv  Detail & Related papers  (2023-12-07T18:59:59Z)
• Limitations of Face Image Generation [12.11955119100926]
  We study the efficacy and shortcomings of generative models in the context of face generation. We identify several limitations of face image generation that include faithfulness to the text prompt, demographic disparities, and distributional shifts. We present an analytical model that provides insights into how training data selection contributes to the performance of generative models.
  arXiv  Detail & Related papers  (2023-09-13T19:33:26Z)
• Toward High Quality Facial Representation Learning [58.873356953627614]
  We propose a self-supervised pre-training framework, called Mask Contrastive Face (MCF) We use feature map of a pre-trained visual backbone as a supervision item and use a partially pre-trained decoder for mask image modeling. Our model achieves 0.932 NME_diag$ for AFLW-19 face alignment and 93.96 F1 score for LaPa face parsing.
  arXiv  Detail & Related papers  (2023-09-07T09:11:49Z)
• Free-ATM: Exploring Unsupervised Learning on Diffusion-Generated Images with Free Attention Masks [64.67735676127208]
  Text-to-image diffusion models have shown great potential for benefiting image recognition. Although promising, there has been inadequate exploration dedicated to unsupervised learning on diffusion-generated images. We introduce customized solutions by fully exploiting the aforementioned free attention masks.
  arXiv  Detail & Related papers  (2023-08-13T10:07:46Z)
• General Facial Representation Learning in a Visual-Linguistic Manner [45.92447707178299]
  We introduce a framework, called FaRL, for general Facial Representation Learning in a visual-linguistic manner. We show that FaRL achieves better transfer performance compared with previous pre-trained models. Our model surpasses the state-of-the-art methods on face analysis tasks including face parsing and face alignment.
  arXiv  Detail & Related papers  (2021-12-06T15:22:05Z)
• LAFITE: Towards Language-Free Training for Text-to-Image Generation [83.2935513540494]
  We propose the first work to train text-to-image generation models without any text data. Our method leverages the well-aligned multi-modal semantic space of the powerful pre-trained CLIP model. We obtain state-of-the-art results in the standard text-to-image generation tasks.
  arXiv  Detail & Related papers  (2021-11-27T01:54:45Z)
• ImageBERT: Cross-modal Pre-training with Large-scale Weak-supervised Image-Text Data [9.3935916515127]
  We introduce a new vision-supervised pre-trained model -- ImageBERT -- for image-text joint embedding. Our model is a Transformer-based model, which takes different modalities as input and models the relationship between them.
  arXiv  Detail & Related papers  (2020-01-22T11:35:58Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.