ClickPrompt: CTR Models are Strong Prompt Generators for Adapting Language Models to CTR Prediction

• URL: http://arxiv.org/abs/2310.09234v5
• Date: Wed, 26 Jun 2024 08:59:47 GMT
• Title: ClickPrompt: CTR Models are Strong Prompt Generators for Adapting Language Models to CTR Prediction
• Authors: Jianghao Lin, Bo Chen, Hangyu Wang, Yunjia Xi, Yanru Qu, Xinyi Dai, Kangning Zhang, Ruiming Tang, Yong Yu, Weinan Zhang,
• Abstract summary: Click-through rate (CTR) prediction has become increasingly indispensable for various Internet applications. Traditional CTR models convert the multi-field categorical data into ID features via one-hot encoding, and extract the collaborative signals among features. We propose a novel model-agnostic framework (i.e., ClickPrompt) where we incorporate CTR models to generate interaction-aware soft prompts.
• Score: 45.15127775876369
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Click-through rate (CTR) prediction has become increasingly indispensable for various Internet applications. Traditional CTR models convert the multi-field categorical data into ID features via one-hot encoding, and extract the collaborative signals among features. Such a paradigm suffers from the problem of semantic information loss. Another line of research explores the potential of pretrained language models (PLMs) for CTR prediction by converting input data into textual sentences through hard prompt templates. Although semantic signals are preserved, they generally fail to capture the collaborative information (e.g., feature interactions, pure ID features), not to mention the unacceptable inference overhead brought by the huge model size. In this paper, we aim to model both the semantic knowledge and collaborative knowledge for accurate CTR estimation, and meanwhile address the inference inefficiency issue. To benefit from both worlds and close their gaps, we propose a novel model-agnostic framework (i.e., ClickPrompt), where we incorporate CTR models to generate interaction-aware soft prompts for PLMs. We design a prompt-augmented masked language modeling (PA-MLM) pretraining task, where PLM has to recover the masked tokens based on the language context, as well as the soft prompts generated by CTR model. The collaborative and semantic knowledge from ID and textual features would be explicitly aligned and interacted via the prompt interface. Then, we can either tune the CTR model with PLM for superior performance, or solely tune the CTR model without PLM for inference efficiency. Experiments on four real-world datasets validate the effectiveness of ClickPrompt compared with existing baselines.

Related papers

• ReasoningLM: Enabling Structural Subgraph Reasoning in Pre-trained Language Models for Question Answering over Knowledge Graph [142.42275983201978]
  We propose a subgraph-aware self-attention mechanism to imitate the GNN for performing structured reasoning. We also adopt an adaptation tuning strategy to adapt the model parameters with 20,000 subgraphs with synthesized questions. Experiments show that ReasoningLM surpasses state-of-the-art models by a large margin, even with fewer updated parameters and less training data.
  arXiv  Detail & Related papers  (2023-12-30T07:18:54Z)
• FLIP: Fine-grained Alignment between ID-based Models and Pretrained Language Models for CTR Prediction [49.510163437116645]
  Click-through rate (CTR) prediction plays as a core function module in personalized online services. Traditional ID-based models for CTR prediction take as inputs the one-hot encoded ID features of tabular modality. Pretrained Language Models(PLMs) has given rise to another paradigm, which takes as inputs the sentences of textual modality. We propose to conduct Fine-grained feature-level ALignment between ID-based Models and Pretrained Language Models(FLIP) for CTR prediction.
  arXiv  Detail & Related papers  (2023-10-30T11:25:03Z)
• Structural Self-Supervised Objectives for Transformers [3.018656336329545]
  This thesis focuses on improving the pre-training of natural language models using unsupervised raw data. In the first part, we introduce three alternative pre-training objectives to BERT's Masked Language Modeling (MLM) In the second part, we proposes self-supervised pre-training tasks that align structurally with downstream applications.
  arXiv  Detail & Related papers  (2023-09-15T09:30:45Z)
• BERT4CTR: An Efficient Framework to Combine Pre-trained Language Model with Non-textual Features for CTR Prediction [12.850529317775198]
  We propose a novel framework BERT4CTR, with the Uni-Attention mechanism that can benefit from the interactions between non-textual and textual features. BERT4CTR can outperform significantly the state-of-the-art frameworks to handle multi-modal inputs and be applicable to Click-Through-Rate (CTR) prediction.
  arXiv  Detail & Related papers  (2023-08-17T08:25:54Z)
• MAP: A Model-agnostic Pretraining Framework for Click-through Rate Prediction [39.48740397029264]
  We propose a Model-agnostic pretraining (MAP) framework that applies feature corruption and recovery on multi-field categorical data. We derive two practical algorithms: masked feature prediction (RFD) and replaced feature detection (RFD)
  arXiv  Detail & Related papers  (2023-08-03T12:55:55Z)
• Large Language Models with Controllable Working Memory [64.71038763708161]
  Large language models (LLMs) have led to a series of breakthroughs in natural language processing (NLP) What further sets these models apart is the massive amounts of world knowledge they internalize during pretraining. How the model's world knowledge interacts with the factual information presented in the context remains under explored.
  arXiv  Detail & Related papers  (2022-11-09T18:58:29Z)
• Leveraging Advantages of Interactive and Non-Interactive Models for Vector-Based Cross-Lingual Information Retrieval [12.514666775853598]
  We propose a novel framework to leverage the advantages of interactive and non-interactive models. We introduce semi-interactive mechanism, which builds our model upon non-interactive architecture but encodes each document together with its associated multilingual queries. Our methods significantly boost the retrieval accuracy while maintaining the computational efficiency.
  arXiv  Detail & Related papers  (2021-11-03T03:03:19Z)
• PTR: Prompt Tuning with Rules for Text Classification [64.1655047016891]
  Fine-tuned pre-trained language models (PLMs) have achieved awesome performance on almost all NLP tasks. We propose prompt tuning with rules (PTR) for many-class text classification. PTR is able to encode prior knowledge of each class into prompt tuning.
  arXiv  Detail & Related papers  (2021-05-24T13:24:02Z)
• Conversational Question Reformulation via Sequence-to-Sequence Architectures and Pretrained Language Models [56.268862325167575]
  This paper presents an empirical study of conversational question reformulation (CQR) with sequence-to-sequence architectures and pretrained language models (PLMs) We leverage PLMs to address the strong token-to-token independence assumption made in the common objective, maximum likelihood estimation, for the CQR task. We evaluate fine-tuned PLMs on the recently-introduced CANARD dataset as an in-domain task and validate the models using data from the TREC 2019 CAsT Track as an out-domain task.
  arXiv  Detail & Related papers  (2020-04-04T11:07:54Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.