The Effectiveness of Bidirectional Generative Patent Language Models

• URL: http://arxiv.org/abs/2211.09690v1
• Date: Sun, 4 Sep 2022 03:12:27 GMT
• Title: The Effectiveness of Bidirectional Generative Patent Language Models
• Authors: Jieh-Sheng Lee
• Abstract summary: A simplified design of the autocomplete function is proposed to increase effectiveness by more than 10%. With the new design, the effectiveness of autocomplete can reach more than 60%, which means that more than 60% of keystrokes can be saved by autocomplete. A key finding is that the autocomplete effectiveness of a model for the same text remains similar no matter where the calculation starts.
• Score: 1.8275108630751844
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Generative patent language models can assist humans to write patent text more effectively. The question is how to measure effectiveness from a human-centric perspective and how to improve effectiveness. In this manuscript, a simplified design of the autocomplete function is proposed to increase effectiveness by more than 10%. With the new design, the effectiveness of autocomplete can reach more than 60%, which means that more than 60% of keystrokes can be saved by autocomplete. Since writing patent text does not necessarily start from the beginning to the end, a question is whether the generative model can assist a user no matter where to start writing. To answer the question, the generative models in this manuscript are pre-trained with training data in both directions. The generative models become bidirectional. Since text generation is bidirectional, the calculation of autocomplete effectiveness can be bidirectional and starts from anywhere in the text. After thorough experiments, a key finding is that the autocomplete effectiveness of a model for the same text remains similar no matter where the calculation starts. The finding indicates that such bidirectional models can assist a user at a similar level, no matter where the user starts to write.

Related papers

• Sequential Decision-Making for Inline Text Autocomplete [14.83046358936405]
  We study the problem of improving inline autocomplete suggestions in text entry systems. We use reinforcement learning to learn suggestion policies through repeated interactions with a target user.
  arXiv  Detail & Related papers  (2024-03-21T22:33:16Z)
• Fine-tuning Language Models for Factuality [96.5203774943198]
  Large pre-trained language models (LLMs) have led to their widespread use, sometimes even as a replacement for traditional search engines. Yet language models are prone to making convincing but factually inaccurate claims, often referred to as 'hallucinations' In this work, we fine-tune language models to be more factual, without human labeling.
  arXiv  Detail & Related papers  (2023-11-14T18:59:15Z)
• Smaller Language Models are Better Black-box Machine-Generated Text Detectors [56.36291277897995]
  Small and partially-trained models are better universal text detectors. We find that whether the detector and generator were trained on the same data is not critically important to the detection success. For instance, the OPT-125M model has an AUC of 0.81 in detecting ChatGPT generations, whereas a larger model from the GPT family, GPTJ-6B, has AUC of 0.45.
  arXiv  Detail & Related papers  (2023-05-17T00:09:08Z)
• Discovering Latent Knowledge in Language Models Without Supervision [72.95136739040676]
  Existing techniques for training language models can be misaligned with the truth. We propose directly finding latent knowledge inside the internal activations of a language model in a purely unsupervised way. We show that despite using no supervision and no model outputs, our method can recover diverse knowledge represented in large language models.
  arXiv  Detail & Related papers  (2022-12-07T18:17:56Z)
• Small Character Models Match Large Word Models for Autocomplete Under Memory Constraints [32.79377465262468]
  We study the more challenging open-domain setting consisting of low frequency user prompt patterns. Character-based representation is effective in reducing the overall model size. We show that a 20M parameter character model performs similar to an 80M parameter word model in the vanilla setting.
  arXiv  Detail & Related papers  (2022-10-06T23:29:59Z)
• PART: Pre-trained Authorship Representation Transformer [64.78260098263489]
  Authors writing documents imprint identifying information within their texts: vocabulary, registry, punctuation, misspellings, or even emoji usage. Previous works use hand-crafted features or classification tasks to train their authorship models, leading to poor performance on out-of-domain authors. We propose a contrastively trained model fit to learn textbfauthorship embeddings instead of semantics.
  arXiv  Detail & Related papers  (2022-09-30T11:08:39Z)
• Evaluating Generative Patent Language Models [1.8275108630751844]
  This manuscript aims to build generative language models in the patent domain. The perspective is to measure the ratio of keystrokes that can be saved by autocompletion. The largest model built in this manuscript is 6B, which is state-of-the-art in the patent domain.
  arXiv  Detail & Related papers  (2022-06-23T08:58:05Z)
• A Model-Agnostic Data Manipulation Method for Persona-based Dialogue Generation [107.82729587882397]
  It is expensive to scale up current persona-based dialogue datasets. Each data sample in this task is more complex to learn with than conventional dialogue data. We propose a data manipulation method, which is model-agnostic to be packed with any persona-based dialogue generation model.
  arXiv  Detail & Related papers  (2022-04-21T03:49:54Z)
• Improving Performance of Automated Essay Scoring by using back-translation essays and adjusted scores [0.0]
  We propose a method to increase the number of essay-score pairs using back-translation and score adjustment. We evaluate the effectiveness of the augmented data using models from prior work. The performance of the models was improved by using augmented data to train the models.
  arXiv  Detail & Related papers  (2022-03-01T11:05:43Z)
• Sketch and Refine: Towards Faithful and Informative Table-to-Text Generation [58.320248632121476]
  We propose a novel two-stage method that combines both Autoregressive and Non-Autoregressive generations (SANA) Our approach includes: (1) skeleton generation with an autoregressive pointer network to select key tokens from the source table; (2) edit-based non-autoregressive generation model to produce texts via iterative insertion and deletion operations. By integrating hard constraints from the skeleton, the non-autoregressive model improves the generation's coverage over the source table and thus enhances its faithfulness.
  arXiv  Detail & Related papers  (2021-05-31T08:18:13Z)
• AutoMeTS: The Autocomplete for Medical Text Simplification [9.18959130745234]
  We introduce a new parallel medical data set consisting of aligned English Wikipedia with Simple English Wikipedia sentences. We show how the additional context of the sentence to be simplified can be incorporated to achieve better results. We also introduce an ensemble model that combines the four PNLMs and outperforms the best individual model by 2.1%.
  arXiv  Detail & Related papers  (2020-10-20T19:20:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.