PatentGPT: A Large Language Model for Intellectual Property

• URL: http://arxiv.org/abs/2404.18255v5
• Date: Wed, 5 Jun 2024 03:02:48 GMT
• Title: PatentGPT: A Large Language Model for Intellectual Property
• Authors: Zilong Bai, Ruiji Zhang, Linqing Chen, Qijun Cai, Yuan Zhong, Cong Wang, Yan Fang, Jie Fang, Jing Sun, Weikuan Wang, Lizhi Zhou, Haoran Hua, Tian Qiu, Chaochao Wang, Cheng Sun, Jianping Lu, Yixin Wang, Yubin Xia, Meng Hu, Haowen Liu, Peng Xu, Licong Xu, Fu Bian, Xiaolong Gu, Lisha Zhang, Weilei Wang, Changyang Tu,
• Abstract summary: Large language models (LLMs) have attracted significant attention due to their exceptional performance across a multitude of natural language process tasks. However, the application of large language models in the Intellectual Property (IP) domain is challenging due to the strong need for specialized knowledge. We present for the first time a low-cost, standardized procedure for training IP-oriented LLMs, meeting the unique requirements of the IP domain.
• Score: 26.31216865513109
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: In recent years, large language models(LLMs) have attracted significant attention due to their exceptional performance across a multitude of natural language process tasks, and have been widely applied in various fields. However, the application of large language models in the Intellectual Property (IP) domain is challenging due to the strong need for specialized knowledge, privacy protection, processing of extremely long text in this field. In this technical report, we present for the first time a low-cost, standardized procedure for training IP-oriented LLMs, meeting the unique requirements of the IP domain. Using this standard process, we have trained the PatentGPT series models based on open-source pretrained models. By evaluating them on the open-source IP-oriented benchmark MOZIP, our domain-specific LLMs outperforms GPT-4, indicating the effectiveness of the proposed training procedure and the expertise of the PatentGPT models in the IP domain. Remarkably, our model surpassed GPT-4 on the 2019 China Patent Agent Qualification Examination, scoring 65 and matching human expert levels. Additionally, the PatentGPT model, which utilizes the SMoE architecture, achieves performance comparable to that of GPT-4 in the IP domain and demonstrates a better cost-performance ratio on long-text tasks, potentially serving as an alternative to GPT-4 within the IP domain.

Related papers

• Patent Language Model Pretraining with ModernBERT [0.0]
  We pretrain 3 domain-specific masked language models for patents using the ModernBERT architecture and a curated corpus of over 60 million patent records.<n>Our approach incorporates architectural optimizations, including FlashAttention, rotary embeddings, and GLU feed-forward layers.<n>Our model, ModernBERT-base-PT, consistently outperforms the general-purpose ModernBERT baseline on three out of four datasets and competitive performance with a baseline PatentBERT.
  arXiv  Detail & Related papers  (2025-09-18T13:04:30Z)
• MCP-Universe: Benchmarking Large Language Models with Real-World Model Context Protocol Servers [86.00932417210477]
  We introduce MCP-Universe, the first comprehensive benchmark specifically designed to evaluate LLMs in realistic and hard tasks through interaction with real-world MCP servers.<n>Our benchmark encompasses 6 core domains spanning 11 different MCP servers: Location Navigation, Repository Management, Financial Analysis, 3D Design, Browser Automation, and Web Searching.<n>We find that even SOTA models such as GPT-5 (43.72%), Grok-4 (33.33%) and Claude-4.0-Sonnet (29.44%) exhibit significant performance limitations.
  arXiv  Detail & Related papers  (2025-08-20T13:28:58Z)
• Evaluating Named Entity Recognition Models for Russian Cultural News Texts: From BERT to LLM [0.0]
  The study utilizes the unique SPbLitGuide dataset, a collection of event announcements from Saint Petersburg spanning 1999 to 2019.<n>A comparative evaluation of diverse NER models is presented, encompassing established transformer-based architectures.<n>The research contributes to a deeper understanding of current NER model capabilities and limitations when applied to morphologically rich languages like Russian.
  arXiv  Detail & Related papers  (2025-06-03T08:11:16Z)
• IPBench: Benchmarking the Knowledge of Large Language Models in Intellectual Property [30.847397185230538]
  We introduce the first comprehensive IP task taxonomy and a large, diverse bilingual benchmark, IPBench. This benchmark is designed to evaluate LLMs in real-world intellectual property applications, encompassing both understanding and generation. We benchmark 16 LLMs, ranging from general-purpose to domain-specific models, and find that even the best-performing model achieves only 75.8% accuracy.
  arXiv  Detail & Related papers  (2025-04-22T02:00:41Z)
• Intellectual Property Protection for Deep Learning Model and Dataset Intelligence [21.757997058357]
  This work systematically summarizes the general and scheme-specific performance evaluation metrics. From proactive IP infringement prevention and reactive IP ownership verification perspectives, it comprehensively investigates and analyzes the existing IPP methods. Finally, we outline prospects for promising future directions that may act as a guide for innovative research.
  arXiv  Detail & Related papers  (2024-11-07T09:02:41Z)
• Prompting Encoder Models for Zero-Shot Classification: A Cross-Domain Study in Italian [75.94354349994576]
  This paper explores the feasibility of employing smaller, domain-specific encoder LMs alongside prompting techniques to enhance performance in specialized contexts. Our study concentrates on the Italian bureaucratic and legal language, experimenting with both general-purpose and further pre-trained encoder-only models. The results indicate that while further pre-trained models may show diminished robustness in general knowledge, they exhibit superior adaptability for domain-specific tasks, even in a zero-shot setting.
  arXiv  Detail & Related papers  (2024-07-30T08:50:16Z)
• IPEval: A Bilingual Intellectual Property Agency Consultation Evaluation Benchmark for Large Language Models [13.103862590594705]
  IPEval comprises 2657 multiple-choice questions across four major dimensions: creation, application, protection, and management of IP. Evaluation methods include zero-shot, 5-few-shot, and Chain of Thought (CoT) for seven LLM types, predominantly in English or Chinese. Results show superior English performance by models like GPT series and Qwen series, while Chinese-centric LLMs excel in Chinese tests.
  arXiv  Detail & Related papers  (2024-06-18T08:18:18Z)
• MoZIP: A Multilingual Benchmark to Evaluate Large Language Models in Intellectual Property [51.43412400869531]
  Large language models (LLMs) have demonstrated impressive performance in various natural language processing (NLP) tasks. We contribute a new benchmark, the first Multilingual-oriented quiZ on Intellectual Property (MoZIP), for the evaluation of LLMs in the IP domain. We also develop a new IP-oriented multilingual large language model (called MoZi), which is a BLOOMZ-based model that has been supervised fine-tuned with multilingual IP-related text data.
  arXiv  Detail & Related papers  (2024-02-26T08:27:50Z)
• OMPGPT: A Generative Pre-trained Transformer Model for OpenMP [6.917568654215119]
  OMPGPT is a novel domain-specific model meticulously designed to harness the inherent strengths of language models for OpenMP pragma generation. We leverage prompt engineering techniques from the NLP domain to create Chain-of-OMP, an innovative strategy designed to enhance OMPGPT's effectiveness.
  arXiv  Detail & Related papers  (2024-01-28T06:06:59Z)
• HuatuoGPT-II, One-stage Training for Medical Adaption of LLMs [61.41790586411816]
  HuatuoGPT-II has shown state-of-the-art performance in Chinese medicine domain on a number of benchmarks. It even outperforms proprietary models like ChatGPT and GPT-4 in some aspects, especially in Traditional Chinese Medicine.
  arXiv  Detail & Related papers  (2023-11-16T10:56:24Z)
• ChipNeMo: Domain-Adapted LLMs for Chip Design [19.43613652552849]
  ChipNeMo aims to explore the applications of large language models (LLMs) for industrial chip design. We adopt the following domain adaptation techniques: domain-adaptive tokenization, domain-adaptive continued pretraining, model alignment with domain-specific instructions, and domain-adapted retrieval models.
  arXiv  Detail & Related papers  (2023-10-31T22:35:58Z)
• The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision) [121.42924593374127]
  We analyze the latest model, GPT-4V, to deepen the understanding of LMMs. GPT-4V's unprecedented ability in processing arbitrarily interleaved multimodal inputs makes it a powerful multimodal generalist system. GPT-4V's unique capability of understanding visual markers drawn on input images can give rise to new human-computer interaction methods.
  arXiv  Detail & Related papers  (2023-09-29T17:34:51Z)
• Exploring the Trade-Offs: Unified Large Language Models vs Local Fine-Tuned Models for Highly-Specific Radiology NLI Task [49.50140712943701]
  We evaluate the performance of ChatGPT/GPT-4 on a radiology NLI task and compare it to other models fine-tuned specifically on task-related data samples. We also conduct a comprehensive investigation on ChatGPT/GPT-4's reasoning ability by introducing varying levels of inference difficulty.
  arXiv  Detail & Related papers  (2023-04-18T17:21:48Z)
• GPT-4 Technical Report [116.90398195245983]
  GPT-4 is a large-scale, multimodal model which can accept image and text inputs and produce text outputs. It exhibits human-level performance on various professional and academic benchmarks, including passing a simulated bar exam with a score around the top 10% of test takers.
  arXiv  Detail & Related papers  (2023-03-15T17:15:04Z)
• Linguistically Informed Masking for Representation Learning in the Patent Domain [7.911344873839031]
  We propose the empirically motivated Linguistically Informed Masking (LIM) method to focus domain-adaptative pre-training on the linguistic patterns of patents. We quantify the relevant differences between patent, scientific and general-purpose language. We demonstrate the impact of balancing the learning from different information sources during domain adaptation for the patent domain.
  arXiv  Detail & Related papers  (2021-06-10T14:20:57Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.