PANORAMA: A Dataset and Benchmarks Capturing Decision Trails and Rationales in Patent Examination

• URL: http://arxiv.org/abs/2510.24774v1
• Date: Sat, 25 Oct 2025 03:24:13 GMT
• Title: PANORAMA: A Dataset and Benchmarks Capturing Decision Trails and Rationales in Patent Examination
• Authors: Hyunseung Lim, Sooyohn Nam, Sungmin Na, Ji Yong Cho, June Yong Yang, Hyungyu Shin, Yoonjoo Lee, Juho Kim, Moontae Lee, Hwajung Hong,
• Abstract summary: We build PANORAMA, a dataset of 8,143 U.S. patent examination records.<n>We decompose the trails into sequential benchmarks that emulate patent professionals' patent review processes.<n>We argue that advancing NLP, including LLMs, in the patent domain requires a deeper understanding of real-world patent examination.
• Score: 44.74519851862391
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Patent examination remains an ongoing challenge in the NLP literature even after the advent of large language models (LLMs), as it requires an extensive yet nuanced human judgment on whether a submitted claim meets the statutory standards of novelty and non-obviousness against previously granted claims -- prior art -- in expert domains. Previous NLP studies have approached this challenge as a prediction task (e.g., forecasting grant outcomes) with high-level proxies such as similarity metrics or classifiers trained on historical labels. However, this approach often overlooks the step-by-step evaluations that examiners must make with profound information, including rationales for the decisions provided in office actions documents, which also makes it harder to measure the current state of techniques in patent review processes. To fill this gap, we construct PANORAMA, a dataset of 8,143 U.S. patent examination records that preserves the full decision trails, including original applications, all cited references, Non-Final Rejections, and Notices of Allowance. Also, PANORAMA decomposes the trails into sequential benchmarks that emulate patent professionals' patent review processes and allow researchers to examine large language models' capabilities at each step of them. Our findings indicate that, although LLMs are relatively effective at retrieving relevant prior art and pinpointing the pertinent paragraphs, they struggle to assess the novelty and non-obviousness of patent claims. We discuss these results and argue that advancing NLP, including LLMs, in the patent domain requires a deeper understanding of real-world patent examination. Our dataset is openly available at https://huggingface.co/datasets/LG-AI-Research/PANORAMA.

Related papers

• SoK: Large Language Model Copyright Auditing via Fingerprinting [69.14570598973195]
  We introduce a unified framework and formal taxonomy that categorizes existing methods into white-box and black-box approaches.<n>We propose LeaFBench, the first systematic benchmark for evaluating LLM fingerprinting under realistic deployment scenarios.
  arXiv  Detail & Related papers  (2025-08-27T12:56:57Z)
• ClaimGen-CN: A Large-scale Chinese Dataset for Legal Claim Generation [56.79698529022327]
  Legal claims refer to the plaintiff's demands in a case and are essential to guiding judicial reasoning and case resolution.<n>This paper explores the problem of legal claim generation based on the given case's facts.<n>We construct ClaimGen-CN, the first dataset for Chinese legal claim generation task.
  arXiv  Detail & Related papers  (2025-08-24T07:19:25Z)
• PEDANTIC: A Dataset for the Automatic Examination of Definiteness in Patent Claims [13.242188189150987]
  PEDANTIC is a dataset of 14k US patent claims annotated with reasons for indefiniteness.<n>A human validation study confirms the pipeline's accuracy in generating high-quality annotations.<n> PEDANTIC provides a valuable resource for patent AI researchers, enabling the development of advanced examination models.
  arXiv  Detail & Related papers  (2025-05-27T15:34:39Z)
• Towards Better Evaluation for Generated Patent Claims [0.0]
  We introduce Patent-CE, the first comprehensive benchmark for evaluating patent claims.<n>We also propose PatClaimEval, a novel multi-dimensional evaluation method specifically designed for patent claims.<n>This research provides the groundwork for more accurate evaluations of automated patent claim generation systems.
  arXiv  Detail & Related papers  (2025-05-16T10:27:16Z)
• Can AI Examine Novelty of Patents?: Novelty Evaluation Based on the Correspondence between Patent Claim and Prior Art [5.655276956391884]
  This paper introduces a novel challenge by evaluating the ability of large language models (LLMs) to assess patent novelty.<n>We present the first dataset specifically designed for novelty evaluation, derived from real patent examination cases.<n>Our study reveals that while classification models struggle to effectively assess novelty, generative models make predictions with a reasonable level of accuracy.
  arXiv  Detail & Related papers  (2025-02-10T10:09:29Z)
• Patent-CR: A Dataset for Patent Claim Revision [0.0]
  This paper presents Patent-CR, the first dataset created for the patent claim revision task in English.<n>It includes both initial patent applications rejected by patent examiners and the final granted versions.
  arXiv  Detail & Related papers  (2024-12-03T16:43:42Z)
• PatentEdits: Framing Patent Novelty as Textual Entailment [62.8514393375952]
  We introduce the PatentEdits dataset, which contains 105K examples of successful revisions. We design algorithms to label edits sentence by sentence, then establish how well these edits can be predicted with large language models. We demonstrate that evaluating textual entailment between cited references and draft sentences is especially effective in predicting which inventive claims remained unchanged or are novel in relation to prior art.
  arXiv  Detail & Related papers  (2024-11-20T17:23:40Z)
• JudgeRank: Leveraging Large Language Models for Reasoning-Intensive Reranking [81.88787401178378]
  We introduce JudgeRank, a novel agentic reranker that emulates human cognitive processes when assessing document relevance. We evaluate JudgeRank on the reasoning-intensive BRIGHT benchmark, demonstrating substantial performance improvements over first-stage retrieval methods. In addition, JudgeRank performs on par with fine-tuned state-of-the-art rerankers on the popular BEIR benchmark, validating its zero-shot generalization capability.
  arXiv  Detail & Related papers  (2024-10-31T18:43:12Z)
• Structural Representation Learning and Disentanglement for Evidential Chinese Patent Approval Prediction [19.287231890434718]
  This paper presents the pioneering effort on this task using a retrieval-based classification approach. We propose a novel framework called DiSPat, which focuses on structural representation learning and disentanglement. Our framework surpasses state-of-the-art baselines on patent approval prediction, while also exhibiting enhanced evidentiality.
  arXiv  Detail & Related papers  (2024-08-23T05:44:16Z)
• Goodhart's Law Applies to NLP's Explanation Benchmarks [57.26445915212884]
  We critically examine two sets of metrics: the ERASER metrics (comprehensiveness and sufficiency) and the EVAL-X metrics. We show that we can inflate a model's comprehensiveness and sufficiency scores dramatically without altering its predictions or explanations on in-distribution test inputs. Our results raise doubts about the ability of current metrics to guide explainability research, underscoring the need for a broader reassessment of what precisely these metrics are intended to capture.
  arXiv  Detail & Related papers  (2023-08-28T03:03:03Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.