Datasets for machine learning and for assessing the intelligence level of automatic patent search systems

• URL: http://arxiv.org/abs/2512.18384v1
• Date: Sat, 20 Dec 2025 14:51:57 GMT
• Title: Datasets for machine learning and for assessing the intelligence level of automatic patent search systems
• Abstract summary: Key to success in automating prior art search in patent research using artificial intelligence lies in developing large datasets for machine learning.<n>This work is dedicated to providing a comprehensive solution to the problem of creating infrastructure for research in this field, including datasets and tools for calculating search quality criteria.
• Score: 39.146761527401424
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The key to success in automating prior art search in patent research using artificial intelligence lies in developing large datasets for machine learning and ensuring their availability. This work is dedicated to providing a comprehensive solution to the problem of creating infrastructure for research in this field, including datasets and tools for calculating search quality criteria. The paper discusses the concept of semantic clusters of patent documents that determine the state of the art in a given subject, as proposed by the authors. A definition of such semantic clusters is also provided. Prior art search is presented as the task of identifying elements within a semantic cluster of patent documents in the subject area specified by the document under consideration. A generator of user-configurable datasets for machine learning, based on collections of U.S. and Russian patent documents, is described. The dataset generator creates a database of links to documents in semantic clusters. Then, based on user-defined parameters, it forms a dataset of semantic clusters in JSON format for machine learning. To evaluate machine learning outcomes, it is proposed to calculate search quality scores that account for semantic clusters of the documents being searched. To automate the evaluation process, the paper describes a utility developed by the authors for assessing the quality of prior art document search.

Related papers

• CoverageBench: Evaluating Information Coverage across Tasks and Domains [57.33043453870487]
  Information coverage is a central aspect for retrieval, especially when the retrieval system is integrated with generative models.<n>Test collections developed around the notion of diversity ranking in web search incorporate multiple aspects that support a concept of coverage in the web domain.
  arXiv  Detail & Related papers  (2026-03-20T15:20:44Z)
• Performance Analysis of Supervised Machine Learning Algorithms for Text Classification [0.0]
  This paper illustrates the text classification process on different datasets using some standard supervised machine learning techniques.<n> Labeled text documents are used to classify the text in supervised classifications.<n> Experimental analysis on real data reveals which model works well in terms of classification accuracy.
  arXiv  Detail & Related papers  (2025-08-31T20:30:52Z)
• ProxAnn: Use-Oriented Evaluations of Topic Models and Document Clustering [52.19512723549318]
  We design a scalable human evaluation protocol that reflects practitioners' real-world usage of models.<n>We use this protocol to collect extensive crowdworker annotations of outputs from a diverse set of topic models.<n>We then use these annotations to validate automated proxies, finding that the best LLM proxies are statistically indistinguishable from a human annotator.
  arXiv  Detail & Related papers  (2025-07-01T15:00:55Z)
• Automated Collection of Evaluation Dataset for Semantic Search in Low-Resource Domain Language [4.5224851085910585]
  Domain-specific languages that use a lot of specific terminology often fall into the category of low-resource languages.<n>This study addresses the challenge of automated collecting test datasets to evaluate semantic search in low-resource domain-specific German language.
  arXiv  Detail & Related papers  (2024-12-13T09:47:26Z)
• Benchmarking pre-trained text embedding models in aligning built asset information [0.0]
  This study presents a comparative benchmark of state-of-the-art text embedding models to evaluate their effectiveness in aligning built asset information with domain-specific technical concepts. The results of our benchmarking across six proposed datasets, covering three tasks of clustering, retrieval, and reranking, highlight the need for future research on domain adaptation techniques.
  arXiv  Detail & Related papers  (2024-11-18T20:54:17Z)
• Object Recognition from Scientific Document based on Compartment Refinement Framework [2.699900017799093]
  It has become increasingly important to extract valuable information from vast resources efficiently. Current data extraction methods for scientific documents typically use rule-based (RB) or machine learning (ML) approaches. We propose a new document layout analysis framework called CTBR(Compartment & Text Blocks Refinement)
  arXiv  Detail & Related papers  (2023-12-14T15:36:49Z)
• Evaluating Generative Ad Hoc Information Retrieval [58.800799175084286]
  generative retrieval systems often directly return a grounded generated text as a response to a query. Quantifying the utility of the textual responses is essential for appropriately evaluating such generative ad hoc retrieval.
  arXiv  Detail & Related papers  (2023-11-08T14:05:00Z)
• Layout-Aware Information Extraction for Document-Grounded Dialogue: Dataset, Method and Demonstration [75.47708732473586]
  We propose a layout-aware document-level Information Extraction dataset, LIE, to facilitate the study of extracting both structural and semantic knowledge from visually rich documents. LIE contains 62k annotations of three extraction tasks from 4,061 pages in product and official documents. Empirical results show that layout is critical for VRD-based extraction, and system demonstration also verifies that the extracted knowledge can help locate the answers that users care about.
  arXiv  Detail & Related papers  (2022-07-14T07:59:45Z)
• Questions Are All You Need to Train a Dense Passage Retriever [123.13872383489172]
  ART is a new corpus-level autoencoding approach for training dense retrieval models that does not require any labeled training data. It uses a new document-retrieval autoencoding scheme, where (1) an input question is used to retrieve a set of evidence documents, and (2) the documents are then used to compute the probability of reconstructing the original question.
  arXiv  Detail & Related papers  (2022-06-21T18:16:31Z)
• CSFCube -- A Test Collection of Computer Science Research Articles for Faceted Query by Example [43.01717754418893]
  We introduce the task of faceted Query by Example. Users can also specify a finer grained aspect in addition to the input query document. We envision models which are able to retrieve scientific papers analogous to a query scientific paper.
  arXiv  Detail & Related papers  (2021-03-24T01:02:12Z)
• Active Learning from Crowd in Document Screening [76.9545252341746]
  We focus on building a set of machine learning classifiers that evaluate documents, and then screen them efficiently. We propose a multi-label active learning screening specific sampling technique -- objective-aware sampling. We demonstrate that objective-aware sampling significantly outperforms the state of the art active learning sampling strategies.
  arXiv  Detail & Related papers  (2020-11-11T16:17:28Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.