Cross-Domain Evaluation of a Deep Learning-Based Type Inference System

• URL: http://arxiv.org/abs/2208.09189v4
• Date: Fri, 28 Jul 2023 08:27:57 GMT
• Title: Cross-Domain Evaluation of a Deep Learning-Based Type Inference System
• Authors: Bernd Gruner, Tim Sonnekalb, Thomas S. Heinze, Clemens-Alexander Brust
• Abstract summary: We investigate Type4Py as a representative of state-of-the-art deep learning-based type inference systems. We address the following problems: class imbalances, out-of-vocabulary words, dataset shifts, and unknown classes. Our dataset enables the evaluation of type inference systems in different domains of software projects.
• Score: 0.44098366957385177
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Optional type annotations allow for enriching dynamic programming languages with static typing features like better Integrated Development Environment (IDE) support, more precise program analysis, and early detection and prevention of type-related runtime errors. Machine learning-based type inference promises interesting results for automating this task. However, the practical usage of such systems depends on their ability to generalize across different domains, as they are often applied outside their training domain. In this work, we investigate Type4Py as a representative of state-of-the-art deep learning-based type inference systems, by conducting extensive cross-domain experiments. Thereby, we address the following problems: class imbalances, out-of-vocabulary words, dataset shifts, and unknown classes. To perform such experiments, we use the datasets ManyTypes4Py and CrossDomainTypes4Py. The latter we introduce in this paper. Our dataset enables the evaluation of type inference systems in different domains of software projects and has over 1,000,000 type annotations mined on the platforms GitHub and Libraries. It consists of data from the two domains web development and scientific calculation. Through our experiments, we detect that the shifts in the dataset and the long-tailed distribution with many rare and unknown data types decrease the performance of the deep learning-based type inference system drastically. In this context, we test unsupervised domain adaptation methods and fine-tuning to overcome these issues. Moreover, we investigate the impact of out-of-vocabulary words.

Related papers

• Seed-Guided Fine-Grained Entity Typing in Science and Engineering Domains [51.02035914828596]
  We study the task of seed-guided fine-grained entity typing in science and engineering domains. We propose SEType which first enriches the weak supervision by finding more entities for each seen type from an unlabeled corpus. It then matches the enriched entities to unlabeled text to get pseudo-labeled samples and trains a textual entailment model that can make inferences for both seen and unseen types.
  arXiv  Detail & Related papers  (2024-01-23T22:36:03Z)
• TIPICAL -- Type Inference for Python In Critical Accuracy Level [1.1666234644810896]
  TIPICAL is a method that combines deep similarity learning with novelty detection. We show that our method can better predict data types in high confidence by successfully filtering out unknown and inaccurate predicted data types.
  arXiv  Detail & Related papers  (2023-08-04T19:16:23Z)
• Meta-learning Pathologies from Radiology Reports using Variance Aware Prototypical Networks [3.464871689508835]
  We propose a simple extension of the Prototypical Networks for few-shot text classification. Our main idea is to replace the class prototypes by Gaussians and introduce a regularization term that encourages the examples to be clustered near the appropriate class centroids.
  arXiv  Detail & Related papers  (2022-10-22T05:22:29Z)
• Panning for gold: Lessons learned from the platform-agnostic automated detection of political content in textual data [48.7576911714538]
  We discuss how these techniques can be used to detect political content across different platforms. We compare the performance of three groups of detection techniques relying on dictionaries, supervised machine learning, or neural networks. Our results show the limited impact of preprocessing on model performance, with the best results for less noisy data being achieved by neural network- and machine-learning-based models.
  arXiv  Detail & Related papers  (2022-07-01T15:23:23Z)
• Cross-Domain Generalization and Knowledge Transfer in Transformers Trained on Legal Data [0.0]
  We analyze the ability of pre-trained language models to transfer knowledge among datasets annotated with different type systems. Prediction of the rhetorical role a sentence plays in a case decision is an important and often studied task in AI & Law.
  arXiv  Detail & Related papers  (2021-12-15T04:23:14Z)
• Human-in-the-Loop Disinformation Detection: Stance, Sentiment, or Something Else? [93.91375268580806]
  Both politics and pandemics have recently provided ample motivation for the development of machine learning-enabled disinformation (a.k.a. fake news) detection algorithms. Existing literature has focused primarily on the fully-automated case, but the resulting techniques cannot reliably detect disinformation on the varied topics, sources, and time scales required for military applications. By leveraging an already-available analyst as a human-in-the-loop, canonical machine learning techniques of sentiment analysis, aspect-based sentiment analysis, and stance detection become plausible methods to use for a partially-automated disinformation detection system.
  arXiv  Detail & Related papers  (2021-11-09T13:30:34Z)
• A Brief Review of Domain Adaptation [1.2043574473965317]
  This paper focuses on unsupervised domain adaptation, where the labels are only available in the source domain. It presents some successful shallow and deep domain adaptation approaches that aim to deal with domain adaptation problems.
  arXiv  Detail & Related papers  (2020-10-07T07:05:32Z)
• A Review of Single-Source Deep Unsupervised Visual Domain Adaptation [81.07994783143533]
  Large-scale labeled training datasets have enabled deep neural networks to excel across a wide range of benchmark vision tasks. In many applications, it is prohibitively expensive and time-consuming to obtain large quantities of labeled data. To cope with limited labeled training data, many have attempted to directly apply models trained on a large-scale labeled source domain to another sparsely labeled or unlabeled target domain.
  arXiv  Detail & Related papers  (2020-09-01T00:06:50Z)
• The Paradigm Discovery Problem [121.79963594279893]
  We formalize the paradigm discovery problem and develop metrics for judging systems. We report empirical results on five diverse languages. Our code and data are available for public use.
  arXiv  Detail & Related papers  (2020-05-04T16:38:54Z)
• Unsupervised Domain Clusters in Pretrained Language Models [61.832234606157286]
  We show that massive pre-trained language models implicitly learn sentence representations that cluster by domains without supervision. We propose domain data selection methods based on such models. We evaluate our data selection methods for neural machine translation across five diverse domains.
  arXiv  Detail & Related papers  (2020-04-05T06:22:16Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.