ReSi: A Comprehensive Benchmark for Representational Similarity Measures

• URL: http://arxiv.org/abs/2408.00531v1
• Date: Thu, 1 Aug 2024 13:08:02 GMT
• Title: ReSi: A Comprehensive Benchmark for Representational Similarity Measures
• Authors: Max Klabunde, Tassilo Wald, Tobias Schumacher, Klaus Maier-Hein, Markus Strohmaier, Florian Lemmerich,
• Abstract summary: This paper presents the first comprehensive benchmark for evaluating representational similarity measures. The representational similarity (ReSi) benchmark consists of (i) six carefully designed tests for similarity measures, (ii) 23 similarity measures, (iii) eleven neural network architectures, and (iv) six datasets, spanning over the graph, language, and vision domains.
• Score: 2.3532263743300432
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Measuring the similarity of different representations of neural architectures is a fundamental task and an open research challenge for the machine learning community. This paper presents the first comprehensive benchmark for evaluating representational similarity measures based on well-defined groundings of similarity. The representational similarity (ReSi) benchmark consists of (i) six carefully designed tests for similarity measures, (ii) 23 similarity measures, (iii) eleven neural network architectures, and (iv) six datasets, spanning over the graph, language, and vision domains. The benchmark opens up several important avenues of research on representational similarity that enable novel explorations and applications of neural architectures. We demonstrate the utility of the ReSi benchmark by conducting experiments on various neural network architectures, real world datasets and similarity measures. All components of the benchmark are publicly available and thereby facilitate systematic reproduction and production of research results. The benchmark is extensible, future research can build on and further expand it. We believe that the ReSi benchmark can serve as a sound platform catalyzing future research that aims to systematically evaluate existing and explore novel ways of comparing representations of neural architectures.

Related papers

• Evaluating Representational Similarity Measures from the Lens of Functional Correspondence [1.7811840395202345]
  Neuroscience and artificial intelligence (AI) both face the challenge of interpreting high-dimensional neural data. Despite the widespread use of representational comparisons, a critical question remains: which metrics are most suitable for these comparisons?
  arXiv  Detail & Related papers  (2024-11-21T23:53:58Z)
• Dynamical similarity analysis uniquely captures how computations develop in RNNs [3.037387520023979]
  Recent findings show that some metrics respond to spurious signals, leading to misleading results. We propose that compositional learning in recurrent neural networks (RNNs) can provide a test case for dynamical representation alignment metrics. We show that the recently proposed Dynamical Similarity Analysis (DSA) is more noise robust and reliably identifies behaviorally relevant representations.
  arXiv  Detail & Related papers  (2024-10-31T16:07:21Z)
• Multilayer Multiset Neuronal Networks -- MMNNs [55.2480439325792]
  The present work describes multilayer multiset neuronal networks incorporating two or more layers of coincidence similarity neurons. The work also explores the utilization of counter-prototype points, which are assigned to the image regions to be avoided.
  arXiv  Detail & Related papers  (2023-08-28T12:55:13Z)
• Similarity of Neural Network Models: A Survey of Functional and Representational Measures [2.56552999376511]
  Measuring similarity of neural networks to understand and improve their behavior has become an issue of great importance and research interest. We provide a comprehensive overview of two complementary perspectives of measuring neural network similarity: (i) representational similarity, which considers how activations of intermediate layers differ, and (ii) functional similarity, which considers how models differ in their outputs.
  arXiv  Detail & Related papers  (2023-05-10T17:33:48Z)
• Revisiting the Evaluation of Image Synthesis with GANs [55.72247435112475]
  This study presents an empirical investigation into the evaluation of synthesis performance, with generative adversarial networks (GANs) as a representative of generative models. In particular, we make in-depth analyses of various factors, including how to represent a data point in the representation space, how to calculate a fair distance using selected samples, and how many instances to use from each set.
  arXiv  Detail & Related papers  (2023-04-04T17:54:32Z)
• Benchmarking the Benchmark -- Analysis of Synthetic NIDS Datasets [4.125187280299247]
  We analyse the statistical properties of benign traffic in three of the more recent and relevant NIDS datasets. Our results show a distinct difference of most of the considered statistical features between the synthetic datasets and two real-world datasets.
  arXiv  Detail & Related papers  (2021-04-19T03:17:37Z)
• Few-Shot Named Entity Recognition: A Comprehensive Study [92.40991050806544]
  We investigate three schemes to improve the model generalization ability for few-shot settings. We perform empirical comparisons on 10 public NER datasets with various proportions of labeled data. We create new state-of-the-art results on both few-shot and training-free settings.
  arXiv  Detail & Related papers  (2020-12-29T23:43:16Z)
• CDEvalSumm: An Empirical Study of Cross-Dataset Evaluation for Neural Summarization Systems [121.78477833009671]
  We investigate the performance of different summarization models under a cross-dataset setting. A comprehensive study of 11 representative summarization systems on 5 datasets from different domains reveals the effect of model architectures and generation ways.
  arXiv  Detail & Related papers  (2020-10-11T02:19:15Z)
• A Semi-Supervised Assessor of Neural Architectures [157.76189339451565]
  We employ an auto-encoder to discover meaningful representations of neural architectures. A graph convolutional neural network is introduced to predict the performance of architectures.
  arXiv  Detail & Related papers  (2020-05-14T09:02:33Z)
• Heterogeneous Network Representation Learning: A Unified Framework with Survey and Benchmark [57.10850350508929]
  We aim to provide a unified framework to summarize and evaluate existing research on heterogeneous network embedding (HNE) As the first contribution, we provide a generic paradigm for the systematic categorization and analysis over the merits of various existing HNE algorithms. As the second contribution, we create four benchmark datasets with various properties regarding scale, structure, attribute/label availability, and etcfrom different sources. As the third contribution, we create friendly interfaces for 13 popular HNE algorithms, and provide all-around comparisons among them over multiple tasks and experimental settings.
  arXiv  Detail & Related papers  (2020-04-01T03:42:11Z)
• Similarity of Neural Networks with Gradients [8.804507286438781]
  We propose to leverage both feature vectors and gradient ones into designing the representation of a neural network. We show that the proposed approach provides a state-of-the-art method for computing similarity of neural networks.
  arXiv  Detail & Related papers  (2020-03-25T17:04:10Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.