DORA: Exploring Outlier Representations in Deep Neural Networks
- URL: http://arxiv.org/abs/2206.04530v4
- Date: Mon, 10 Jul 2023 15:42:34 GMT
- Title: DORA: Exploring Outlier Representations in Deep Neural Networks
- Authors: Kirill Bykov, Mayukh Deb, Dennis Grinwald, Klaus-Robert M\"uller,
Marina M.-C. H\"ohne
- Abstract summary: We present DORA, the first data-agnostic framework for analyzing the representational space of Deep Neural Networks (DNNs)
Central to our framework is the proposed Extreme-Activation (EA) distance measure, which assesses similarities between representations.
We validate the EA metric quantitatively, demonstrating its effectiveness both in controlled scenarios and real-world applications.
- Score: 0.0
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: Deep Neural Networks (DNNs) excel at learning complex abstractions within
their internal representations. However, the concepts they learn remain opaque,
a problem that becomes particularly acute when models unintentionally learn
spurious correlations. In this work, we present DORA (Data-agnOstic
Representation Analysis), the first data-agnostic framework for analyzing the
representational space of DNNs. Central to our framework is the proposed
Extreme-Activation (EA) distance measure, which assesses similarities between
representations by analyzing their activation patterns on data points that
cause the highest level of activation. As spurious correlations often manifest
in features of data that are anomalous to the desired task, such as watermarks
or artifacts, we demonstrate that internal representations capable of detecting
such artifactual concepts can be found by analyzing relationships within neural
representations. We validate the EA metric quantitatively, demonstrating its
effectiveness both in controlled scenarios and real-world applications.
Finally, we provide practical examples from popular Computer Vision models to
illustrate that representations identified as outliers using the EA metric
often correspond to undesired and spurious concepts.
Related papers
- Dynamical similarity analysis uniquely captures how computations develop in RNNs [3.037387520023979]
Recent findings show that some metrics respond to spurious signals, leading to misleading results.
We propose that compositional learning in recurrent neural networks (RNNs) can provide a test case for dynamical representation alignment metrics.
We show that the recently proposed Dynamical Similarity Analysis (DSA) is more noise robust and reliably identifies behaviorally relevant representations.
arXiv Detail & Related papers (2024-10-31T16:07:21Z) - Interactive dense pixel visualizations for time series and model attribution explanations [8.24039921933289]
DAVOTS is an interactive visual analytics approach to explore raw time series data, activations of neural networks, and attributions in a dense-pixel visualization.
We apply clustering approaches to the visualized data domains to highlight groups and present ordering strategies for individual and combined data exploration.
arXiv Detail & Related papers (2024-08-27T14:02:21Z) - Tipping Points of Evolving Epidemiological Networks: Machine
Learning-Assisted, Data-Driven Effective Modeling [0.0]
We study the tipping point collective dynamics of an adaptive susceptible-infected (SIS) epidemiological network in a data-driven, machine learning-assisted manner.
We identify a complex effective differential equation (eSDE) in terms physically meaningful coarse mean-field variables.
We study the statistics of rare events both through repeated brute force simulations and by using established mathematical/computational tools.
arXiv Detail & Related papers (2023-11-01T19:33:03Z) - Robust Saliency-Aware Distillation for Few-shot Fine-grained Visual
Recognition [57.08108545219043]
Recognizing novel sub-categories with scarce samples is an essential and challenging research topic in computer vision.
Existing literature addresses this challenge by employing local-based representation approaches.
This article proposes a novel model, Robust Saliency-aware Distillation (RSaD), for few-shot fine-grained visual recognition.
arXiv Detail & Related papers (2023-05-12T00:13:17Z) - Experimental Observations of the Topology of Convolutional Neural
Network Activations [2.4235626091331737]
Topological data analysis provides compact, noise-robust representations of complex structures.
Deep neural networks (DNNs) learn millions of parameters associated with a series of transformations defined by the model architecture.
In this paper, we apply cutting edge techniques from TDA with the goal of gaining insight into the interpretability of convolutional neural networks used for image classification.
arXiv Detail & Related papers (2022-12-01T02:05:44Z) - Sparse Relational Reasoning with Object-Centric Representations [78.83747601814669]
We investigate the composability of soft-rules learned by relational neural architectures when operating over object-centric representations.
We find that increasing sparsity, especially on features, improves the performance of some models and leads to simpler relations.
arXiv Detail & Related papers (2022-07-15T14:57:33Z) - On Neural Architecture Inductive Biases for Relational Tasks [76.18938462270503]
We introduce a simple architecture based on similarity-distribution scores which we name Compositional Network generalization (CoRelNet)
We find that simple architectural choices can outperform existing models in out-of-distribution generalizations.
arXiv Detail & Related papers (2022-06-09T16:24:01Z) - Temporal Relevance Analysis for Video Action Models [70.39411261685963]
We first propose a new approach to quantify the temporal relationships between frames captured by CNN-based action models.
We then conduct comprehensive experiments and in-depth analysis to provide a better understanding of how temporal modeling is affected.
arXiv Detail & Related papers (2022-04-25T19:06:48Z) - Explainable Adversarial Attacks in Deep Neural Networks Using Activation
Profiles [69.9674326582747]
This paper presents a visual framework to investigate neural network models subjected to adversarial examples.
We show how observing these elements can quickly pinpoint exploited areas in a model.
arXiv Detail & Related papers (2021-03-18T13:04:21Z) - Relation-Guided Representation Learning [53.60351496449232]
We propose a new representation learning method that explicitly models and leverages sample relations.
Our framework well preserves the relations between samples.
By seeking to embed samples into subspace, we show that our method can address the large-scale and out-of-sample problem.
arXiv Detail & Related papers (2020-07-11T10:57:45Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.