What Sketch Explainability Really Means for Downstream Tasks

• URL: http://arxiv.org/abs/2403.09480v1
• Date: Thu, 14 Mar 2024 15:22:33 GMT
• Title: What Sketch Explainability Really Means for Downstream Tasks
• Abstract summary: We explore the unique modality of sketch for explainability, emphasising the profound impact of human strokes. We propose a lightweight and portable explainability solution -- a seamless plugin that integrates effortlessly with any pre-trained model.
• Score: 121.85845328822641
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: In this paper, we explore the unique modality of sketch for explainability, emphasising the profound impact of human strokes compared to conventional pixel-oriented studies. Beyond explanations of network behavior, we discern the genuine implications of explainability across diverse downstream sketch-related tasks. We propose a lightweight and portable explainability solution -- a seamless plugin that integrates effortlessly with any pre-trained model, eliminating the need for re-training. Demonstrating its adaptability, we present four applications: highly studied retrieval and generation, and completely novel assisted drawing and sketch adversarial attacks. The centrepiece to our solution is a stroke-level attribution map that takes different forms when linked with downstream tasks. By addressing the inherent non-differentiability of rasterisation, we enable explanations at both coarse stroke level (SLA) and partial stroke level (P-SLA), each with its advantages for specific downstream tasks.

Related papers

• Context-Aware Interpretable Representations for Retrieval and Graph Convolutional Network Classification [3.5232028627328895]
  We propose a novel unsupervised framework that integrates Manifold Learning strategies with Rank-based Interpretable Graph Embeddings.<n>Our approach effectively bridges these gaps by first characterizing the contextual information of the dataset through manifold analysis and subsequently generating sparse, self-explainable embeddings.
  arXiv  Detail & Related papers  (2026-08-29T02:29:59Z)
• Provable Benefits of Task-Specific Prompts for In-context Learning [44.768199865867494]
  In this work, we consider a novel setting where the global task distribution can be partitioned into a union of conditional task distributions. We then examine the use of task-specific prompts and prediction heads for learning the prior information associated with the conditional task distribution using a one-layer attention model.
  arXiv  Detail & Related papers  (2025-03-03T22:37:03Z)
• Spurious Feature Eraser: Stabilizing Test-Time Adaptation for Vision-Language Foundation Model [86.9619638550683]
  Vision-language foundation models have exhibited remarkable success across a multitude of downstream tasks due to their scalability on extensive image-text paired data. However, these models display significant limitations when applied to downstream tasks, such as fine-grained image classification, as a result of decision shortcuts''
  arXiv  Detail & Related papers  (2024-03-01T09:01:53Z)
• Enhancing Few-shot CLIP with Semantic-Aware Fine-Tuning [61.902254546858465]
  Methods based on Contrastive Language-Image Pre-training have exhibited promising performance in few-shot adaptation tasks. We propose fine-tuning the parameters of the attention pooling layer during the training process to encourage the model to focus on task-specific semantics.
  arXiv  Detail & Related papers  (2023-11-08T05:18:57Z)
• Disentangled Representation Learning with Transmitted Information Bottleneck [57.22757813140418]
  We present textbfDisTIB (textbfTransmitted textbfInformation textbfBottleneck for textbfDisd representation learning), a novel objective that navigates the balance between information compression and preservation.
  arXiv  Detail & Related papers  (2023-11-03T03:18:40Z)
• Drawing out of Distribution with Neuro-Symbolic Generative Models [49.79371715591122]
  Drawing out of Distribution is a neuro-symbolic generative model of stroke-based drawing. DooD operates directly on images, requires no supervision or expensive test-time inference. We evaluate DooD on its ability to generalise across both data and tasks.
  arXiv  Detail & Related papers  (2022-06-03T21:40:22Z)
• Explaining, Evaluating and Enhancing Neural Networks' Learned Representations [2.1485350418225244]
  We show how explainability can be an aid, rather than an obstacle, towards better and more efficient representations. We employ such attributions to define two novel scores for evaluating the informativeness and the disentanglement of latent embeddings. We show that adopting our proposed scores as constraints during the training of a representation learning task improves the downstream performance of the model.
  arXiv  Detail & Related papers  (2022-02-18T19:00:01Z)
• BDA-SketRet: Bi-Level Domain Adaptation for Zero-Shot SBIR [52.78253400327191]
  BDA-SketRet is a novel framework performing a bi-level domain adaptation for aligning the spatial and semantic features of the visual data pairs. Experimental results on the extended Sketchy, TU-Berlin, and QuickDraw exhibit sharp improvements over the literature.
  arXiv  Detail & Related papers  (2022-01-17T18:45:55Z)
• Aggregative Self-Supervised Feature Learning from a Limited Sample [12.555160911451688]
  We propose two strategies of aggregation in terms of complementarity of various forms to boost the robustness of self-supervised learned features. Our experiments on 2D natural image and 3D medical image classification tasks under limited data scenarios confirm that the proposed aggregation strategies successfully boost the classification accuracy.
  arXiv  Detail & Related papers  (2020-12-14T12:49:37Z)
• Proactive Pseudo-Intervention: Causally Informed Contrastive Learning For Interpretable Vision Models [103.64435911083432]
  We present a novel contrastive learning strategy called it Proactive Pseudo-Intervention (PPI) PPI leverages proactive interventions to guard against image features with no causal relevance. We also devise a novel causally informed salience mapping module to identify key image pixels to intervene, and show it greatly facilitates model interpretability.
  arXiv  Detail & Related papers  (2020-12-06T20:30:26Z)
• Anatomy of Catastrophic Forgetting: Hidden Representations and Task Semantics [24.57617154267565]
  We investigate how forgetting affects representations in neural network models. We find that deeper layers are disproportionately the source of forgetting. We also introduce a novel CIFAR-100 based task approximating realistic input distribution shift.
  arXiv  Detail & Related papers  (2020-07-14T23:31:14Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.