DANCE improves saliency maps by adding subtle input variations.
problem Poor performance of saliency methods in saturated gradients, adversarial perturbations, and inter-feature dependence.
method Two-step procedure: 1) Perturbation mechanism, 2) Aggregation of saliency maps.
result DANCE saliency method outperforms existing methods qualitatively and quantitatively.
New nonlinear saliency maps improve deep neural network interpretability.
problem Lack of understanding why and how deep neural networks make decisions.
method Developed novel nonlinear saliency maps to better interpret deep neural networks.
result Nonlinear saliency maps provide more specific drivers of classification on complex examples.
New method improves interpretability of fMRI decoding models.
problem Uninterpretable deep neural networks in fMRI decoding.
method Adversarial training to make DNNs robust to noise and improved saliency map methods.
result Saliency maps from adversarial-trained DNNs are more interpretable than those from other methods.
Study validates saliency maps of GNNs using selective inference.
problem Reliability of GNN saliency maps in graph-structured data.
method Statistical testing framework with selective inference to control Type I error rate.
result Valid p-values for salient subgraphs, ensuring meaningful information. The paper benchmarks saliency methods for time series predictions.
problem Benchmarking saliency methods for time series predictions.
method Compared saliency-based interpretability methods across diverse neural architectures.
result Saliency methods fail to reliably identify feature importance over time in time series data.
Input-cell attention improves RNN saliency detection over time.
problem Vanishing saliency in RNNs makes them unreliable for detecting important features at arbitrary time intervals.
method Proposes input-cell attention, a novel RNN cell structure that uses a fixed-size matrix embedding to attend to different inputs from current or previous time steps.
result Input-cell attention produces a saliency map that can detect important features regardless of their occurrence in time.
New method for visualizing high-level concepts in generative models.
problem Challenges in evaluating and visualizing concepts in generative models.
method Introduces a method to compute concept saliency maps for latent representations of known or novel high-level concepts.
result Concept saliency maps highlight input features important for high-level concepts.
Saliency Map, the gradient of the score function with respect to the input, is the most basic technique for interpreting deep neural network decisions. However, saliency maps are often visually noisy. Although several hypotheses were proposed to account for this phenomenon, there are few works that provide rigorous ana…
Recent studies on the adversarial vulnerability of neural networks have shown that models trained to be more robust to adversarial attacks exhibit more interpretable saliency maps than their non-robust counterparts. We aim to quantify this behavior by considering the alignment between input image and saliency map. We h…
In this work we develop a fast saliency detection method that can be applied to any differentiable image classifier. We train a masking model to manipulate the scores of the classifier by masking salient parts of the input image. Our model generalises well to unseen images and requires a single forward pass to perform …
Bayesian optimisation generates saliency maps for black-box models.
problem Generating saliency maps for models without access to parameters.
method Bayesian optimisation sampling method to find global salient regions.
result Approach outperforms grid-based methods and performs similarly to gradient-based methods.
Bayesian explanations are more resilient to adversarial attacks than deterministic ones.
problem Stability of saliency-based explanations under adversarial attacks in Neural Networks.
method Empirical and theoretical analysis of Bayesian vs deterministic Neural Networks.
result Bayesian explanations are more stable under adversarial perturbations and direct attacks.
Proposes a method to quantify the reliability of salient regions in deep learning models using p-values.
problem Difficulty in assessing the reliability of saliency maps generated by deep learning models.
method Proposes a selective inference framework to quantify the reliability of salient regions as selected hypotheses by deep learning models.
result The method can provably control the probability of false positive detections of salient regions.
Deep learning interpretation is essential to explain the reasoning behind model predictions. Understanding the robustness of interpretation methods is important especially in sensitive domains such as medical applications since interpretation results are often used in downstream tasks. Although gradient-based saliency …
CRITS improves time series classification with interpretable local explanations.
problem Lack of detailed explanations in time series classification models.
method CRITS uses convolutional kernels, max-pooling, and rectified linear units to extract feature weights.
result CRITS provides intrinsically interpretable local explanations without requiring gradients or random perturbations.
We introduce a new tool for interpreting neural net responses, namely full-gradients, which decomposes the neural net response into input sensitivity and per-neuron sensitivity components. This is the first proposed representation which satisfies two key properties: completeness and weak dependence, which provably cann…
A new LDA variant improves multi-label classification performance.
problem Improving multi-label classification performance.
method Saliency-based weights redefine between-class and within-class scatter matrices for multi-label classification.
result The proposed method leads to performance improvements in various multi-label classification problems.
A new method extracts linguistic objects from text using CNNs.
problem Lack of interpretability in deep learning models for text.
method Weighted extension of Text Deconvolution Saliency (wTDS) measure.
result Extracts interpretable linguistic objects from text.
A novel method using saliency maps detects adversarial attacks effectively.
problem Vulnerability of neural networks to adversarial attacks.
method Combining saliency maps with additional noises and inconsistency strategy.
result High detection success rate of adversarial attacks on various datasets.
Enhances counterfactual explanations with more valid and informative saliency maps.
problem Lack of valid counterfactual explanations in existing models.
method Introduces a modified approach to CELS model by removing mask normalization.
result Demonstrates higher validity and more informative counterfactual explanations.
With recent advances in deep learning, neuroimaging studies increasingly rely on convolutional networks (ConvNets) to predict diagnosis based on MR images. To gain a better understanding of how a disease impacts the brain, the studies visualize the salience maps of the ConvNet highlighting voxels within the brain major…
Study identifies and mitigates causes of image misclassifications in CNN models.
problem Improving model interpretability and accuracy in image classification.
method Trained six CNN architectures on CIFAR-10, used conditional confusion matrices and misclassification networks to identify morphological similarity and non-essential information interference as causes of misclassification. Developed a method to reduce misclassifications by erasing pixels within top 5% saliency map bounding boxes.
result Identified two causes of misclassification: morphological similarity and non-essential information interference, and developed a method to reduce misclassifications caused by the latter.
Saliency methods have emerged as a popular tool to highlight features in an input deemed relevant for the prediction of a learned model. Several saliency methods have been proposed, often guided by visual appeal on image data. In this work, we propose an actionable methodology to evaluate what kinds of explanations a g…
Novel method combines neural network features with survival models for ICU infections.
problem Improving predictive models of ICU infections while maintaining interpretability.
method Semi-parametric approach combining low-resolution and high-resolution data.
result Improved predictive power with interpretability maintained.
Unsupervised method discovers interpretable directions in GAN latent space.
problem Discovering interpretable directions in GAN latent space without supervision.
method Model-agnostic procedure to identify directions corresponding to semantic manipulations.
result Findings include directions for background removal and competitive saliency detection performance.
New method creates universal perturbations to fool neural network interpretations.
problem Vulnerability of gradient-based saliency maps to adversarial perturbations.
method Gradient-based optimization and PCA-based approach to create UPI.
result Existence and successful application of Universal Perturbation for Interpretation (UPI).
Classic feature selection techniques remove those features that are either irrelevant or redundant, achieving a subset of relevant features that help to provide a better knowledge extraction. This allows the creation of compact models that are easier to interpret. Most of these techniques work over the whole dataset, b…
RCAV quantifies model sensitivity to semantic concepts, improving interpretability methods.
problem Lack of semantic interpretability in image classification models.
method RCAV calculates concept gradients and ascent steps to assess model sensitivity to semantic concepts.
result RCAV yields more accurate and robust interpretations of model behavior.
Two modified tests improve the reliability of evaluating explanation methods.
problem Methodological concerns in evaluating explanation methods for saliency maps.
method Proposed modifications to the Model Parameter Randomisation Test (MPRT): Smooth MPRT and Efficient MPRT.
result Enhanced metric reliability, facilitating more trustworthy deployment of explanation methods.
A method to combine saliency metrics for better CNN pruning decisions.
problem Improving CNN pruning decisions by combining multiple saliency metrics.
method Proposes a method to compose different saliency metrics for better CNN pruning decisions.
result The composition of saliencies avoids many poor pruning choices identified by individual saliencies.
The study evaluates saliency metrics for image classifier outputs, finding inconsistencies and unreliability.
problem Inconsistencies and unreliability in saliency metrics for evaluating pixel relevance.
method Investigated existing saliency metrics, calculated and compared their consistency, and applied psychometric testing methods.
result Saliency metrics can be statistically unreliable and inconsistent, affecting comparative rankings.
Study evaluates different saliency maps for CT image classification.
problem Understanding how deep learning models make decisions in medical imaging.
method Compared several saliency maps using three evaluation metrics.
result Two versions of SmoothGrad performed best across all metrics.
SmoothGrad and VarGrad are techniques that enhance the empirical quality of standard saliency maps by adding noise to input. However, there were few works that provide a rigorous theoretical interpretation of those methods. We analytically formalize the result of these noise-adding methods. As a result, we observe two …
Convolutional-deconvolution networks can be adopted to perform end-to-end saliency detection. But, they do not work well with objects of multiple scales. To overcome such a limitation, in this work, we propose a recurrent attentional convolutional-deconvolution network (RACDNN). Using spatial transformer and recurrent …
Improved saliency maps for deep neural networks with reduced noise.
problem Noisy explanations in Integrated Gradients for deep neural networks.
method SmoothTaylor, adaptive noising, and SmoothGrad techniques.
result SmoothTaylor and adaptive noising generate better quality saliency maps.
Study evaluates saliency maps on artificial data with different backgrounds.
problem Objective evaluation of saliency methods on artificial data with varying backgrounds.
method Developed a framework to generate artificial data with synthetic lesions and a known ground truth map, evaluated two data sets with different backgrounds (Perlin noise and 2D brain MRI slices).
result Heatmaps vary strongly between saliency methods and backgrounds.
Deep learning models designed for visual classification tasks on natural images have become prevalent in medical image analysis. However, medical images differ from typical natural images in many ways, such as significantly higher resolutions and smaller regions of interest. Moreover, both the global structure and loca…
The study predicts Kronecker coefficients using interpretable machine learning models.
problem Predicting Kronecker coefficients of the symmetric group.
method Employed interpretable machine learning models with input features of triples of partitions and b-loadings.
result Achieved an accuracy of approximately 83% and over 99% with transformer-based models.
Pruning unimportant parameters can allow deep neural networks (DNNs) to reduce their heavy computation and memory requirements. A saliency metric estimates which parameters can be safely pruned with little impact on the classification performance of the DNN. Many saliency metrics have been proposed, each within the con…
In this paper we propose a novel method for detecting adversarial examples by training a binary classifier with both origin data and saliency data. In the case of image classification model, saliency simply explain how the model make decisions by identifying significant pixels for prediction. A model shows wrong classi…
Saliency methods can aid understanding of deep neural networks. Recent years have witnessed many improvements to saliency methods, as well as new ways for evaluating them. In this paper, we 1) present a novel region-based attribution method, XRAI, that builds upon integrated gradients (Sundararajan et al. 2017), 2) int…
New approach makes adversarial examples less suspicious without changing perceptual salience.
problem Robustness of deep neural networks to unsuspicious adversarial examples.
method Splitting images into foreground and background, allowing larger perturbations in background while maintaining low cognitive salience.
result Dual-perturbation attacks are effective against classifiers robust to conventional attacks and adversarial training yields more robust classifiers.
Saliency methods aim to explain the predictions of deep neural networks. These methods lack reliability when the explanation is sensitive to factors that do not contribute to the model prediction. We use a simple and common pre-processing step ---adding a constant shift to the input data--- to show that a transformatio…
Method uses ANN to estimate incentive salience from large behavioral data.
problem Estimating incentive salience in naturalistic settings.
method Artificial Neural Networks (ANNs) for latent state approximation.
result ANNs produce better representations for predicting future behaviour.
New saliency evaluations focus on completeness and soundness, improving explanations.
problem Current saliency evaluations focus on completeness but ignore soundness.
method Introduces new intrinsic evaluation metrics based on completeness and soundness.
result Simple saliency method matches or outperforms prior methods in new evaluations.
Convolutional neural networks predict the analytic rank of elliptic curves accurately.
problem Predicting the analytic rank of elliptic curves over Q.
method Applied one-dimensional convolutional neural networks to Frobenius traces.
result High accuracy predictions for analytic rank across various conductors.
Puzzle Mix optimizes mixup by leveraging saliency and local statistics.
problem Overfitting and adversarial vulnerability of deep networks.
method Puzzle Mix alternates between multi-label and saliency-based objectives for optimal mixing masks.
result Puzzle Mix outperforms other mixup methods in generalization and adversarial robustness.
We propose a novel perspective to understand deep neural networks in an interpretable disentanglement form. For each semantic class, we extract a class-specific functional subnetwork from the original full model, with compressed structure while maintaining comparable prediction performance. The structure representation…