Framework evaluates post-hoc interpretability methods in time-series classification.
problem Lack of suitable post-hoc interpretability methods for time-series classification.
method Proposes a framework with quantitative metrics to assess interpretability methods.
result Addresses several drawbacks of existing methods, including dependence on human judgement and data distribution shift.
Post-hoc model-agnostic interpretation methods such as partial dependence plots can be employed to interpret complex machine learning models. While these interpretation methods can be applied regardless of model complexity, they can produce misleading and verbose results if the model is too complex, especially w.r.t. f…
Mathematical study shows post-hoc explanations are better than attention weights alone.
problem Understanding the internal behavior of attention-based models.
method Mathematical analysis of a simple attention-based architecture.
result Post-hoc explanations provide more useful insights than attention weights alone.
Counterfactual post-hoc interpretability approaches have been proven to be useful tools to generate explanations for the predictions of a trained blackbox classifier. However, the assumptions they make about the data and the classifier make them unreliable in many contexts. In this paper, we discuss three desirable pro…
Most of the work on interpretable machine learning has focused on designing either inherently interpretable models, which typically trade-off accuracy for interpretability, or post-hoc explanation systems, whose explanation quality can be unpredictable. Our method, ExpO, is a hybridization of these approaches that regu…
FLANs process each feature separately for better interpretability.
problem Need for interpretable machine learning models in critical scenarios.
method Feature-wise latent representations summed for prediction.
result FLANs enhance interpretability without sacrificing performance.
New method calibrates neural network predictions for better reliability.
problem Improper probability estimates from deep networks leading to unreliable predictions.
method Proposes a constrained optimization approach for a monotonic calibration map.
result Achieves state-of-the-art performance across various datasets and models.
Most existing interpretable methods explain a black-box model in a post-hoc manner, which uses simpler models or data analysis techniques to interpret the predictions after the model is learned. However, they (a) may derive contradictory explanations on the same predictions given different methods and data samples, and…
iKF method uncovers complex variable interactions for scientific discovery.
problem Limited interpretability of existing models in decision-making applications.
method Iterative Kings' Forests (iKF) method to uncover multi-order interactions.
result iKF provides strong interpretive power for explainable modeling.
Kolmogorov-Arnold Networks offer interpretable models for energy applications.
problem Lack of interpretability in modern machine learning methods for sensitive industries.
method Symbolic regression with Kolmogorov-Arnold Networks compared to traditional feedforward neural networks.
result Kolmogorov-Arnold Networks yield perfectly interpretable models and learn real, physical relations.
Post-hoc interpretability approaches have been proven to be powerful tools to generate explanations for the predictions made by a trained black-box model. However, they create the risk of having explanations that are a result of some artifacts learned by the model instead of actual knowledge from the data. This paper f…
This review explores methods to explain deep neural networks and their applications.
problem Understanding the decision-making process of deep neural networks.
method Overview of interpretability methods, theoretical foundations, and comparative evaluations.
result Demonstrates the effectiveness of explainable AI in various applications.
S-LIME stabilizes LIME for more reliable model explanations.
problem Instability of post hoc explanation methods like LIME.
method Uses hypothesis testing based on central limit theorem to stabilize explanations.
result Demonstrates effectiveness of S-LIME on simulated and real-world data.
Paper introduces GAMs for interpretable learning-to-rank models.
problem Need for transparent ranking models in legal or policy scenarios.
method Developed generalized additive models (GAMs) for ranking tasks using neural networks.
result Neural ranking GAMs achieve better performance than traditional GAMs while maintaining interpretability.
A framework for quantifying uncertainty in feature importance values.
problem Stable interpretation of feature importance values in machine learning models.
method A novel method based on pairwise comparisons of feature importance values to produce confidence intervals for feature ranks.
result The method produces simultaneous confidence intervals for feature ranks, enabling selection of top-k important features.
PACE explains ViTs by modeling patch-level concept distributions, surpassing existing methods.
problem Lack of trustworthy post-hoc explanations for Vision Transformers (ViTs)
method Variational Bayesian explanation framework (PACE)
result PACE surpasses state-of-the-art methods in meeting desiderata for ViT explanations.
Paper proposes hybrid approach for transparent credit scoring models.
problem Lack of transparency in machine learning models limits their use in regulated environments.
method Post-hoc interpretation of black-box models guides feature selection, followed by training glass-box models.
result Reduces feature usage from 106 to 10 while maintaining comparable performance.
DFI maps covariates to latent representations for feature importance.
problem Feature importance when predictors are statistically dependent.
method Disentangled Feature Importance (DFI) using entropic optimal transport.
result DFI yields stable, interpretable, uncertainty-quantified attributions of shared predictive signal.
In this paper, we study two challenging problems in explainable AI (XAI) and data clustering. The first is how to directly design a neural network with inherent interpretability, rather than giving post-hoc explanations of a black-box model. The second is implementing discrete k-means with a differentiable neural net…
New ML algorithms improve model interpretability without sacrificing performance.
problem Lack of interpretability in complex machine learning models.
method Developed new algorithms based on fANOVA framework, including GAMI-Lin-T and GAMI-Net.
result GAMI-Lin-T and GAMI-Net perform comparably to EBM and better in interpretability.
As machine learning black boxes are increasingly being deployed in domains such as healthcare and criminal justice, there is growing emphasis on building tools and techniques for explaining these black boxes in an interpretable manner. Such explanations are being leveraged by domain experts to diagnose systematic error…
Supervised machine learning models boast remarkable predictive capabilities. But can you trust your model? Will it work in deployment? What else can it tell you about the world? We want models to be not only good, but interpretable. And yet the task of interpretation appears underspecified. Papers provide diverse and s…
PCBMs turn any neural network into interpretable models without dense annotations.
problem Restrictive nature of CBMs and lack of dense concept annotations in training data.
method Introduce PCBMs that can turn any neural network into interpretable models without dense annotations.
result PCBMs can turn any neural network into interpretable models without dense annotations, improving interpretability and performance.
XDeep is an open-source Python package developed to interpret deep models for both practitioners and researchers. Overall, XDeep takes a trained deep neural network (DNN) as the input, and generates relevant interpretations as the output with the post-hoc manner. From the functionality perspective, XDeep integrates a w…
CB-APM uses analyst consensus as a bottleneck to interpret stock returns.
problem Tackles the challenge of understanding and predicting stock returns using professional beliefs.
method Embeds analyst consensus as a structural bottleneck, treating it as a sufficient statistic for market information.
result CB-APM portfolios exhibit strong monotonic return gradients and robust across different economic conditions.
HS improves tree-based models' accuracy and interpretability without changing their structure.
problem Overfitting in tree-based models.
method Hierarchical Shrinkage (HS) post-hoc algorithm that shrinks tree predictions towards ancestor means.
result HS significantly improves predictive performance and interpretability of decision trees and RFs.
Post-hoc transforms can reverse model performance trends, especially in noisy settings.
problem Post-hoc transforms can reverse model performance trends, especially in noisy settings.
method Empirical study and analysis of post-hoc transforms like temperature scaling, ensembling, and SWA.
result Post-hoc reversal can prevent double descent and mitigate mismatches between test loss and test error.
Most of the work on interpretable machine learning has focused on designing either inherently interpretable models, which typically trade-off accuracy for interpretability, or post-hoc explanation systems, which lack guarantees about their explanation quality. We propose an alternative to these approaches by directly r…
New methods identify concepts in trained embeddings reliably without human labels.
problem Identifying interpretable concepts in trained embedding spaces without human labels.
method Explicitly connecting concept discovery to PCA and ICA, proposing novel approaches for dependent concepts.
result Proven methods outperform competitors on a variety of experiments, achieving up to 29% better alignment with ground truth.
Proposes a method for explaining tabular data using copulas.
problem Lack of ground truth for explainability in complex datasets.
method Uses copulas to specify statistical properties and build intuition.
result Demonstrates improved explainability on logistic regression and correlation use cases.
Post-hoc calibration improves uncertainty under domain shift.
problem Improving uncertainty calibration under domain shift.
method Apply perturbations to validation set before post-hoc calibration.
result Perturbation step results in better calibration under domain shift.
Shallow trees in ensemble models make models more interpretable and sometimes better.
problem Lack of transparency in high-performing tree ensemble models.
method Developed an interpretation algorithm to convert tree ensembles into functional ANOVA representations. Proposed strategies to enhance interpretability.
result Shallow trees in ensemble models can lead to better generalization performance and improved interpretability.
Integrating causal machine learning with inherently interpretable models for decision support.
problem Providing causal insights and decision support through machine learning models.
method Proposing an approach that integrates causal machine learning with inherently interpretable models.
result The proposed approach achieves competitive performance in prediction and what-if analysis while offering transparency on the system structure, causal relationships among variables, and functional forms connecting them.
A framework learns interpretable models with minimal accuracy loss.
problem Tackles interpretability in deep learning models.
method Jointly learns a predictive model and its interpretation model with minimal loss of accuracy.
result Generates interpretable models with minimal loss of accuracy.
New method interprets deep neural networks for better recommendation system understanding.
problem Making deep neural networks explainable for better user trust and understanding.
method Proposes a novel formulation of interpretable deep neural networks using masked weights and hidden features.
result Demonstrates models achieving close predictive performance with informative attributions.
CalArena benchmarks post-hoc calibration methods across various tasks.
problem Inconsistent evaluations of post-hoc calibration methods.
method Large-scale benchmark with 2000 experiments, covering diverse models and settings.
result Smooth calibration functions outperform binning-based approaches.
BICauseTree improves causal effect estimation by identifying clusters and balancing treatment allocation.
problem Improving interpretability and transparency in causal effect models from observational data.
method Hierarchical bias-driven stratification using decision trees with a customized objective function.
result BICauseTree provides interpretable causal effect estimation and is comparable to existing methods.
Meta-Cal improves post-hoc calibration of neural networks.
problem Improving the accuracy of uncalibrated neural network predictions.
method Meta-Cal uses a base calibrator and a ranking model with constraints to provide high-probability bounds.
result Meta-Cal significantly outperforms existing methods in post-hoc multi-class classification calibration.
NIMO combines interpretability with neural networks for clear feature effects.
problem Lack of interpretability in deep learning models.
method Combines neural networks with linear regression using adaptive ridge regression.
result NIMO provides faithful and intelligible feature effects while maintaining good predictive performance.
The adoption of deep learning techniques in genomics has been hindered by the difficulty of mechanistically interpreting the models that these techniques produce. In recent years, a variety of post-hoc attribution methods have been proposed for addressing this neural network interpretability problem in the context of g…
Developed an explainable DRL model for financial portfolio management.
problem Inability of DRL agents to provide interpretable financial investment policies.
method Integrating PPO with feature importance techniques (SHAP, LIME) to enhance transparency.
result Ability to interpret DRL agent actions in prediction time.
A framework uses preprocessing to improve psychiatric questionnaire predictions while maintaining interpretability.
problem Weak predictive accuracy and limited interpretability of psychiatric questionnaires.
method Two-stage method: stable preprocessing followed by a linear mapping.
result REFINE outperforms other interpretable approaches in psychiatric and non-psychiatric prediction tasks.
Study post-hoc Learning to Defer using density-ratio losses.
problem Optimizing decision-making between models and experts.
method Density-ratio losses for post-hoc L2D scorers, derived from class-probability estimation.
result The approach recovers known results and introduces new connections to expert comparison and anomaly detection.
Post-hoc calibration of neural networks using g-Layers proves theoretical justification.
problem Ensuring the confidence of neural network decisions in real-world applications.
method Proves theoretical justification for post-hoc calibration methods by adding g-Layers and minimizing NLL.
result Proves that adding g-Layers and minimizing NLL can lead to a calibrated network.
End-to-end method improves neural network calibration during training.
problem Improving neural network calibration for regression problems.
method Quantile Recalibration Training integrates post-hoc calibration into model training.
result Improved predictive accuracy and calibration in a large-scale experiment.
Survey interprets foundation models' inner workings.
problem Limited explainability of black-box FMs.
method Interpretable methods to reveal FMs' mechanisms.
result Identifies next research directions for FMs.
Systematic review of ML explainability in process mining.
problem Understanding the black-box nature of ML models in process mining.
method Systematic literature review using PRISMA framework.
result Identification of key trends and challenges in interpretability.
New approach interprets machine learning models through feature space transformations.
problem Interpreting models with strongly dependent features in high-dimensional spaces.
method Feature space transformations, including PCA and partial orthogonalization.
result Enhances model interpretation tools for domain experts.