SED integrates synthesis, execution, and debugging for neural program synthesis.
problem Challenges in synthesizing complex programs that match specifications.
method SED combines synthesis, execution, and debugging to improve neural program generation.
result SED reduces error rates and outperforms standard decoding methods.
Cockpit provides tools for debugging deep neural networks during training.
problem Limited real-time training diagnostics for deep learning models.
method Cockpit uses novel higher-order information about gradient distribution and curvature.
result Facilitates identification of learning phases and failure modes.
Survey of methods to visualize neural network features.
problem Understanding neural network activation patterns.
method Activation Maximization and Feature Visualization via Optimization.
result Probabilistic interpretation of AM techniques.
Activates speech DNNs to generate understandable examples.
problem Difficulty in understanding DNN classifications for speech.
method Activation maximization to generate speech samples.
result Activation maximization can generate understandable speech samples.
Proposes a method for neural networks to learn causal relationships and humans to contest and modify them.
problem Neural networks learn relevant causal relationships unclearly and are black-box, making them hard to debug.
method Two-way interaction between neural networks and humans, allowing contestation and modification of causal graphs.
result Improves predictive performance up to 2.4x and produces smaller networks up to 7x compared to SOTA.
Machine learning models are notoriously difficult to interpret and debug. This is particularly true of neural networks. In this work, we introduce automated software testing techniques for neural networks that are well-suited to discovering errors which occur only for rare inputs. Specifically, we develop coverage-guid…
Interactive tool helps RL researchers debug and understand their models.
problem Challenges in debugging reinforcement learning algorithms.
method Design and implementation of an interactive visualization tool.
result Addresses features missing from previous tools for RL.
3DB framework tests and debugs computer vision models using photorealistic simulation.
problem Discovering vulnerabilities and understanding model decision-making in computer vision systems.
method Unified framework using photorealistic simulation to test and debug vision models.
result Insights generated by 3DB transfer to the physical world, enabling robustness analysis.
TS-Insight visualizes Thompson Sampling for better debugging and trust.
problem Thompson Sampling's black box nature hinders debugging and trust.
method TS-Insight is a visual analytics tool that traces evolving posteriors and evidence counts.
result Visualizations help in verifying, diagnosing, and explaining Thompson Sampling dynamics.
VerifAI toolkit improves neural network-based aircraft taxiing system safety.
problem Improving safety of autonomous aircraft taxiing systems using neural networks.
method Unified approach to formal analysis and retraining of AI systems, including falsification, debugging, and retraining.
result Improved neural network performance and reduced failure cases in aircraft taxiing system.
Unlike traditional programs (such as operating systems or word processors) which have large amounts of code, machine learning tasks use programs with relatively small amounts of code (written in machine learning libraries), but voluminous amounts of data. Just like developers of traditional programs debug errors in the…
Sparse linear models improve neural network debuggability.
problem Improving neural network interpretability and debugging.
method Using sparse linear models over learned deep feature representations.
result The approach leads to more debuggable and accurate neural networks.
The paper develops methods to identify and correct buggy data in linear regression models.
problem Identifying and correcting buggy data in linear regression models.
method Formulated a general statistical algorithm for identifying buggy points, provided theoretical guarantees, and proposed an algorithm for tuning parameter selection.
result Theoretical guarantees and empirical results show the effectiveness of the proposed debugging algorithm.
MDP Playground tests RL agents across various dimensions for better understanding and debugging.
problem Understanding and debugging reinforcement learning agents across diverse environments and dimensions.
method Controlled testbed with adjustable dimensions for different RL challenges.
result Insights into agent performance and interaction with various dimensions.
Generative models help debug data issues in private, decentralized datasets.
problem Manual data inspection is impractical for private, decentralized datasets.
method Differentially private federated generative models.
result Generative models can effectively debug data issues in private datasets.
Paper proposes counterfactual explanations for ML on multivariate time series data.
problem Lack of user trust and difficulty in debugging ML frameworks using multivariate time series data.
method Proposes a novel explainability technique for providing counterfactual explanations.
result Outperforms state-of-the-art explainability methods in metrics like faithfulness and robustness.
We propose learning flexible but interpretable functions that aggregate a variable-length set of permutation-invariant feature vectors to predict a label. We use a deep lattice network model so we can architect the model structure to enhance interpretability, and add monotonicity constraints between inputs-and-outputs.…
PrototypeML simplifies neural network design and development.
problem Error-prone code and time-consuming model design.
method Visual interface for neural networks, abstracting PyTorch.
result Reduces model design and development time, easier debugging.
The ability to interpret machine learning models has become increasingly important now that machine learning is used to inform consequential decisions. We propose an approach called model extraction for interpreting complex, blackbox models. Our approach approximates the complex model using a much more interpretable mo…
Despite their increasing popularity and success in a variety of supervised learning problems, deep neural networks are extremely hard to interpret and debug: Given and already trained Deep Neural Net, and a set of test inputs, how can we gain insight into how those inputs interact with different layers of the neural ne…
New approach identifies and explains errors in machine learning pipelines.
problem Challenges in identifying and explaining errors in complex machine learning pipelines.
method Uses iteration and provenance to automatically infer root causes of failures.
result Significantly improves precision and recall compared to state-of-the-art methods.
Bayesian approach calibrates DNN confidence for field use.
problem DNN models give false predictions with high confidence in real-world applications.
method Bayesian approach using Gaussian Process Regression to correct confidence with minimal labeled operation data.
result Significantly reduces high-confidence errors with minimal labeled data.
Neural model predicts procedure names in stripped binaries.
problem Reverse engineering stripped executables with limited debug information.
method Combines static analysis with neural models to predict procedure names.
result Improves prediction accuracy by 28% and 100% over state-of-the-art models.
Bayes-TrEx finds in-distribution examples for model inspection.
problem Challenges in interpreting neural networks, especially high-confidence failures and ambiguous classifications.
method Bayesian sampling approach to find in-distribution examples with specified prediction confidence.
result Bayes-TrEx enables more flexible holistic model analysis than just inspecting the test set.
How can we explain the predictions of a black-box model? In this paper, we use influence functions -- a classic technique from robust statistics -- to trace a model's prediction through the learning algorithm and back to its training data, thereby identifying training points most responsible for a given prediction. To …
COSET benchmarks neural program embeddings using diverse source-code datasets.
problem Evaluating neural program embeddings is challenging due to lack of straightforward metrics.
method COSET framework with labeled programs, transformations, and a pilot study.
result COSET identifies strengths and weaknesses of neural models and program characteristics.
The paper analyzes methods to identify influential data points in deep models.
problem Interpreting deep learning models and debugging datasets.
method Curated experiments to analyze influence of data points on classifiers.
result Training loss-based sample selection outperformed other methods in detecting mislabels.
We introduce a new algorithm named WGAN, an alternative to traditional GAN training. In this new model, we show that we can improve the stability of learning, get rid of problems like mode collapse, and provide meaningful learning curves useful for debugging and hyperparameter searches. Furthermore, we show that the co…
Deep reinforcement learning has led to several recent breakthroughs, though the learned policies are often based on black-box neural networks. This makes them difficult to interpret and to impose desired specification constraints during learning. We present an iterative framework, MORL, for improving the learned polici…
FinRL simplifies deep RL for stock trading, making it accessible to beginners.
problem Lack of accessible tools for beginners in deep RL for stock trading.
method Developed a DRL library with reproducible tutorials and backtesting.
result FinRL streamlines development and comparison of trading strategies.
CB-SLICE identifies concept-based error slices in deep learning models.
problem Systematic errors in deep learning models on specific groups.
method Concept Bottleneck Models (CBMs) and concept representations.
result CB-SLICE outperforms state-of-the-art methods in error slice identification.
Autonomous AI systems will be entering human society in the near future to provide services and work alongside humans. For those systems to be accepted and trusted, the users should be able to understand the reasoning process of the system, i.e. the system should be transparent. System transparency enables humans to fo…
Research proposes a test case generation system for deep learning models using dataset properties.
problem Automated generation of extensive test cases for deep learning models is challenging.
method Measures dataset quality and proposes a test case generation system guided by dataset properties.
result Systematic test case generation for deep learning models is effective.
Net2Vis automates CNN visualization for publications.
problem Lack of consistent visual representations in deep learning papers.
method Proposes a visual grammar and automated system for generating publication-ready CNN visualizations.
result Reduces time and ambiguity in generating network visualizations.
Saliency methods have emerged as a popular tool to highlight features in an input deemed relevant for the prediction of a learned model. Several saliency methods have been proposed, often guided by visual appeal on image data. In this work, we propose an actionable methodology to evaluate what kinds of explanations a g…
Markov Chain Monte Carlo (MCMC) algorithms are a workhorse of probabilistic modeling and inference, but are difficult to debug, and are prone to silent failure if implemented naively. We outline several strategies for testing the correctness of MCMC algorithms. Specifically, we advocate writing code in a modular way, w…
Interpretation of a machine learning induced models is critical for feature engineering, debugging, and, arguably, compliance. Yet, best of breed machine learning models tend to be very complex. This paper presents a method for model interpretation which has the main benefit that the simple interpretations it provides …
FinRL automates trading in quantitative finance with deep reinforcement learning.
problem Steep development curve for traders to automate trading decisions.
method Open-source framework implementing DRL algorithms and reward functions.
result FinRL simplifies strategy design and reduces debugging workloads.
PCBMs turn any neural network into interpretable models without dense annotations.
problem Restrictive nature of CBMs and lack of dense concept annotations in training data.
method Introduce PCBMs that can turn any neural network into interpretable models without dense annotations.
result PCBMs can turn any neural network into interpretable models without dense annotations, improving interpretability and performance.
Deep learning has recently seen rapid development and received significant attention due to its state-of-the-art performance on previously-thought hard problems. However, because of the internal complexity and nonlinear structure of deep neural networks, the underlying decision making processes for why these models are…
This paper proposes a framework to learn explainable rules from knowledge graphs for better recommendation.
problem Combining side information with explainability in recommendation systems.
method Joint learning framework integrating rule induction from knowledge graphs with a rule-guided neural recommendation model.
result Significant improvements in item recommendation performance over baselines.
Tree ensembles such as random forests and boosted trees are accurate but difficult to understand, debug and deploy. In this work, we provide the inTrees (interpretable trees) framework that extracts, measures, prunes and selects rules from a tree ensemble, and calculates frequent variable interactions. An rule-based le…
Captum library simplifies model interpretability for PyTorch.
problem Improving model interpretability for various PyTorch models.
method Unified, open-source library with generic attribution algorithms and metrics.
result Unified and scalable model interpretability for multimodal inputs.
Automatic differentiation (AD) is an essential primitive for machine learning programming systems. Tangent is a new library that performs AD using source code transformation (SCT) in Python. It takes numeric functions written in a syntactic subset of Python and NumPy as input, and generates new Python functions which c…
TREX explains tree ensembles by identifying key training examples.
problem Identifying which training examples most influence tree ensemble predictions.
method TREX builds a surrogate model using a kernel that captures tree ensemble structure, approximating the original model.
result TREX provides accurate and effective explanations for tree ensembles.
Study robustness of global feature effect explanations in machine learning models.
problem Vulnerability of global feature effect explanations to data and model perturbations.
method Theoretical bounds and experimental evaluation of partial dependence plots and accumulated local effects.
result Quantifies the gap between best and worst-case scenarios of misinterpreting machine learning predictions globally.
Interpretation and diagnosis of machine learning models have gained renewed interest in recent years with breakthroughs in new approaches. We present Manifold, a framework that utilizes visual analysis techniques to support interpretation, debugging, and comparison of machine learning models in a more transparent and i…
We use flip points to explain and audit deep learning models, revealing decision boundaries and improving model performance.
problem Lack of interpretability in deep learning models hinders their use in important applications.
method Flip points are used to analyze decision boundaries of deep learning models with continuous output scores.
result Flip points reveal the least changes in input that would alter a model's classification, enabling better understanding and improvement of model behavior.