Identifies open problems in quality assurance of safety-critical ML systems.
problem Quality assurance for safety-critical ML systems, especially in automated driving.
method Identified and classified open problems using automated-driving vehicles as an example.
result Open problems require interdisciplinary knowledge from various fields.
Predicts failure of autonomous vehicle steering control models.
problem Evaluating and predicting failure of machine learning models in safety-critical applications.
method Trains a student model to predict the main model's error based on saliency maps.
result Preliminary results show the failure predictor model works on autonomous vehicle steering control systems.
Deep-PrAE improves rare-event simulation for black-box systems.
problem Evaluating rare safety-critical events in learning-based systems.
method Combines deep neural networks with IS to create statistically guaranteed estimations.
result Deep-PrAE provides accurate bounds on safety-critical event probabilities.
Develops a method to simulate rare dangerous events in autonomous systems.
problem Rare dangerous events in safety-critical systems are hard to test in real-world settings.
method Combines exploration, exploitation, and optimization techniques for rare-event simulation.
result Provides rigorous guarantees for the performance of the method.
Tackles bridging machine learning and control theory for safety-critical systems.
problem Ensuring reliability and safety in machine learning applications for safety-critical systems.
method Review of recent advances in learning and control theory, historical context.
result Importance of control theorists joining the conversation on learning-related problems.
Survey of algorithms for testing AI-driven CPS safety.
problem Testing AI-driven CPS for safety in complex environments.
method Survey of applied algorithms for safety validation.
result Survey of existing tools and techniques for safety validation.
Paper proposes a dataset quality process for ML systems.
problem Inadequate standards for ML datasets in safety-critical systems.
method Proposes a dataset specification and verification process.
result Demonstrates the process on a railway signal recognition system.
Paper verifies safety of tree ensembles in safety-critical systems.
problem Ensuring safety of machine learning in safety-critical systems.
method Extract equivalence classes and formally verify input-output mappings of tree ensembles.
result Method is practical for tree ensembles trained on low-dimensional data.
Generates multimodal safety-critical scenarios for robustness evaluation of decision-making algorithms.
problem Lack of comprehensive evaluation of neural network robustness under real-world scenarios.
method Proposes a flow-based multimodal scenario generator using weighted likelihood maximization and gradient-based sampling.
result Demonstrates improved testing efficiency and multimodal modeling capability compared to traditional methods.
Meta-active learning optimizes control of safety-critical systems by efficiently learning dynamics and configurations.
problem Efficiently learning system dynamics and optimal configurations for safety-critical systems like deep brain stimulation.
method Meta-learning an acquisition function using LSTM, cast as meta-learning, with a mixed-integer linear program policy.
result Achieved a 46% increase in information gain and a 20% speedup in computation time over baselines.
DriveFI uses ML to find critical faults in AVs, saving time and resources.
problem Lack of end-to-end fault assessment in AVs under realistic scenarios.
method Machine learning-based fault injection engine (DriveFI) that identifies safety-critical faults.
result Found 561 safety-critical faults in less than 4 hours, compared to weeks of random injection.
Paper reviews robustness in machine learning models and discusses training and certification methods.
problem Ensuring reliability of machine learning models in safety-critical systems.
method Reviews formalisms and discusses training and certification techniques.
result Identifies future research directions in robust machine learning.
ViTaX provides formal guarantees for targeted explanations in safety-critical systems.
problem Need trustworthy explanations for safety-critical deep neural networks.
method Formal reachability analysis for targeted, semifactual explanations.
result First method to provide formally guaranteed explanations of model resilience.
Safe RL for safety-critical tasks using CBFs and model-free RL.
problem Limited safety guarantees in RL for real-world applications.
method Combines model-free RL with CBFs and GP dynamics modeling.
result Demonstrates greater sample efficiency and safety in learning.
The paper tackles MAP inference over non-convex constraints in safety-critical settings.
problem Efficiently computing MAP predictions subject to non-convex constraints is challenging.
method The paper investigates conditions for exact and efficient MAP inference over continuous variables and devises scalable algorithms for both tractable and general cases.
result The proposed methods outperform constraint-agnostic baselines and scale to complex densities.
UAIL uses uncertainty estimation to improve control systems in safety-critical tasks.
problem Improving control systems in safety-critical domains like autonomous driving.
method UAIL applies Monte Carlo Dropout to estimate uncertainty in control output and selectively acquire new training data.
result UAIL can reliably predict infractions and outperforms existing algorithms.
CalNF models rare failures with limited data, improving safety in autonomous systems.
problem Challenges in modeling and debugging rare safety-critical failures due to limited data.
method CalNF, a self-regularized framework for posterior learning from limited data.
result Achieves state-of-the-art performance on data-limited failure modeling and inverse problems.
New metric predicts neural network reliability under novel conditions.
problem Verifying neural networks' safety in novel scenarios.
method ML Dependability metric, Task Undependability, Harmful Undependability.
result Accurately predicts reliability under novel conditions.
CARL safely adapts RL agents for safety-critical tasks.
problem Safety hazards in RL for safety-critical tasks.
method CARL combines model-based RL and cautious adaptation.
result CARL achieves higher rewards with fewer failures in safety-critical tasks.
Paper improves neural network robustness analysis for safety-critical systems.
problem Uncertainty in neural network outputs for safety-critical systems.
method Unified propagation and partition approaches to provide tighter bounds.
result Proposed algorithms give tighter bounds than existing methods for the same computation time.
Quantifier elimination enhances safety assurance of deep neural networks.
problem Rigorously assure safe operation of sophisticated, autonomous systems like DNNs.
method Use quantifier elimination as a formal method to enhance safety assurance.
result Initial results show QE can precisely analyze robustness of DNNs.
Combining feature importance estimates improves reliability of machine learning predictions.
problem Lack of consensus on feature importance quantification makes explanations unreliable.
method Proposes a feature importance fusion framework combining multiple quantifiers.
result Feature importance ensembles reduce prediction error by 15%.
RL algorithm uses LTL to specify goals for MDPs, ensuring policy satisfaction.
problem Ensuring safety-critical RL policies meet specified goals formally.
method Formulates goals using LTL, translates to LDGBA, shapes synchronous reward function.
result Algorithm synthesizes policies satisfying LTL goals with maximal probability.
New algorithm constructs PAC prediction sets for meta-learning.
problem Uncertainty quantification in safety-critical systems.
method Proposes a novel algorithm to construct PAC prediction sets.
result Prediction sets satisfy a PAC guarantee with high probability over future tasks.
Paper proposes a method to test and verify control systems with machine learning components.
problem Testing and verifying control systems with machine learning components is challenging.
method Gradient-based method combined with randomized search to find adversarial samples.
result Method outperforms Simulated Annealing optimization in finding adversarial samples.
Safe Active Learning (SAL) for GPODE models improves model performance without compromising safety.
problem Efficient and safe data collection for GPODE models.
method Sequential data suggestion constrained by safety, maximizing information gain.
result SAL GPODE outperforms standard data collection methods.
Enhances monitoring of industrial systems with limited data.
problem Early detection of rare faults in safety-critical systems with limited training data.
method Feature alignment techniques (variational encoder, adversarial training) to align features from different units.
result Feature alignment improves robustness of one-class classifier for health monitoring.
The paper evaluates and improves uncertainty estimates in neural networks for safety-critical applications.
problem Quantifying uncertainty in neural networks for safety-critical systems.
method Proposes a statistical test for evaluating uncertainty realism in neural networks and transfers a classification architecture to image-to-image tasks.
result The variational U-Net architecture significantly improves uncertainty realism in image-to-image tasks compared to a plain model.
ARTEO algorithm optimizes safety-critical systems with uncertainty.
problem Decision-making under uncertainty with safety constraints in real-time optimization.
method ARTEO algorithm uses multi-armed bandits as a mathematical programming problem subject to safety constraints, learning unknown characteristics through exploration and incorporating uncertainty quantification.
result ARTEO achieves less cumulative regret with accurate and safe decisions.
New technique reduces verification time for neural networks.
problem Verifying neural networks for safety-critical applications.
method Shadow prices for more efficient input partitioning.
result Significant reduction in computation times for verification.
Paper proposes a method to predict deep neural network confidences with guarantees.
problem Quantifying uncertainty in deep neural networks for safety-critical applications.
method Uses Clopper-Pearson confidence intervals and histogram binning for calibrated prediction.
result Demonstrates the effectiveness of predicted confidences in improving DNN performance and safety.
Survey of methods for ML assurance across its lifecycle stages.
problem Ensuring safety of ML in safety-critical applications.
method Comprehensive survey of methods at each ML lifecycle stage.
result Identification of open challenges in ML assurance.
We develop a new method to estimate failure probabilities in complex systems.
problem Estimating failure probabilities in safety-critical autonomous systems is challenging due to the rarity of failures and large state spaces.
method We propose an adaptive importance sampling algorithm that minimizes forward Kullback-Leibler divergence and uses Markov score ascent methods.
result Our method provides more accurate failure probability estimates than existing techniques.
Proposes neural delay differential equations for stable system identification with partially observed states.
problem Learning stable models for systems with partial or delayed observations.
method Augments states with history, uses neural delay differential equations, and ensures stability through time delay analysis.
result The approach ensures stability of learned models for partially observed systems.
KCRL learns stable policies for nonlinear systems with formal guarantees.
problem Lack of stabilization guarantees in RL methods for safety-critical systems.
method KCRL uses Krasovskii's Lyapunov functions as a stability constraint and a primal-dual approach to learn stabilizing policies.
result KCRL guarantees learning a stabilizing policy in a finite number of interactions.
BagCert efficiently certifies robustness against adversarial patches on image classifiers.
problem Adversarial patches pose a threat to autonomous systems' perception component.
method BagCert combines model architecture and certification procedure for efficient inference.
result BagCert certifies 10,000 examples in 43 seconds on a single GPU, achieving 86% clean and 60% certified accuracy against 5x5 patches.
A framework identifies worst-case decision points in safety-critical scenarios, improving risk assessment by 10 hours.
problem Identifying worst-case outcomes in safety-critical decision-making under uncertainty.
method Explicitly estimating distributions of expected return to identify dead-ends, tuning based on risk tolerance.
result Significantly improves risk assessment, providing indications 10 hours earlier and increasing detection by 20%.
SGPA calibrates transformer uncertainty for safety-critical tasks.
problem Uncertainty estimation in transformer models for safety-critical domains.
method Bayesian inference in transformer's output space using sparse Gaussian processes.
result SGPA-based Transformers improve in-distribution calibration and out-of-distribution robustness.
Paper aims to ensure reliable detection of out-of-distribution data with certifiable worst-case guarantees.
problem Deep neural networks are overconfident with OOD inputs, posing safety risks.
method Enforces low confidence and bounds in an l∞-ball around OOD points using interval bound propagation (IBP). result Certifiable worst-case guarantees for OOD detection are possible without significant loss in accuracy.
Paper proposes method for optimal control of unknown systems with latent states.
problem Jointly estimating dynamics and latent states in systems with unmeasurable states.
method Combination of particle Markov chain Monte Carlo methods and scenario theory.
result Probabilistic performance guarantees for optimal input trajectories.
Proposes novel method for detecting novel scenarios in autonomous systems.
problem Detecting when a machine learning model makes a trustworthy prediction in dynamic, real-world situations.
method Leverages trained model's learned information and a new image similarity metric.
result Demonstrates the method's efficacy on real-world driving and indoor racing datasets.
Semantic embeddings improve safety-critical classifier performance.
problem Improving interpretability and error detection in safety-critical neural networks.
method Created embeddings from symbolic domain knowledge, used for misprediction interpretation and error detection, introduced semantic distance for confidence measurement.
result Semantic distance achieves near state-of-the-art performance in a traffic sign classifier, faster than other methods.
VerifAI toolkit improves neural network-based aircraft taxiing system safety.
problem Improving safety of autonomous aircraft taxiing systems using neural networks.
method Unified approach to formal analysis and retraining of AI systems, including falsification, debugging, and retraining.
result Improved neural network performance and reduced failure cases in aircraft taxiing system.
This paper compares uncertainty estimation methods for deep learning in autonomous vehicles.
problem Ensuring safety in autonomous vehicles through accurate uncertainty quantification in deep learning models.
method A comparative survey of uncertainty quantification methods in deep neural networks.
result Different methods for uncertainty quantification in DNNs have advantages and downsides for specific AV tasks and types of uncertainty.
Safe reinforcement learning for autonomous vehicles using prediction constraints.
problem Safe reinforcement learning for safety-critical applications like autonomous vehicles.
method Use prediction to constrain exploration in reinforcement learning models.
result Successfully learned intersection handling behaviors on an autonomous vehicle.
Tree ensemble method tackles multi-objective constrained optimization in energy systems.
problem Complex, multi-objective, and constrained optimization problems in energy systems.
method Data-driven tree ensemble approach for black-box problems with heterogeneous variable spaces.
result Competitive performance and sampling efficiency compared to state-of-the-art tools.
Safe learning of stochastic dynamics with safety constraints.
problem Learning controlled stochastic dynamics with safety constraints.
method Iterative expansion of a safe control set using kernel-based confidence bounds.
result The method ensures safe exploration and efficient estimation of system dynamics.
Research tackles safety of deep learning in safety-critical tasks.
problem Safety concerns of deep learning in perception tasks for autonomous agents.
method Technical enumeration and discussions on safety concerns and mitigation methods.
result Need for more mitigation methods to ensure safety of deep learning.