New approach checks neural networks for safety properties efficiently.
problem Ensuring neural networks are robust to adversarial and accidental perturbations.
method Efficient formal safety analysis of neural networks using tight output bounds.
result Significantly outperforms existing techniques for larger networks.
ViTaX provides formal guarantees for targeted explanations in safety-critical systems.
problem Need trustworthy explanations for safety-critical deep neural networks.
method Formal reachability analysis for targeted, semifactual explanations.
result First method to provide formally guaranteed explanations of model resilience.
This paper formalizes AI safety using hypothesis testing in GenAI.
problem Ensuring safety of generative AI tools that create realistic content.
method Formalization of computational safety through hypothesis testing and signal processing.
result Demonstrates how AI safety can be assessed quantitatively using mathematical frameworks.
Quantifier elimination enhances safety assurance of deep neural networks.
problem Rigorously assure safe operation of sophisticated, autonomous systems like DNNs.
method Use quantifier elimination as a formal method to enhance safety assurance.
result Initial results show QE can precisely analyze robustness of DNNs.
Formal constraints improve RL safety in complex environments.
problem Safety constraints in reinforcement learning for complex environments.
method Specify constraints in formal languages, instantiate as finite automata, augment MDP states, learn dense cost function.
result Improved safety in training RL algorithms over various constraints.
This paper explores formal verification for autonomous systems, identifying limitations and proposing improvements.
problem Ensuring safety of autonomous systems like self-driving cars and drones.
method Formal verification techniques based on formal methods, analyzing three assumptions and their limitations.
result Preliminary work to improve the strength of evidence provided by formal verification.
VerifAI toolkit improves neural network-based aircraft taxiing system safety.
problem Improving safety of autonomous aircraft taxiing systems using neural networks.
method Unified approach to formal analysis and retraining of AI systems, including falsification, debugging, and retraining.
result Improved neural network performance and reduced failure cases in aircraft taxiing system.
ART trains neural nets to be both accurate and safe.
problem Ensuring neural nets are both accurate and safe during training.
method Integrates an optimization-based abstraction refinement loop into the learning process.
result Enables training provably correct networks with respect to safety properties.
Paper verifies safety of tree ensembles in safety-critical systems.
problem Ensuring safety of machine learning in safety-critical systems.
method Extract equivalence classes and formally verify input-output mappings of tree ensembles.
result Method is practical for tree ensembles trained on low-dimensional data.
New method provides scalable safety guarantees for RL agents.
problem Safe reinforcement learning in real-life scenarios.
method State-augmentation and shield design for probabilistic avoidance.
result Strict formal safety guarantees for RL agents, scalable and practical.
New framework formalizes RLHF trilemma: improving safety, fairness, and robustness is computationally infeasible.
problem Aligning large language models with diverse human values while maintaining computational feasibility and robustness.
method Complexity-theoretic analysis integrating statistical learning theory and robust optimization.
result Achieving both representativeness (epsilon <= 0.01) and robustness (delta <= 0.001) for global-scale populations requires super-polynomial operations.
PEREGRiNN verifies safety of ReLU NNs by penalizing relaxation in a greedy manner.
problem Formal verification of safety specifications for ReLU NNs.
method Uses a relaxed convex program to verify polytopic input/output constraints, penalizing relaxation and forcing largest relaxations to early layers.
result Significantly faster and more properties verified compared to other approaches.
GP3 framework efficiently analyzes Gaussian processes on GPUs.
problem Certifiable safety in machine learning applications.
method GP3 framework using interval analysis and multi-resolution sampling on GPUs.
result Efficient analysis of Gaussian processes with certifiable safety.
Survey of algorithms for testing AI-driven CPS safety.
problem Testing AI-driven CPS for safety in complex environments.
method Survey of applied algorithms for safety validation.
result Survey of existing tools and techniques for safety validation.
Machine learning algorithms increasingly influence our decisions and interact with us in all parts of our daily lives. Therefore, just as we consider the safety of power plants, highways, and a variety of other engineered socio-technical systems, we must also take into account the safety of systems involving machine le…
Safe imitation learning with a safety layer for flexible training.
problem Flexible yet safe imitation learning for complex tasks.
method Theory and modular method with a safety layer for continuous policy, adversarial training, and worst-case safety guarantees.
result Robustness advantage of safety layer during training compared to test time.
Recent work has shown that state-of-the-art classifiers are quite brittle, in the sense that a small adversarial change of an originally with high confidence correctly classified input leads to a wrong classification again with high confidence. This raises concerns that such classifiers are vulnerable to attacks and ca…
Paper improves neural network robustness analysis for safety-critical systems.
problem Uncertainty in neural network outputs for safety-critical systems.
method Unified propagation and partition approaches to provide tighter bounds.
result Proposed algorithms give tighter bounds than existing methods for the same computation time.
RL algorithm uses LTL to specify goals for MDPs, ensuring policy satisfaction.
problem Ensuring safety-critical RL policies meet specified goals formally.
method Formulates goals using LTL, translates to LDGBA, shapes synchronous reward function.
result Algorithm synthesizes policies satisfying LTL goals with maximal probability.
Project analyzes traffic videos to improve Jakarta's safety.
problem Improving traffic safety in Jakarta.
method Developed a pipeline to analyze traffic videos, turning them into usable databases.
result Better understanding of traffic challenges and safety risks.
Extends neural net safety guarantees by proving structural properties.
problem Proving formal guarantees for complex DNN architectures.
method Proves structural properties related to neural net structure to infer safety properties.
result Identifies a larger region of input space for safety properties.
New framework verifies reinforcement learning systems without altering neural networks.
problem Lack of assurance guarantees in reinforcement learning applications.
method Repurposes formal verification techniques for reinforcement learning, synthesizing simpler programs that preserve safety.
result Synthesized programs ensure safety of reinforcement learning systems without modifying neural networks.
In recent years, car makers and tech companies have been racing towards self driving cars. It seems that the main parameter in this race is who will have the first car on the road. The goal of this paper is to add to the equation two additional crucial parameters. The first is standardization of safety assurance --- wh…
Framework for safely updating machine learning models.
problem Continuous updates to machine learning models can lead to unintended consequences.
method Formalizes the problem as computing the largest locally invariant domain (LID), uses tractable primal-dual formulation.
result Matches or exceeds heuristic baselines for avoiding forgetting while providing formal safety guarantees.
Toolkit improves neural network safety for self-driving cars.
problem Ensuring safety of neural networks in autonomous driving systems.
method Structured approach using Goal Structuring Notation and recent scientific results.
result Improves quality of a level-3 autonomous driving component.
A new RL model ensures safe learning in uncertain environments.
problem Safe reinforcement learning in uncertain, partially observable environments.
method Lyapunov-based uncertainty quantification and Transformers for memory.
result Significant improvement in safety and optimality in grid-world tasks.
RLVR maintains safety while improving reasoning capabilities in LLMs.
problem Safety-capability tradeoff in fine-tuning LLMs.
method Reinforcement Learning with Verifiable Rewards (RLVR) and theoretical analysis.
result RLVR can enhance reasoning while maintaining safety guardrails.
Develops first robustness verification for complex Transformers.
problem Certify prediction behavior of Transformers with complex self-attention layers.
method Resolves challenges of cross-nonlinearity and cross-position dependency in Transformers.
result Certified robustness bounds are significantly tighter than Interval Bound Propagation.
This work provides safety guarantees for iterative GP predictions.
problem Analytical intractability of uncertainty tracking in iterative GP predictions.
method Deriving formal probability error bounds for iterative GP predictions.
result Formal bounds ensure that GP trajectories lie within specified regions with high probability.
The paper proves neural networks are almost always surjective, impacting model safety.
problem Ensuring neural networks can generate any output, including harmful content.
method Analyzing fundamental neural architectures and generative models.
result Many neural architectures are almost always surjective, allowing for arbitrary outputs.
Although aviation accidents are rare, safety incidents occur more frequently and require a careful analysis to detect and mitigate risks in a timely manner. Analyzing safety incidents using operational data and producing event-based explanations is invaluable to airline companies as well as to governing organizations s…
This paper formalizes Uniswap v3 using PTA and FST for rigorous analysis.
problem Formal modeling of Uniswap v3's concentrated liquidity for rigorous analysis.
method Formal state machine models using PTA and FST, proving rounding bounds.
result Formal justification of Uniswap v3's ε-slack and rounding safety. Theoretical limits on verifying self-improving systems without risking unbounded utility.
problem Formalizing and proving the limits of safety verification for self-improving systems.
method Developed dual conditions and used Holder's inequality, NP counting method, and Lipschitz bounds to establish impossibility and ceiling results.
result A classifier-based safety gate cannot simultaneously permit unbounded beneficial self-modification and bounded cumulative risk.
Paper reviews robustness in machine learning models and discusses training and certification methods.
problem Ensuring reliability of machine learning models in safety-critical systems.
method Reviews formalisms and discusses training and certification techniques.
result Identifies future research directions in robust machine learning.
Paper presents LSTMMDN for hourly bike flow estimation in Copenhagen.
problem Sparse or unavailable hourly bike flow data for safety analysis.
method Hybrid LSTM MDN model for hourly bike flow estimation.
result 66-77% more accurate bike flow estimates compared to calibration factors.
FANNet analyzes noise tolerance and training bias in neural networks.
problem Low noise tolerance and input sensitivity in neural networks lead to failures on unseen inputs.
method Formal analysis using model checking under different noise ranges.
result Noise tolerance of ±11% for the trained network, sensitive input nodes identified, and biasness confirmed. LPF provides formal guarantees for aggregating multi-evidence in probabilistic tasks.
problem Lack of formal guarantees for multi-evidence reasoning in AI.
method LPF uses variational autoencoders and Sum-Product Networks to aggregate evidence items.
result Proves multiple formal guarantees including calibration preservation and error decay.
A novel approach for safe offline RL using latent safety constraints.
problem Balancing safety constraints and reward maximization in offline RL.
method Conditional Variational Autoencoders for latent safety modeling, Constrained Reward-Return Maximization.
result Our approach maintains safety compliance while optimizing rewards, outperforming existing methods.
Real time large scale streaming data pose major challenges to forecasting, in particular defying the presence of human experts to perform the corresponding analysis. We present here a class of models and methods used to develop an automated, scalable and versatile system for large scale forecasting oriented towards saf…
The variability of the clusters generated by clustering techniques in the domain of latitude and longitude variables of fatal crash data are significantly unpredictable. This unpredictability, caused by the randomness of fatal crash incidents, reduces the accuracy of crash frequency (i.e., counts of fatal crashes per c…
Develops a method to simulate rare dangerous events in autonomous systems.
problem Rare dangerous events in safety-critical systems are hard to test in real-world settings.
method Combines exploration, exploitation, and optimization techniques for rare-event simulation.
result Provides rigorous guarantees for the performance of the method.
Logarithmic regret strategies for safe multi-armed bandits with safety risk constraints.
problem Maximizing reward while avoiding unsafe arms under safety risk constraints.
method Doubly optimistic strategies with pseudo-regret formulation.
result Logarithmic regret bounds for safe multi-armed bandits.
Safe Bayesian Optimization algorithms are improved to ensure safety in real-world applications.
problem Ensuring safety in Bayesian Optimization algorithms for real-world applications.
method Investigated and improved three safety-related issues of SafeOpt-type algorithms: frequentist uncertainty bounds, RKHS norm assumptions, and discrete search spaces.
result Introduced Real-{eta}-SafeOpt, Lipschitz-only Safe Bayesian Optimization (LoSBO), and Lipschitz-only GP-UCB (LoS-GP-UCB) algorithms that retain safety guarantees and superior performance.
By building on a recently introduced genetic-inspired attribute-based conceptual framework for safety risk analysis, we propose a novel methodology to compute construction univariate and bivariate construction safety risk at a situational level. Our fully data-driven approach provides construction practitioners and aca…
Safe algorithm for linear bandits with safety constraints, matching previous results.
problem Designing safe bandit algorithms with linear safety constraints.
method Linear Thompson Sampling with frequentist regret analysis.
result Frequentist regret of order O(d3/2log1/2d⋅T1/2log3/2T). New Transformers maintain Lipschitz continuity for robustness.
problem Ensuring robustness in Transformers for safety-sensitive applications.
method Introducing gradient-descent-type in-context Transformers with explicit Euler steps of negative gradient flows.
result Universal approximation theorem for Lipschitz continuous Transformers.
Synthesizes machine learning applications in reliability and safety.
problem Navigating the fragmented literature on ML for reliability and safety.
method Overview of ML categories, review of applications, discussion of Deep Learning.
result Machine learning can provide novel insights and improve accident prevention.
Survey on biases in image analysis for industrial safety.
problem Bias in machine learning algorithms affects industrial safety-critical applications.
method Survey and analysis of recent advances in bias detection and mitigation.
result Need for new methods to detect and mitigate biases in image analysis for safety-critical applications.