RLVR maintains safety while improving reasoning capabilities in LLMs.
problem Safety-capability tradeoff in fine-tuning LLMs.
method Reinforcement Learning with Verifiable Rewards (RLVR) and theoretical analysis.
result RLVR can enhance reasoning while maintaining safety guardrails.
The paper reviews machine learning safety techniques for autonomous vehicles.
problem Challenges in machine learning safety for autonomous vehicles.
method Organizes practical safety techniques to complement engineering safety.
result Enhances dependability and safety of machine learning algorithms in autonomous vehicles.
Research tackles safety of deep learning in safety-critical tasks.
problem Safety concerns of deep learning in perception tasks for autonomous agents.
method Technical enumeration and discussions on safety concerns and mitigation methods.
result Need for more mitigation methods to ensure safety of deep learning.
The paper discusses safety assessment for AI systems, focusing on machine learning models.
problem Safety assessment of AI systems, especially machine learning models, in safety-related applications.
method Analyzed AI models as statistical models and proposed a new budget allocation for AI safety.
result Safety assessment of AI systems requires a new approach focusing on the model used, not just the system.
Fine-tuning LLMs improves capability but harms safety, study finds.
problem Balancing capability and safety in LLM fine-tuning.
method Theoretical framework and numerical experiments for two safety-aware fine-tuning strategies.
result Characterization of fundamental limits of safety-capability trade-off in LLM fine-tuning.
New safety measure reduces failures in learning systems.
problem Challenges in learning to control physical systems due to failures.
method Implicitly captures system dynamics to failure states using Gaussian processes for model-free active sampling.
result Significant reduction in failures during learning with estimated safety measure.
RAGuard improves safety in LLMs for offshore wind maintenance.
problem Conventional LLMs fail with specialised or unexpected scenarios in offshore wind maintenance.
method Integrates safety-critical documents alongside technical manuals in RAG framework.
result RAGuard increases safety recall from almost 0% to over 50% while maintaining technical recall above 60%.
Survey of algorithms for testing AI-driven CPS safety.
problem Testing AI-driven CPS for safety in complex environments.
method Survey of applied algorithms for safety validation.
result Survey of existing tools and techniques for safety validation.
Safe imitation learning with a safety layer for flexible training.
problem Flexible yet safe imitation learning for complex tasks.
method Theory and modular method with a safety layer for continuous policy, adversarial training, and worst-case safety guarantees.
result Robustness advantage of safety layer during training compared to test time.
This paper formalizes AI safety using hypothesis testing in GenAI.
problem Ensuring safety of generative AI tools that create realistic content.
method Formalization of computational safety through hypothesis testing and signal processing.
result Demonstrates how AI safety can be assessed quantitatively using mathematical frameworks.
A new algorithm for identifying the best arm in linear feedback with safety constraints.
problem Identifying the best arm in linear feedback with safety constraints.
method A gap-based algorithm that ensures safety while minimizing sample complexity.
result The algorithm achieves meaningful sample complexity while ensuring safety.
Safety filter for unknown discrete-time systems with learned models and noise covariance.
problem Ensuring safety for unknown discrete-time linear systems with Gaussian noise.
method Develops a learning-based safety filter using empirical model and noise covariance, optimizing control actions to stay within safety constraints.
result Minimally modifies nominal control actions to ensure safety with high probability, tightening constraints as more data is collected.
SafeML monitors ML systems for safety and security risks.
problem Ensuring safety and explainability of ML systems in safety-critical domains.
method Statistical difference measures of ECDF to detect distributional shifts.
result Approach can detect invalid application contexts of ML components.
Paper defines and quantifies safety risks in deep neural networks.
problem Safety concerns in deep neural networks applied to critical sectors.
method Defines safety property, computes maximum safe radius, identifies new risk class, develops algorithm.
result Method achieves competitive performance in safety quantification.
Study best arm identification with safety constraints in bandit problems.
problem Real-world decision-making with safety constraints.
method Analyzed linear and monotonic reward and safety constraints, proposed algorithms.
result Guaranteed safe learning in both linear and general reward/safety constraint settings.
Safe reinforcement learning tackles safety constraints with linear approximations.
problem Ensuring safety in reinforcement learning without violating constraints.
method Modeling safety as a linear cost function, developing SLUCB-QVI and RSLUCB-QVI algorithms for MDPs with linear function approximation.
result Achieved a nearly optimal regret bound for safe reinforcement learning, matching state-of-the-art unsafe algorithms.
Enhances safety of 3D object detection neural networks.
problem Ensuring robustness and safety of 3D object detection systems.
method Symbolic error propagation, specialized loss function, safety-aware non-max-inclusion algorithm.
result Improved safety and robustness of 3D object detection neural networks.
Each year, around 6 million car accidents occur in the U.S. on average. Road safety features (e.g., concrete barriers, metal crash barriers, rumble strips) play an important role in preventing or mitigating vehicle crashes. Accurate maps of road safety features is an important component of safety management systems for…
CARL safely adapts RL agents for safety-critical tasks.
problem Safety hazards in RL for safety-critical tasks.
method CARL combines model-based RL and cautious adaptation.
result CARL achieves higher rewards with fewer failures in safety-critical tasks.
A novel approach for safe offline RL using latent safety constraints.
problem Balancing safety constraints and reward maximization in offline RL.
method Conditional Variational Autoencoders for latent safety modeling, Constrained Reward-Return Maximization.
result Our approach maintains safety compliance while optimizing rewards, outperforming existing methods.
New method improves safety analytics by addressing imbalanced data issues.
problem Imbalanced safety datasets lead to inaccurate predictions and management problems.
method Extended accident triangle theory and three oversampling methods.
result Robust improvements in machine learning algorithms for safety analytics.
LLM safety alignment explained as divergence estimation.
problem Aligning large language models to avoid harmful outputs.
method Presented a theoretical framework showing alignment methods as divergence estimators.
result KLDO method improves safety alignment using compliance-refusal datasets.
Machine learning algorithms increasingly influence our decisions and interact with us in all parts of our daily lives. Therefore, just as we consider the safety of power plants, highways, and a variety of other engineered socio-technical systems, we must also take into account the safety of systems involving machine le…
Safe Bayesian Optimization algorithms are improved to ensure safety in real-world applications.
problem Ensuring safety in Bayesian Optimization algorithms for real-world applications.
method Investigated and improved three safety-related issues of SafeOpt-type algorithms: frequentist uncertainty bounds, RKHS norm assumptions, and discrete search spaces.
result Introduced Real-{eta}-SafeOpt, Lipschitz-only Safe Bayesian Optimization (LoSBO), and Lipschitz-only GP-UCB (LoS-GP-UCB) algorithms that retain safety guarantees and superior performance.
Researchers introduce a method to assess the safety of interpretable machine learning models.
problem Ensuring safety in machine learning models that are easy to understand.
method Introduce maximum deviation as an optimization problem to find the largest deviation from a safe reference model.
result Interpretability helps in assessing the safety of machine learning models.
New method provides scalable safety guarantees for RL agents.
problem Safe reinforcement learning in real-life scenarios.
method State-augmentation and shield design for probabilistic avoidance.
result Strict formal safety guarantees for RL agents, scalable and practical.
By building on a recently introduced genetic-inspired attribute-based conceptual framework for safety risk analysis, we propose a novel methodology to compute construction univariate and bivariate construction safety risk at a situational level. Our fully data-driven approach provides construction practitioners and aca…
Algorithm minimizes regret while adhering to unknown safety constraints.
problem Online learning with unknown safety constraints.
method General meta-algorithm leveraging online regression and learning oracles.
result Concrete algorithm with T \sqrt{T} T regret for linear constraints. The paper shows why AI safety doesn't generalize across tasks.
problem AI safety fails to generalize across unseen tasks.
method Theoretical analysis of linear-quadratic control with H ∞ H_{\infty} H ∞ -robustness, empirical demonstrations in simulated quadcopter navigation and CRM. result The mapping from task specification to an optimal controller has a higher Lipschitz constant with safety requirements than without, indicating inherent complexity of safety.
Study probabilistic safety of BNNs under adversarial attacks.
problem Evaluate vulnerability of BNNs to adversarial attacks.
method Relaxation techniques from non-convex optimization to compute probabilistic safety bounds.
result Certify probabilistic safety of BNNs with millions of parameters.
Formal constraints improve RL safety in complex environments.
problem Safety constraints in reinforcement learning for complex environments.
method Specify constraints in formal languages, instantiate as finite automata, augment MDP states, learn dense cost function.
result Improved safety in training RL algorithms over various constraints.
Safe learning of stochastic dynamics with safety constraints.
problem Learning controlled stochastic dynamics with safety constraints.
method Iterative expansion of a safe control set using kernel-based confidence bounds.
result The method ensures safe exploration and efficient estimation of system dynamics.
Machine learning algorithms are increasingly influencing our decisions and interacting with us in all parts of our daily lives. Therefore, just like for power plants, highways, and myriad other engineered sociotechnical systems, we must consider the safety of systems involving machine learning. In this paper, we first …
Unified control theory and machine learning for safety in uncertain systems.
problem Safety guarantees for systems with measurement model uncertainty.
method Measurement-Robust Control Barrier Functions (MR-CBFs) for control synthesis.
result MR-CBFs ensure safety in perception systems with measurement model uncertainty.
Logarithmic regret strategies for safe multi-armed bandits with safety risk constraints.
problem Maximizing reward while avoiding unsafe arms under safety risk constraints.
method Doubly optimistic strategies with pseudo-regret formulation.
result Logarithmic regret bounds for safe multi-armed bandits.
A new method reduces the computational burden of safety alignment for large language models.
problem Safety concerns in large language models and the need to align them with human preferences.
method Optimal dualization approach to reduce constrained alignment to an unconstrained problem.
result Our algorithms MoCAN and PeCAN significantly reduce computational burden and improve training stability.
Urban traffic systems worldwide are suffering from severe traffic safety problems. Traffic safety is affected by many complex factors, and heavily related to all drivers' behaviors involved in traffic system. Drivers with aggressive driving behaviors increase the risk of traffic accidents. In order to manage the safety…
ARTEO algorithm optimizes safety-critical systems with uncertainty.
problem Decision-making under uncertainty with safety constraints in real-time optimization.
method ARTEO algorithm uses multi-armed bandits as a mathematical programming problem subject to safety constraints, learning unknown characteristics through exploration and incorporating uncertainty quantification.
result ARTEO achieves less cumulative regret with accurate and safe decisions.
Synthesizes machine learning applications in reliability and safety.
problem Navigating the fragmented literature on ML for reliability and safety.
method Overview of ML categories, review of applications, discussion of Deep Learning.
result Machine learning can provide novel insights and improve accident prevention.
The paper tackles safe exploration in RL by a conservative safety critic.
problem Safe exploration in reinforcement learning (RL) when partially trained policies are deployed.
method Learning a conservative safety estimate through a critic, provably bounding catastrophic failures.
result The approach provably converges to competitive task performance with significantly lower catastrophic failure rates.
High-dimensional models pose both safety benefits and risks.
problem Emergent problems in safety alignment due to high-dimensional representations.
method Detailed visualizations and lower-dimensional subspace projections.
result Dimensional reduction preserves safety alignment while avoiding linear jailbreaking.
CitySim dataset captures vehicle trajectories for safety research.
problem Lack of fine-grain vehicle trajectories for safety-oriented research.
method Five-step procedure: video stabilization, object filtering, stitching, detection, and error filtering.
result CitySim dataset improves safety evaluations and facilitates digital-twin research.
A new RL model ensures safe learning in uncertain environments.
problem Safe reinforcement learning in uncertain, partially observable environments.
method Lyapunov-based uncertainty quantification and Transformers for memory.
result Significant improvement in safety and optimality in grid-world tasks.
Neural network predicts nonlinear safety behavior based on personality traits.
problem Predicting construction workers' unsafe behaviors based on personality traits.
method Developed a forecasting model using neural network algorithms.
result Nonlinear relationship exists between personality traits and safety behavior.
This work improves safety validation of autonomous vehicles by finding interpretable failures.
problem Finding interpretable failures of autonomous systems in simulation.
method Signal temporal logic expressions optimized for high likelihood and human interpretability.
result Our methodology finds more interpretable failures with higher likelihood compared to baseline approaches.
AI systems that explain their decisions can be monitored for harmful intentions.
problem Monitoring AI systems' decision-making processes for harmful intentions is imperfect and can miss some misbehavior.
method Monitoring the chain of thought (CoT) of AI systems that communicate in human language.
result CoT monitoring is a promising but fragile approach to AI safety.
Investigates safe decision-making in interactive environments.
problem Learning the best safe decision in real-time systems.
method Reduces to a constrained linear bandits problem, proposes adaptive experimental design-based algorithm.
result First results on best-arm identification in linear bandits with safety constraints.
Reinforcement learning is a promising approach to synthesizing policies for challenging robotics tasks. A key problem is how to ensure safety of the learned policy---e.g., that a walking robot does not fall over or that an autonomous car does not run into an obstacle. We focus on the setting where the dynamics are know…