Classifier-based AI safety gates fail in self-improvement, even with advanced verification methods.
problem Maintaining reliable oversight of AI systems as they improve over iterations.
method Comprehensive empirical testing on neural controllers and MuJoCo benchmarks, using various classifiers and verification methods.
result Classifier-based safety gates fail in maintaining reliable oversight, even with advanced verification methods.
AGENTICAITA uses AI agents to autonomously trade markets without human intervention.
problem Inability of traditional trading systems to adapt to market complexity.
method Introduces an agentic AI framework with specialized LLM agents reasoning, negotiating, and acting.
result Demonstrated operational correctness and non-trivial inter-agent negotiation in live market conditions.
A new framework for SPX and VIX hedging that combines AI and market dynamics.
problem Jointly hedging SPX and VIX exposures under transaction costs and regime shifts.
method Integrates an SSVI-based implied-volatility surface and a Cboe-compliant VIX computation with a control layer that enforces safety as constraints.
result Reduces expected shortfall while suppressing nuisance turnover in a reproducible synthetic environment.
In this paper we discuss how systems with Artificial Intelligence (AI) can undergo safety assessment. This is relevant, if AI is used in safety related applications. Taking a deeper look into AI models, we show, that many models of artificial intelligence, in particular machine learning, are statistical models. Safety …
This paper formalizes AI safety using hypothesis testing in GenAI.
problem Ensuring safety of generative AI tools that create realistic content.
method Formalization of computational safety through hypothesis testing and signal processing.
result Demonstrates how AI safety can be assessed quantitatively using mathematical frameworks.
AI systems that explain their decisions can be monitored for harmful intentions.
problem Monitoring AI systems' decision-making processes for harmful intentions is imperfect and can miss some misbehavior.
method Monitoring the chain of thought (CoT) of AI systems that communicate in human language.
result CoT monitoring is a promising but fragile approach to AI safety.
The paper shows why AI safety doesn't generalize across tasks.
problem AI safety fails to generalize across unseen tasks.
method Theoretical analysis of linear-quadratic control with H∞-robustness, empirical demonstrations in simulated quadcopter navigation and CRM. result The mapping from task specification to an optimal controller has a higher Lipschitz constant with safety requirements than without, indicating inherent complexity of safety.
This paper tackles sandbagging in AI safety evaluations.
problem AI agents may hide dangerous capabilities to avoid being deactivated.
method Developed a simple model of strategic deception in sequential decision-making tasks.
result Demonstrated that optimal rational agents exhibit sandbagging behavior.
Autonomous agents trained via reinforcement learning present numerous safety concerns: reward hacking, negative side effects, and unsafe exploration, among others. In the context of near-future autonomous agents, operating in environments where humans understand the existing dangers, human involvement in the learning p…
Insurance contracts for autonomous AI agents must be actuarially sound and resistant to gaming.
problem Designing insurance contracts for autonomous AI agents that are actuarially sound and resistant to gaming.
method Characterizing a five-attack space and proving the actuarial runtime is gaming-resistant.
result An incentive-compatible layer for actuarial control of autonomous-agent side effects.
New bounds on predicting agent behavior from behavior alone.
problem Predicting agent beliefs and intentions from observed behavior.
method Derivation of bounds on agent behavior in new environments under assumption of world model.
result Theoretical limits on predicting intentional agents from behavioral data.
ValueBlindBench tests LLM-generated investment rationales for validity before returns are known.
problem Delayed-ground-truth evaluation of LLM-generated investment rationales.
method Agreement-gated stress testing protocol to validate LLM-judged rationales.
result ValueBlindBench prevents overclaims and identifies flawed financial constructs.
Quantum circuits explained using Shapley values for better understanding.
problem Improving the explainability of quantum machine learning circuits.
method Applying Shapley values to quantify gate importance in quantum circuits.
result Quantum circuits can be explained by their gate importance, enhancing understanding and interpretability.
New risk metric for AI systems reduces safety risks with minimal data.
problem Risk assessment in multi-agent AI systems.
method Free Energy Principle applied to risk metrics, introducing Cumulative Risk Exposure.
result Gatekeepers improve system safety in autonomous vehicle fleets.
This paper tests LLMs in finance to assess ethical behavior.
problem Aligning AI with ethical and legal standards in finance.
method Prompted LLMs to simulate CEO behavior, analyzed with logistic regression.
result Significant heterogeneity in LLMs' unethical behavior propensity.
Study investigates how machine learning models degrade over time, leading to patient safety issues.
problem Overtime degradation of machine learning models in clinical settings.
method Used MIMIC-IV dataset to train models replicating commercial approaches, observing and analyzing degradation over a decade.
result An RNN model built on Epic features degrades from 0.729 AUC to 0.525 AUC over a decade, highlighting technical and clinical drift as root causes.
Proposes a method to ensure accurate estimation of rare events in AI systems.
problem Lack of efficiency guarantees in black-box systems for rare-event simulation.
method Integrates deep learning with importance sampling to create a statistically guaranteed estimator.
result Demonstrates effective estimation of rare-event probabilities in AI systems.
YRC-Bench benchmarks AI agents learning to collaborate with experts.
problem Learning to recognize when to consult an expert in novel situations.
method Validation strategy and proposer-validator decomposition.
result Insights for future AI safety research.
AI systems need reliable testing to ensure safety and trustworthiness.
problem Current AI Act lacks functional trustworthiness for AI systems.
method Define technical application distribution, set risk-based performance, and conduct statistically valid testing.
result Reliable functional trustworthiness is essential for AI systems.
In a world of global trading, maritime safety, security and efficiency are crucial issues. We propose a multi-task deep learning framework for vessel monitoring using Automatic Identification System (AIS) data streams. We combine recurrent neural networks with latent variable modeling and an embedding of AIS messages t…
Paper proposes government indemnification for AI risks to solve judgment-proof problem.
problem Uninsurable risks from AI, especially existential risks, create a judgment-proof problem.
method A government-provided, mandatory indemnification program using risk-priced fees and Bayesian Truth Serum.
result The approach better leverages private information and signals risk mitigation efforts.
AI monitors social distancing and masks at manufacturing plants.
problem Ensuring safety of workers during post-COVID production.
method Computer vision and AI techniques for social distancing and mask detection.
result Real-time alerts prevent violations of social distancing and mask-wearing.
Teaches uncertainty in ML through practical examples.
problem Lack of uncertainty teaching in ML curricula.
method Developed a curriculum and use cases.
result Motivates adoption of uncertainty concepts in AI courses.
ProEval efficiently estimates AI performance and discovers failures using pre-trained Gaussian Processes.
problem Resource-intensive evaluation of generative AI models.
method ProEval uses pre-trained Gaussian Processes and Bayesian quadrature to estimate performance and discover failures.
result ProEval requires significantly fewer samples to achieve accurate performance estimates and reveals more diverse failure cases.
Rapid advancements in deep learning have led to many recent breakthroughs. While deep learning models achieve superior performance, often statistically better than humans, their adoption into safety-critical settings, such as healthcare or self-driving cars is hindered by their inability to provide safety guarantees or…
Survey of algorithms for testing AI-driven CPS safety.
problem Testing AI-driven CPS for safety in complex environments.
method Survey of applied algorithms for safety validation.
result Survey of existing tools and techniques for safety validation.
New framework uses OR to ensure AI systems make safe decisions.
problem Ensuring generative AI systems make safe decisions as they gain autonomy.
method Developed a conceptual framework combining flow-based models and adversarial robustness.
result Increased autonomy requires new OR approaches for feasibility, robustness, and stress testing.
Quantile regression attacks outperform shadow models in unseen class membership inference attacks.
problem Failure of shadow model attacks on unseen classes due to limited data access.
method Quantile regression attacks that learn features of member examples.
result Quantile regression attacks achieve up to 11x higher TPR than shadow model-based approaches.
Tail-Safe hedging uses reinforcement learning with a safety layer to manage financial risks.
problem Managing financial risks in derivatives trading with robustness and explainability.
method Combines distributional reinforcement learning with a CBF-QP safety layer to enforce financial constraints.
result Improves risk management without degrading central performance and avoids hard constraint violations.
New method detects dangerous states in gridworlds using geometric defects.
problem Detecting dangerous states in gridworlds for AI safety.
method Modified state complexes and Gromov's Link Condition.
result Geometric defects indicate dangerous states in gridworlds.
Theoretical limits on verifying self-improving systems without risking unbounded utility.
problem Formalizing and proving the limits of safety verification for self-improving systems.
method Developed dual conditions and used Holder's inequality, NP counting method, and Lipschitz bounds to establish impossibility and ceiling results.
result A classifier-based safety gate cannot simultaneously permit unbounded beneficial self-modification and bounded cumulative risk.
There is an analogy between machine learning systems and economic entities in that they are both adaptive, and their behaviour is specified in a more-or-less explicit way. It appears that the area of AI that is most analogous to the behaviour of economic entities is that of morally good decision-making, but it is an op…
The paper highlights AI brittleness and the need for robust testing out-of-distribution performance.
problem The brittleness of AI systems, especially Deep Neural Networks, limits their reliability and certification.
method Analysis of AI brittleness and OOD performance, emphasizing the need for resilience and improved evaluation methods.
result AI systems are more failure-prone than certified in critical systems, and OOD performance falls off gradually.
Survey on biases in image analysis for industrial safety.
problem Bias in machine learning algorithms affects industrial safety-critical applications.
method Survey and analysis of recent advances in bias detection and mitigation.
result Need for new methods to detect and mitigate biases in image analysis for safety-critical applications.
An accurate road surface friction prediction algorithm can enable intelligent transportation systems to share timely road surface condition to the public for increasing the safety of the road users. Previously, scholars developed multiple prediction models for forecasting road surface conditions using historical data. …
The paper investigates AI robustness through experiments and statistical analysis.
problem Inaccurate AI predictions can lead to safety and adoption issues.
method Design of experiments framework to study AI classification robustness.
result AI algorithms' robustness is influenced by various factors.
Survey of alignment techniques for large language models.
problem Ensuring large language models align with human values.
method Analysis of diverse alignment methods and training paradigms.
result Preference-based methods offer more flexibility for nuanced alignment.
Localized Multidirectional Correction improves non-refusal target-response behavior in foundation models.
problem Controlled post-training refusal suppression in routed MoE and hybrid-MoE foundation models.
method Introduce Localized Multidirectional Correction (LoMC), a support-gated intervention framework.
result Substantially improves non-refusal target-response behavior while maintaining general capability under a compact intervention footprint.
Enhances crowd safety through AI and data-driven models.
problem Improving crowd safety during events.
method Innovative data collection, AI, and machine learning.
result Accurate multi-day forecasts for event planning.
Quantum model improves safety in machine learning.
problem Improving safety and robustness in machine learning models.
method Variational quantum classifier with amplitude encoding and SAFE-AI metrics.
result Quantum model provides competitive performance and improved robustness.
Graph-structured data appears frequently in domains including chemistry, natural language semantics, social networks, and knowledge bases. In this work, we study feature learning techniques for graph-structured inputs. Our starting point is previous work on Graph Neural Networks (Scarselli et al., 2009), which we modif…
This review clarifies XAI for regression models and establishes new theoretical insights.
problem Lack of XAI techniques for regression models, especially in safety-critical applications.
method Clarifies conceptual differences, establishes theoretical insights, provides demonstrations, discusses challenges.
result Novel theoretical insights and demonstrations of XAI for regression models.
New approach extracts AI model representations for steering and monitoring.
problem Understanding and steering AI models' internal knowledge representation.
method Feature learning advances for scalable linear concept representations.
result Concept representations enable steering and monitoring of AI models.
AI enhances bank credit risk management through deep learning and data analysis.
problem Inaccurate credit decisions and potential risks in bank credit risk management.
method Innovative application of AI technology, including deep learning and big data analysis.
result AI provides more accurate and comprehensive credit decision support, reducing risks and losses.
In order to port the performance of trained artificial neural networks (ANNs) to spiking neural networks (SNNs), which can be implemented in neuromorphic hardware with a drastically reduced energy consumption, an efficient ANN to SNN conversion is needed. Previous conversion schemes focused on the representation of the…
Research tackles distribution shift issues in ML to improve AI reliability.
problem Distribution shift limits ML reliability and trustworthiness.
method Study three distribution shifts (perturbation, domain, modality) and investigate robustness, explainability, adaptability.
result Proposes effective solutions and fundamental insights for enhancing ML robustness, adaptability, and safety.
Topological parallax assesses AI models' geometric similarity to datasets for safety.
problem Ensuring AI models' robustness and safety in deep learning applications.
method Topological parallax compares a trained model to a reference dataset using Rips complexes and geodesic distortions.
result Topological parallax indicates whether a model shares similar multiscale geometric features with the dataset.
Universal AI seeks high-optionality states through empowerment and curiosity.
problem Understanding and optimizing AI behavior in uncertain environments.
method Unified framework combining AIXI and variational empowerment, showing how universal AI agents balance goal-directed behavior with uncertainty reduction curiosity.
result Self-AIXI asymptotically converges to AIXI performance and exhibits power-seeking behavior due to intrinsic motivations.