In this paper, we consider a problem of failure prediction in the context of predictive maintenance applications. We present a new approach for rare failures prediction, based on a general methodology, which takes into account peculiar properties of technical systems. We illustrate the applicability of the method on th…
New method finds failures in high-fidelity simulators with fewer steps.
problem Finding failures in high-fidelity simulators is expensive and impractical.
method Adaptive stress testing with backward algorithm adaptation from low-fidelity to high-fidelity.
result Significantly fewer high-fidelity simulation steps needed to find failures.
CalNF models rare failures with limited data, improving safety in autonomous systems.
problem Challenges in modeling and debugging rare safety-critical failures due to limited data.
method CalNF, a self-regularized framework for posterior learning from limited data.
result Achieves state-of-the-art performance on data-limited failure modeling and inverse problems.
Unified model predicts multi-mode failure with multi-sensor data.
problem Independent failure mode and RUL prediction ignores inherent relationship.
method Hierarchical Bayesian framework with Cox model, Gaussian process, and multinomial distributions.
result Robust uncertainty quantification and accurate prediction of multi-mode failure.
Study extreme-case Value-at-Risk under IFR distributions, providing guidance for risk management.
problem Understanding extreme-case risk measures under distributional ambiguity and increasing failure rate.
method Characterized extreme-case range Value-at-Risk under mean and variance constraints with increasing failure rate.
result Characterized specific characteristics of extreme-case distributions under IFR constraints.
ProEval efficiently estimates AI performance and discovers failures using pre-trained Gaussian Processes.
problem Resource-intensive evaluation of generative AI models.
method ProEval uses pre-trained Gaussian Processes and Bayesian quadrature to estimate performance and discover failures.
result ProEval requires significantly fewer samples to achieve accurate performance estimates and reveals more diverse failure cases.
New framework assesses extreme errors in machine learning models.
problem Current validation methods fail to quantify extreme errors in high-stakes domains.
method Uses Extreme Value Theory (EVT) to estimate worst-case failures.
result Establishes EVT as a fundamental tool for assessing model reliability.
Evaluation and validation of complicated control systems are crucial to guarantee usability and safety. Usually, failure happens in some very rarely encountered situations, but once triggered, the consequence is disastrous. Accelerated Evaluation is a methodology that efficiently tests those rarely-occurring yet critic…
Study predicts P2P lending platform failures using machine learning.
problem Predicting failures of P2P lending platforms in China.
method Used machine learning models with filter and wrapper methods, forward selection, and backward elimination.
result Identified robust variables for predicting platform failures with high AUC and F1 scores.
Deep neural networks predict CVCM track circuit failures early.
problem Subtle anomalies in CVCM track circuits lead to failures, causing disruptions.
method Deep neural networks classify anomalies before they escalate.
result Deep neural networks achieve 99.31% overall accuracy in detecting CVCM failures.
BAMS uses Bayesian sampling to discover AV failures more efficiently and accurately.
problem Discovering potential failure cases in autonomous vehicles efficiently and accurately.
method Bayesian adaptive multifidelity sampling (BAMS) prioritizes exploration of low performance regions.
result BAMS discovers 10 times more issues than traditional methods with narrower rate estimates.
Revisits PPO design choices, exposing failure modes and proposing alternatives.
problem Failure modes of standard PPO in new environments.
method Revisits standard PPO design choices, exposes failure modes, and proposes alternative approaches.
result Alternative design choices prevent failure modes in new environments.
Novel asynchronous SGD method resists Byzantine attacks without server storage.
problem Asynchronous distributed learning with Byzantine attacks and failures.
method Buffered Asynchronous SGD (BASGD) and its momentum variant (BASGDm).
result BASGD and BASGDm resist non-omniscient and omniscient attacks without server storage.
Proposes a federated learning approach for RUL prediction from nonparametric degradation and failure signals.
problem Cost-effective RUL prediction from limited, non-shared CM signals with unknown parametric forms.
method Joint modeling of nonlinear degradation signals and failure events using federated learning.
result Superior RUL prediction compared to alternatives, validated through simulations and real data.
As machine learning systems move from computer-science laboratories into the open world, their accountability becomes a high priority problem. Accountability requires deep understanding of system behavior and its failures. Current evaluation methods such as single-score error metrics and confusion matrices provide aggr…
Forward-Euler fails for simulating Wasserstein gradient flows with KL divergence.
problem Simulating Wasserstein gradient flows with forward-Euler discretization fails for KL divergence.
method Forward-Euler discretization for Wasserstein gradient flows with KL divergence.
result Forward-Euler discretization can be incorrect for Wasserstein gradient flows with KL divergence.
This work uses SVM to identify track component failures in AC Track Circuits.
problem Detecting and identifying specific track component failures in AC Track Circuits.
method Applied SVM classifier to STDS track circuit data.
result Successfully classified 15 different track component failures.
Predict water pipe failures using machine learning and survival analysis.
problem Difficulty in accessing water pipes for maintenance.
method Classical and modern classifiers for short-term prediction, survival analysis for long-term forecast, and oversampling technique for imbalanced data.
result Identifies important risk factors for water pipe failures.
Study quantifies events leading to Terra project failure in 2022.
problem Terra project's fragility and dependency on Anchor protocol.
method Systematic review of social media news, hourly and transaction data analysis, network science techniques.
result Identified trigger events and analyzed dependency structures using network science.
End-to-End training (E2E) is becoming more and more popular to train complex Deep Network architectures. An interesting question is whether this trend will continue-are there any clear failure cases for E2E training? We study this question in depth, for the specific case of E2E training an ensemble of networks. Our str…
DG separates successes and failures by gating updates with advantage and surprisal.
problem Negative learning from surprising data in distributed reinforcement learning.
method DG gates each update with the product of advantage and surprisal, suppressing failures and preserving successes.
result DG outperforms other methods in various challenging reinforcement learning tasks.
Economics tool predicts failure times in reliability systems.
problem Predicting optimal failure times in weighted k-out-of-n reliability systems with heterogeneous component failure.
method Using rational expectations to analyze and predict failure times in reliability systems with heterogeneous component failure.
result Different measures are optimal for predicting system failure depending on component failure distributions.
In power systems, an asset class is a group of power equipment that has the same function and shares similar electrical or mechanical characteristics. Predicting failures for different asset classes is critical for electric utilities towards developing cost-effective asset management strategies. Previously, physical ag…
We consider the task of mapping pseudocode to long programs that are functionally correct. Given test cases as a mechanism to validate programs, we search over the space of possible translations of the pseudocode to find a program that passes the validation. However, without proper credit assignment to localize the sou…
In survival analysis, estimating the failure time distribution is an important and difficult task, since usually the data is subject to censoring. Specifically, in this paper we consider current status data, a type of data where all of the observations are censored. The format of the data is such that the failure time …
In environments with continuous state and action spaces, state-of-the-art actor-critic reinforcement learning algorithms can solve very complex problems, yet can also fail in environments that seem trivial, but the reason for such failures is still poorly understood. In this paper, we contribute a formal explanation of…
PAGER detects failures in deep regression models using a new framework.
problem Detecting failures in deep regression models.
method PAGER uses a combination of epistemic uncertainty and manifold non-conformity scores.
result PAGER accurately characterizes and detects failures in deep regressors.
Risk Advisor predicts and mitigates ML deployment failures.
problem Predicting and mitigating test-time failure risks of ML systems.
method Post-hoc meta-learner for estimating failure risks and uncertainties.
result Reliably predicts deployment-time failure risks across various ML models.
This paper develops mfEGRA, a multifidelity active learning method using data-driven adaptively refined surrogates for failure boundary location in reliability analysis. This work addresses the issue of prohibitive cost of reliability analysis using Monte Carlo sampling for expensive-to-evaluate high-fidelity models by…
Prognostics and Health Management (PHM) is an emerging engineering discipline which is concerned with the analysis and prediction of equipment health and performance. One of the key challenges in PHM is to accurately predict impending failures in the equipment. In recent years, solutions for failure prediction have evo…
CB-SLICE identifies concept-based error slices in deep learning models.
problem Systematic errors in deep learning models on specific groups.
method Concept Bottleneck Models (CBMs) and concept representations.
result CB-SLICE outperforms state-of-the-art methods in error slice identification.
This work improves safety validation of autonomous vehicles by finding interpretable failures.
problem Finding interpretable failures of autonomous systems in simulation.
method Signal temporal logic expressions optimized for high likelihood and human interpretability.
result Our methodology finds more interpretable failures with higher likelihood compared to baseline approaches.
Framework predicts remaining useful life of DSH subsystems under unknown failure modes.
problem Predicting remaining useful life of DSH subsystems with unknown failure modes.
method Unsupervised framework using mixture of Gaussian regressions and Expectation-Maximization algorithm.
result Improved prediction accuracy and interpretability of RUL.
As machine learning (ML) models, trained on real-world datasets, become common practice, it is critical to measure and quantify their potential biases. In this paper, we focus on renal failure and compare a commonly used traditional risk score, Tangri, with a more powerful machine learning model, which has access to a …
The paper calculates the likelihood of a financial market failure involving multiple major banks.
problem Estimating the probability of a market failure involving multiple globally important banks.
method Multivariate Cox process across G-SIBs, deriving various theorems on market failure probabilities.
result The probability of a market failure increases with the number of G-SIBs and is inevitable if there are too many.
Paper proposes a method to predict disk failures using multi-layer domain adaptive learning.
problem Traditional machine learning models struggle to predict disk failures due to limited data.
method Multi-layer domain adaptive learning with source and target domains.
result The proposed method improves failure prediction accuracy on disk data with few failure samples.
We aim to predict and explain service failures in supply-chain networks, more precisely among last-mile pickup and delivery services to customers. We analyze a dataset of 500,000 services using (1) supervised classification with Random Forests, and (2) Association Rules. Our classifier reaches an average sensitivity of…
GE finds failures in autonomous systems without domain heuristics.
problem Finding failures in autonomous systems without domain-specific heuristics.
method Adaptive stress testing using go-explore (GE) algorithm.
result GE finds failures in scenarios other RL techniques cannot solve.
This work uses image generation models to find vision model bugs.
problem Automatically discovering failures in vision models.
method Conditional text-to-image generation and captioning models.
result Demonstrated utility of large-scale generative models to find vision model bugs.
Researchers develop PAIN to improve self-driving safety through adversarial training.
problem Overfitting and poor generalizability of neural networks in self-driving vehicles.
method PAIN combines adversarial training in CARLA simulation to generate edge cases.
result Trained self-driving vehicles are more resilient to environmental uncertainty and less prone to collisions.
Method distinguishes between failures and domain shifts in industrial data streams.
problem Confusing domain shifts with failures in industrial data.
method Modified Page-Hinkley changepoint detector and supervised domain-adaptation-based anomaly detection.
result Allows differentiation between failures and domain shifts.
System predicts respiratory failure up to 8 hours early.
problem Early detection of respiratory failure in ICU patients.
method Machine learning on ICU patient monitoring data.
result System outperforms traditional clinical decision-making.
New method avoids failures in physics-constrained systems using active learning.
problem Handling fatal failures in systems governed by physics constraints.
method Develops a novel active learning method that considers implicit physics constraints.
result Achieves zero-failure in composite fuselage assembly process without explicit failure regions.
This paper addresses the problem of evaluating learning systems in safety critical domains such as autonomous driving, where failures can have catastrophic consequences. We focus on two problems: searching for scenarios when learned agents fail and assessing their probability of failure. The standard method for agent e…
DeepSIP predicts network failures' impact using CNN from syslog and traffic data.
problem Predicting service impact from network failures.
method Temporal multimodal CNN for predicting time to recovery and traffic loss.
result DeepSIP reduced prediction error by approximately 50%.
We introduce a general framework for models of cascade and contagion processes on networks, to identify their commonalities and differences. In particular, models of social and financial cascades, as well as the fiber bundle model, the voter model, and models of epidemic spreading are recovered as special cases. To uni…
In the last two years, more than 200 papers have been written on how machine learning (ML) systems can fail because of adversarial attacks on the algorithms and data; this number balloons if we were to incorporate papers covering non-adversarial failure modes. The spate of papers has made it difficult for ML practition…
Many real-world complex systems across natural, social, and economical domains consist of manifold layers to form multiplex networks. The multiple network layers give rise to nonlinear effect for the emergent dynamics of systems. Especially, weak layers that can potentially play significant role in amplifying the vulne…