AUC is unreliable in rare event settings but stable with moderate numbers of events.
problem Misleading performance metrics in rare event settings.
method Simulation study varying dataset sizes and event rates.
result AUC is unreliable in rare event settings but stable with moderate numbers of events.
This paper evaluates data enrichment techniques for rare event detection in manufacturing.
problem Rare events in manufacturing lead to unplanned downtime and high energy consumption.
method Time series data augmentation, sampling, and imputation techniques combined with supervised machine learning.
result Data enrichment enhances rare failure event detection and prediction by up to 48%.
Study compares resampling methods for rare event prediction in longitudinal studies.
problem Predicting rare events in longitudinal follow-up studies.
method Comparison of resampling methods to improve standard regression models.
result Effect of sampling rate on model predictive performance.
A cased-based reasoning method predicts rare events on strategic sites using satellite imagery.
problem Manual prediction of rare events on strategic sites is impractical due to large datasets.
method Case-based reasoning approach incorporating expert knowledge for irregular time series and small datasets.
result The method significantly outperforms random selection on challenging applications.
Proposes a method to improve rare event prediction in healthcare.
problem Rare event classification in healthcare with low prevalence labels.
method Variational disentanglement approach to semi-parametric learning.
result Outperforms existing alternatives in mortality prediction on COVID-19 cohort.
Develops methods for spectral estimation and rare-event prediction in complex systems.
problem Challenges in understanding dynamics in complex systems with many degrees of freedom.
method Inexact iterative numerical linear algebra methods for spectral estimation and rare-event prediction.
result Demonstrates methods on low-dimensional and high-dimensional models, showing their effectiveness.
A real-world dataset is provided from a pulp-and-paper manufacturing industry. The dataset comes from a multivariate time series process. The data contains a rare event of paper break that commonly occurs in the industry. The data contains sensor readings at regular time-intervals (x's) and the event label (y). The pri…
PRESTO improves rare event prediction by shrinking towards proportional odds model.
problem Difficult to predict rare events due to class imbalance.
method PRESTO relaxes proportional odds model by estimating separate weights for transitions between categories, imposing L1 penalty to shrink towards proportional odds.
result PRESTO consistently estimates decision boundary weights under sparsity assumption, improving rare probability estimation.
The paper introduces diagnostic transport maps to improve the reliability of rare event predictions.
problem Improper calibration of predictive distributions, especially for rare events.
method Diagnostic transport maps to adjust base model's probabilities for better calibration.
result Diagnostic transport maps improve predictive performance for rare events, including 24-hour rapid intensity change.
A new method uses deep learning to predict rare events in complex systems.
problem Predicting rare and extreme events in non-equilibrium systems.
method A deep learning approach that minimizes the geometrical action.
result The method accurately predicts rare events in various complex systems.
In this paper, we consider a problem of failure prediction in the context of predictive maintenance applications. We present a new approach for rare failures prediction, based on a general methodology, which takes into account peculiar properties of technical systems. We illustrate the applicability of the method on th…
Paper introduces ML for rare-event prediction in patent quality estimation.
problem Lack of predictive modeling in econ, management, tech forecasting.
method Introduces ML approach for optimizing predictive performance.
result Demonstrates synergy between ML and inferential statistics.
Generative model simulates rare events for better decision making.
problem Rare events impact decision making but are hard to sample.
method Normalizing Flow coupled with Importance Sampling.
result Accurate estimation of rare events improves decision outcomes.
This paper shows that one cannot learn the probability of rare events without imposing further structural assumptions. The event of interest is that of obtaining an outcome outside the coverage of an i.i.d. sample from a discrete distribution. The probability of this event is referred to as the "missing mass". The impo…
Develops a method to estimate rare-event probabilities under distributional uncertainty.
problem Distributional uncertainty limits the effectiveness of rare-event simulation techniques.
method Wasserstein distributionally robust rare-event simulation (DRIS) framework.
result DRIS achieves vanishing relative error in estimating rare-event probabilities.
AI boosts study of rare weather extremes with lower costs.
problem Difficulty in studying rare weather events due to limited data and models.
method Coupling AI forecasts with physics models using rare-event algorithms.
result Efficiently characterizes very rare events like once-per-millennium heatwaves.
A new method reduces uncertainty in predicting rare extreme events without assuming their presence in training data.
problem Predicting rare and extreme events in complex systems with high uncertainty.
method Extreme Event Aware (e2a or η) learning, which enforces extreme event statistics during training.
result Models generate unprecedented extreme events even when training data lacks extremes.
The paper analyzes logistic regression for rare events data, deriving new insights on estimator efficiency and sampling strategies.
problem Binary logistic regression for rare events data with significantly fewer events than controls.
method Derives asymptotic distribution of MLE, proves under-sampling advantage, and compares over-sampling efficiency.
result Under-sampling a small proportion of nonevents can improve efficiency in rare events data analysis.
Proposes a method to ensure accurate estimation of rare events in AI systems.
problem Lack of efficiency guarantees in black-box systems for rare-event simulation.
method Integrates deep learning with importance sampling to create a statistically guaranteed estimator.
result Demonstrates effective estimation of rare-event probabilities in AI systems.
Enhanced diffusion sampling improves rare event sampling in biomolecular simulations.
problem Efficiently sampling rare transition events in biomolecular systems.
method Quantitative steering protocols to generate biased ensembles and exact reweighting.
result Fast, accurate, and scalable estimation of equilibrium properties.
Enhanced diffusion sampling tackles rare event sampling in biomolecular simulations.
problem Efficiently sampling rare transition events in biomolecular simulations.
method Quantitative steering protocols to generate biased ensembles, followed by exact reweighting.
result Fast, accurate, and scalable estimation of equilibrium properties for folding free energies.
Proposes a method to refine PDE-driven high-dimensional rare-event simulation.
problem Challenges in constructing accurate surrogates for rare-event simulation.
method Adaptive importance sampling framework that refines a locally constructed surrogate.
result Achieves accuracy comparable to true-model adaptive importance sampling with fewer high-fidelity evaluations.
Efficiently estimates rare events using multifidelity modeling.
problem Estimating rare events with computationally expensive models.
method Active learning with multifidelity modeling, adapting the number of high-fidelity simulations based on problem complexity and desired accuracy.
result Significantly reduced the number of high-fidelity model calls while maintaining accuracy.
New deep learning method handles rare and imbalanced events in time series.
problem Challenges in event detection in time series data, especially rare and imbalanced events.
method Supervised regression-based deep learning approach that handles various types of events.
result Superior performance across diverse domains, particularly for rare events and imbalanced datasets.
Deep-PrAE improves rare-event simulation for black-box systems.
problem Evaluating rare safety-critical events in learning-based systems.
method Combines deep neural networks with IS to create statistically guaranteed estimations.
result Deep-PrAE provides accurate bounds on safety-critical event probabilities.
It is common in modern prediction problems for many predictor variables to be counts of rarely occurring events. This leads to design matrices in which many columns are highly sparse. The challenge posed by such "rare features" has received little attention despite its prevalence in diverse areas, ranging from natural …
Develops a method to simulate rare dangerous events in autonomous systems.
problem Rare dangerous events in safety-critical systems are hard to test in real-world settings.
method Combines exploration, exploitation, and optimization techniques for rare-event simulation.
result Provides rigorous guarantees for the performance of the method.
NOFIS uses normalizing flows to estimate rare event probabilities more efficiently.
problem Accurate estimation of rare event probabilities using conventional methods is inefficient and resource-intensive.
method NOFIS learns a sequence of proposal distributions by minimizing KL divergence losses and estimates rare event probability using importance sampling.
result NOFIS outperforms baseline approaches in estimating rare event probabilities across 10 distinct test cases.
Reduces false positives in classifying rare online platforms.
problem Challenges in accurately identifying rare online platforms with ML.
method Calibrated probabilities and ensembles to reduce bias.
result Significantly reduces false positives in rare event detection.
New test uncovers causal links in rare event dynamics.
problem Causal discovery for rare event phenomena in dynamic systems.
method Nonparametric conditional independence test on time-invariant data.
result Validated across simulated and real-world datasets.
A new method estimates rare events using tensor trains.
problem Estimating rare event probabilities in high-dimensional problems.
method Approximating optimal importance distribution via tensor-train decompositions and compositions.
result Better variance reduction and efficient computation of rare event probabilities.
PHINN: A generative model for rare-event time series using persistent homology
problem Generating rare events in time series
method Flow-matching framework with dynamic Betti curves and persistence landscape loss
result Outperforms statistical and diffusion baselines in topological fidelity and tail coverage
Develops RES metrics for stable rare-event forecasting evaluation.
problem Challenges in evaluating forecasts of rare events.
method Rare-event-stable (RES) metrics designed to maintain stable thresholds under extreme rarity.
result RES metrics maintain stable thresholds, consistent model rankings, and near-complete prevalence invariance.
Adaptive Multilevel Splitting improves rare event pricing for financial derivatives.
problem Efficient pricing of binary options in rare event regimes with discontinuous payoffs.
method Adaptive Multilevel Splitting (AMS) reformulates rare-event problem as conditional events.
result AMS achieves up to 200-fold improvements over standard Monte Carlo, preserving unbiasedness.
Study rare-event simulation for neural networks and random forests.
problem Safety evaluation and robustness quantification of machine learning models.
method Importance sampling scheme integrating large deviations and sequential mixed integer programming.
result Efficiency guarantees and numerical demonstrations for various neural network architectures.
Quantum model captures rare financial events not seen by Gaussian statistics.
problem Underestimation of rare financial events by Gaussian statistics.
method Quantum Bohmian Mechanics applied to multifractal random walk (MRW) models.
result Rare financial events generate a potential barrier in quantum potentials.
New method uses active importance sampling for rare event optimization in high-dimensional problems.
problem Optimizing complex, high-dimensional functions with rare events.
method Combines rare events sampling with neural network optimization.
result Importance sampling reduces asymptotic variance, improving generalization.
Markov chain Monte Carlo (MCMC) algorithms for hidden Markov models often rely on the forward-backward sampler. This makes them computationally slow as the length of the time series increases, motivating the development of sub-sampling-based approaches. These approximate the full posterior by using small random subsequ…
SS-GEN simulates rare events in heavy and light-tailed data.
problem Estimating probabilities of extreme events in multivariate data.
method Self-Similar Generative Estimation (SS-GEN) decomposes tail distribution into radial and angular components.
result SS-GEN generates representative extreme scenarios and estimates rare-event probabilities beyond observed data.
MIM-based GAN improves rare event generation in GANs.
problem Improving rare event generation in GANs.
method Adopting MIM (exponential form of information metric) to replace KL divergence in GANs.
result MIM-based GAN achieves state-of-the-art performance in anomaly detection.
New method learns diffusion bridges for rare events.
problem Simulating rare events in diffusion processes.
method Iterative online learning based on self-consistency.
result Strong performance in various empirical settings.
New risk models use chaotic attractors to predict extreme events.
problem Predicting Black Swan events in financial markets.
method Combining heavy-tailed priors with chaotic dynamics (Lorenz and Rossler systems).
result Models generate volatility clustering, fat tails, and extreme events.
Jeffreys Flow improves robustness of Boltzmann generators for rare event sampling.
problem Rare events and metastable trapping in sampling physical systems with rough energy landscapes.
method Introduces Jeffreys Flow, a robust generative framework using Parallel Tempering distillation and symmetric Jeffreys divergence to mitigate mode collapse and improve mode coverage.
result Minimizing Jeffreys divergence suppresses mode collapse and corrects inaccuracies in multi-modal distributions.
The paper explains how importance sampling can be used for optimization of rare events.
problem Minimizing tail risks in stochastic optimization formulations.
method Importance sampling for reducing sample requirements in estimating rare events.
result Effective importance sampling techniques for optimization of rare events.
Estimates treatment effects in rare extreme events using EVT.
problem Estimating treatment effects in rare, impactful events like extreme climate events.
method Introduces a novel framework using EVT and multivariate regular variation for consistent treatment effect estimation.
result Developed a consistent estimator for extreme treatment effects with rigorous non-asymptotic analysis.
Study compares two methods for predicting extreme atmospheric events.
problem Forecasting threshold exceedances of atmospheric variables like temperature and wind speed.
method Direct vs. full distribution probabilistic methods for rare events.
result Full distribution approach outperforms direct method for extreme events.
We study the problem of predicting rare critical transition events for a class of slow-fast nonlinear dynamical systems. The state of the system of interest is described by a slow process, whereas a faster process drives its evolution and induces critical transitions. By taking advantage of recent advances in reservoir…
TRAKNN detects rare atmospheric trajectories efficiently.
problem Detecting rare atmospheric anomalies over long periods.
method Unsupervised, recurrence-based kNN algorithm.
result Rare trajectories correspond to physical anomalies.