PITMonitor monitors model calibration over time with formal error guarantees.
problem Fixed-sample tests applied to models over time can lead to false alarms.
method PITMonitor uses mixture e-processes to detect distributional shifts in probability integral transforms.
result PITMonitor achieves competitive detection rates on river's FriedmanDrift benchmark.
The significance of air pollution and the problems associated with it are fueling deployments of air quality monitoring stations worldwide. The most common approach for air quality monitoring is to rely on environmental monitoring stations, which unfortunately are very expensive both to acquire and to maintain. Hence e…
A method to monitor probability predictions for calibration loss in image classification models.
problem Maintaining calibration in machine learning predictions over time.
method Cumulative sum-based approach with dynamic limits for early detection of miscalibration.
result Early detection of operational context changes impacting image classification performance.
Work addresses long-term accuracy issues in IoT air quality sensors.
problem Limited accuracy of IoT air quality sensors in long-term field deployments.
method Adaptive machine learning strategies for network calibration.
result Prolongs the validity of multisensor calibration models for continuous learning.
Framework monitors insurance pricing models for drift and recalibration.
problem Maintaining predictive performance of pricing models in evolving insurance portfolios.
method Formalizes deviance loss and Murphy's score, studies Gini score, develops monitoring framework.
result Framework guides decisions on refitting or recalibrating pricing models.
Calibrated strategies can be obtained by performing strategies that have no internal regret in some auxiliary game. Such strategies can be constructed explicitly with the use of Blackwell's approachability theorem, in an other auxiliary game. We establish the converse: a strategy that approaches a convex B-set can be…
RESPIRE calibrates low-cost air-quality sensors for CO levels, resistant to outliers.
problem Calibrating LCAQ sensors against regulatory-grade monitors is expensive and time-consuming.
method PROvably outlier-resistant semi-parametric regression technique.
result RESPIRE offers improved prediction in cross-site, cross-season, and cross-sensor settings.
Simple online monitor detects unsafe LLM outputs.
problem LLMs generate unsafe outputs despite training.
method Thresholding external verifier signal to decide alarms.
result Simple design competitive with advanced methods.
Machine learning components such as deep neural networks are used extensively in Cyber-Physical Systems (CPS). However, they may introduce new types of hazards that can have disastrous consequences and need to be addressed for engineering trustworthy systems. Although deep neural networks offer advanced capabilities, t…
Online monitor detects classifier drift and adapts predictions.
problem Silent degradation of classifier accuracy under distributional shift.
method Sliding-window KS statistic with calibrated alarm thresholds.
result 86.6% valid detection across various shift conditions.
Researchers calibrate an adaptive Farmer-Joshi model to recover stylized facts in financial markets.
problem Recovering stylized facts in financial markets using the Farmer-Joshi model.
method Calibrated an adaptive Farmer-Joshi model using genetic and Nelder-Mead algorithms, incorporating agent adaptation.
result The adaptive model recovers additional stylized facts, including auto-correlations and kurtosis, compared to the original model.
New monitoring method detects ML risk models' performance changes in medical interventions.
problem Monitoring ML risk models in healthcare is complicated by confounding medical interventions.
method Developed a new score-based CUSUM monitoring procedure with dynamic control limits.
result Valid inference is possible if conditional exchangeability or time-constant selection bias hold.
Machine learning models deployed in real-world applications are often evaluated with precision-based metrics such as F1-score or AUC-PR (Area Under the Curve of Precision Recall). Heavily dependent on the class prior, such metrics make it difficult to interpret the variation of a model's performance over different subp…
SVI and GP surrogates improve calibration of ABMs in epidemiology.
problem Calibrating stochastic ABMs in epidemiology is computationally expensive.
method Stein Variational Inference (SVI) with Gaussian process (GP) surrogates.
result SVI maintains comparable predictive accuracy and calibration effectiveness to MCMC.
Develops a method to continuously audit black-box conditional quantile forecasts.
problem Continuous monitoring of black-box forecasts under changing data streams and regimes.
method Distribution-free and game-theoretic testing framework for non-i.i.d. losses.
result Derives finite-time detection guarantees for miscalibrated forecasts based on features.
C-PP-COAD detects anomalies with limited real data, reducing dependency on real calibration data.
problem Limited real calibration data for online anomaly detection.
method Context-aware prediction-powered conformal online anomaly detection (C-PP-COAD).
result Significantly reduces dependency on real calibration data without compromising FDR control.
We provide novel theoretical insights on structured prediction in the context of efficient convex surrogate loss minimization with consistency guarantees. For any task loss, we construct a convex surrogate that can be optimized via stochastic gradient descent and we prove tight bounds on the so-called "calibration func…
Paper proposes real-time VaR estimation using quantile regression forest with conformal calibration.
problem Real-time estimation of Value at Risk (VaR) in rapidly changing markets.
method Quantile regression forest trained offline, real-time VaR estimates via observed risk factors, conformalized estimator for reliability.
result The proposed method provides reliable real-time VaR estimates.
LSCI provides locally adaptive prediction sets for operator models with tighter coverage.
problem Generating robust, calibrated uncertainty quantification for operator models.
method Local Sliced Conformal Inference (LSCI) for operator models.
result LSCI yields tighter prediction sets with stronger adaptivity compared to conformal baselines.
The Monte Carlo pathwise sensitivities approach is well established for smooth payoff functions. In this work, we present a new Monte Carlo algorithm that is able to calculate the pathwise sensitivities for discontinuous payoff functions. Our main tool is to combine the one-step survival idea of Glasserman and Staum wi…
Spectral Adaptive Conformal Prediction for Structured Non-Exchangeable Data
problem Improving prediction intervals for non-exchangeable time-indexed datasets
method Spectral adaptive conformal prediction
result Improves on fixed spectral weighting while monitoring uncertainty changes
A new method detects changes in machine learning models over time.
problem Automatic monitoring of machine learning models trained on evolving data.
method Score-based statistical hypothesis test for change detection.
result The method can detect changes in any number of model components.
The paper calibrates shrinkage covariance estimators for spectral functionals in high dimensions.
problem Calibrating shrinkage covariance estimators for spectral functionals in high dimensions.
method Derives first-order null laws, distribution-free Davis-Kahan bands, and calibrated tests for spectral functionals under shrinkage.
result Calibrated tests and intervals for spectral functionals are provided, addressing the issue of estimation noise and shrinkage bias.
A novel bootstrap method improves concept drift detection in predictive models.
problem Detecting changes in predictive relationships (concept drift) in data-driven applications.
method Developed a nested bootstrap procedure to calibrate control limits using the entire initial sample.
result The method yields more accurate baseline models and faster CL setup times.
Study detects spoofing in high-frequency trading using micro-structural analysis.
problem Challenges in detecting spoofing due to complex electronic platforms and high-frequency trading.
method Micro-structural study in a simplified setting, optimization of spoofing strategy, monitoring with Wasserstein distance.
result Optimal spoofing strategy and its impact on market imbalance quantified.
Machine learning identifies melting points in thermocouples for automatic calibration.
problem Manual calibration of thermocouples is error-prone and time-consuming.
method Machine learning approach to recognize and quantify the melting point of thermocouples.
result 100% accuracy in detecting melting points and high R2 of 0.99 for calibration drift predictions.
Fiber nonlinear interference (NLI) modeling and monitoring are the key building blocks to support elastic optical networks (EONs). In the past, they were normally developed and investigated separately. Moreover, the accuracy of the previously proposed methods still needs to be improved for heterogenous dynamic optical …
Deep learning calibrates CO2 storage formations from seismic and well data.
problem Uncertainty in CO2 storage formation properties.
method Two deep learning models for well and seismic data, integrated into MCMC history matching.
result Significant uncertainty reduction in key parameters and accurate CO2 plume predictions.
Online monitoring system for safety classifiers with shift detection and conformal adaptation
problem Detecting and adapting to distributional shifts in deployed safety classifiers
method Calibrated sequential statistics for online monitoring, conformal abstention for adaptation
result 86.6% valid detection with mean latency of 39.5 steps
AI systems that explain their decisions can be monitored for harmful intentions.
problem Monitoring AI systems' decision-making processes for harmful intentions is imperfect and can miss some misbehavior.
method Monitoring the chain of thought (CoT) of AI systems that communicate in human language.
result CoT monitoring is a promising but fragile approach to AI safety.
GraphSVR forecasts urban air pollution robustly across stations and seasons.
problem Nonlinear, nonstationary, spatiotemporally dependent urban air pollution forecasting challenges.
method Combines graph convolutional learning and support vector regression.
result GraphSVR improves predictive accuracy and maintains stable performance across seasons and outlier-prone episodes.
The study compares on-chain option prices with a model and finds significant differences.
problem Measuring and comparing on-chain option prices with a model-based benchmark.
method Used a two-regime MS-AR-(GJR)-GARCH model to estimate volatility and GLS to compare prices.
result On-chain option prices are significantly higher than model-based benchmarks, especially for call options.
Third part of a study on liquidity risk in asset management, focusing on managing the asset-liability liquidity risk.
problem Managing the asset-liability liquidity risk in asset management.
method Develops a methodological and practical framework for liquidity stress testing programs.
result Proposes measurement, management, and monitoring tools for controlling the liquidity gap.
This research tackles monitoring machine learning algorithms post-deployment, addressing performativity issues.
problem Monitoring machine learning algorithms after deployment, especially when they affect their own data-generating process.
method Uses causal inference techniques to navigate performativity and compares different monitoring criteria and data sources.
result Different monitoring systems have varying operating characteristics and implications for ML monitoring design.
Paper improves ETF tail-risk monitoring reliability.
problem Unreliable ETF risk monitoring under degraded data.
method Combines quality checks, prediction, scoring, and adjustment.
result Improves tail-risk monitoring, especially during stressed periods.
Bayesian framework improves financial risk management and compliance.
problem Uncertainty in financial risk forecasting and compliance.
method Integrated Bayesian analytics framework for precise uncertainty quantification.
result Proposed DLM model produces more accurate VaR estimates compared to baseline models.
The paper adds explanation to predictive process monitoring.
problem Equipping predictive business process monitoring with explanation capabilities.
method Used game theory of Shapley Values to obtain robust explanations.
result First time explanations given in predictive business process monitoring.
IDS algorithm optimizes sequential decisions in various monitoring settings.
problem Optimizing sequential decisions in complex monitoring scenarios.
method Information-directed sampling (IDS) algorithm for linear partial monitoring.
result IDS achieves nearly worst-case rate optimality in finite-action games.
A new method monitors unstructured 3D shapes without registration.
problem Error-prone registration and mesh reconstruction steps in PCD monitoring.
method Intrinsic geometric properties of shapes, using Laplacian and geodesic distances.
result Effective monitoring of defects without registration and mesh reconstruction.
Focuses on monitoring and explaining models in real-world applications.
problem Ensuring high quality machine learning services in production environments.
method Statistical techniques for model performance and data monitoring, explanations of predictions.
result Challenges and solutions for implementing monitoring and explanation in production models.
Neural system optimizes glucose levels in diabetics.
problem Limited research on continuous glucose maintenance devices.
method Differential predictive control with neural policy and differentiable modeling.
result Improves glucose level optimization in real-time.
We present a numerical scheme to calculate fluctuation identities for exponential Lévy processes in the continuous monitoring case. This includes the Spitzer identities for touching a single upper or lower barrier, and the more difficult case of the two-barriers exit problem. These identities are given in the Fourier-L…
A fast Monte Carlo method for additive processes and option pricing.
problem Efficiently pricing path-dependent options with additive processes.
method Developed a fast Monte Carlo scheme for additive processes, analyzing and reducing numerical error sources.
result Shows significant reduction in error (1 bp or below) for pricing path-dependent options.
The relation between performance and stress is described by the Yerkes-Dodson Law but varies significantly between individuals. This paper describes a method for determining the individual optimal performance as a function of physiological signals. The method is based on attention and reasoning tests of increasing comp…
Optimal probing framework for scalable network monitoring.
problem Efficiently monitor growing cloud networks with limited budgets.
method A- and E-optimal experimental designs, Frank-Wolfe algorithm approximations.
result Significant reduction in probing budget with low estimation errors.
Machine learning predicts ship performance changes over time.
problem Estimating ship hydrodynamic performance over time.
method Machine learning methods (NL-PCR, NL-PLSR, probabilistic ANN) calibrated with in-service data.
result Probabilistic ANN model performs best in predicting ship performance changes.
The paper proposes a method to create efficient remote monitoring models.
problem Large and complex machine learning models are unsuitable for remote monitoring on edge devices.
method Decompose the model into a simple local monitoring function and a complex correction term evaluated on the server.
result The proposed framework learns monitoring models with significantly reduced complexity that maintain safety.
Emerging wearable sensors have enabled the unprecedented ability to continuously monitor human activities for healthcare purposes. However, with so many ambient sensors collecting different measurements, it becomes important not only to maintain good monitoring accuracy, but also low power consumption to ensure sustain…