Paper proposes a method to monitor industrial processes under closed-loop control.
problem Difficulty distinguishing between real process faults and normal operating conditions changes.
method Develops a distributed monitoring system by capturing static and dynamic characteristics of large-scale closed-loop industrial processes.
result The method effectively distinguishes between real process faults and normal operating conditions changes.
MLDemon monitors ML systems post-deployment, improving reliability with real-time performance estimates and expert labels.
problem Ensuring reliability of machine learning systems post-deployment, especially when user inputs differ from training data.
method Integrates unlabeled and on-demand labeled data to monitor ML model performance in real-time, deciding when to acquire expert labels.
result Outperforms existing approaches in temporal datasets with diverse distribution drifts, providing theoretical optimality for distribution drifts.
PITMonitor monitors model calibration over time with formal error guarantees.
problem Fixed-sample tests applied to models over time can lead to false alarms.
method PITMonitor uses mixture e-processes to detect distributional shifts in probability integral transforms.
result PITMonitor achieves competitive detection rates on river's FriedmanDrift benchmark.
Adaptive monitoring for AI systems detects and diagnoses shifts in data distribution.
problem Continuous monitoring of AI systems to detect and address unsafe behavior.
method Weighted-conformal martingales (WCTMs) for online monitoring of AI systems.
result Improved performance over state-of-the-art baselines on real-world datasets.
KQT-EWMA monitors multivariate data streams online with flexible and practical change detection.
problem Online monitoring of multivariate data streams for detecting changes.
method Combines Kernel-QuantTree histogram and EWMA statistic for non-parametric monitoring.
result Controls Average Run Length (ARL0) while achieving comparable detection delays.
Risk monitoring detects when TTA models degrade at test time.
problem Detecting when TTA models degrade at test time.
method Extended risk monitoring tools based on sequential testing with confidence sequences.
result Demonstrated effectiveness of TTA monitoring framework across various datasets and methods.
New Thompson sampling algorithm for stochastic partial monitoring achieves logarithmic regret.
problem Limited feedback in sequential learning problems.
method Developed a novel Thompson-sampling-based algorithm to sample from the posterior distribution exactly.
result Achieved logarithmic regret bound of O(log T) for a linearized variant of the problem.
TonY simplifies distributed ML job management.
problem Managing distributed ML jobs is complex and resource-intensive.
method TonY is an open-source orchestrator for distributed ML jobs.
result TonY simplifies distributed ML job management.
Online monitor detects classifier drift and adapts predictions.
problem Silent degradation of classifier accuracy under distributional shift.
method Sliding-window KS statistic with calibrated alarm thresholds.
result 86.6% valid detection across various shift conditions.
The paper monitors model deterioration using uncertainty estimation.
problem Challenges in monitoring deployed machine learning models.
method Non-parametric bootstrapped uncertainty estimates and SHAP values.
result The approach detects model deterioration and uncertainty effectively.
Process Monitoring involves tracking a system's behaviors, evaluating the current state of the system, and discovering interesting events that require immediate actions. In this paper, we consider monitoring temporal system state sequences to help detect the changes of dynamic systems, check the divergence of the syste…
A new method uses BiGANs to assess operational risks in distribution networks.
problem Operational risk assessment in distribution networks using online monitoring data.
method Unsupervised approach based on BiGANs with adversarial feature learning and operational risk assessment.
result The approach provides more accurate risk assessment and discovers latent data structures.
Optimal probing framework for scalable network monitoring.
problem Efficiently monitor growing cloud networks with limited budgets.
method A- and E-optimal experimental designs, Frank-Wolfe algorithm approximations.
result Significant reduction in probing budget with low estimation errors.
Framework improves marine mammal monitoring in noisy underwater environments.
problem Underwater bioacoustic monitoring challenges due to overlapping calls and variable noise.
method Multi-step attention-guided framework with segmentation and mid-level fusion.
result Improved signal discrimination, reduced false positives, reliable representations.
IUPM monitors machine learning models under gradual shifts using optimal transport and active labeling.
problem Gradual distribution shifts lead to unnoticed accuracy declines in machine learning models.
method Incremental Uncertainty-aware Performance Monitoring (IUPM) using optimal transport and active labeling.
result IUPM outperforms existing baselines in gradual shift scenarios and guides label acquisition more effectively.
Proposes real-time risk monitoring for machine learning systems under unknown shifts.
problem Dynamic distribution shifts challenge real-world machine learning systems' risk assurances.
method Sequential hypothesis testing with 'testing by betting' to detect risk violations.
result Effective real-time risk monitoring under various unknown shifts.
SafeML monitors ML systems for safety and security risks.
problem Ensuring safety and explainability of ML systems in safety-critical domains.
method Statistical difference measures of ECDF to detect distributional shifts.
result Approach can detect invalid application contexts of ML components.
A method models continuous-time glucose distributions in children with diabetes.
problem Capturing subtle temporal changes in glucose distributions.
method Probabilistic framework using Gaussian mixtures and neural ODEs.
result Detects treatment-related improvements in glucose dynamics.
This research tackles group fairness in predictive process monitoring by ensuring predictions are independent of sensitive group membership.
problem Predictive models using biased historical data can perpetuate unfair behavior in new cases.
method Investigates independence through metrics like ΔDP and a composite loss function balancing predictive performance and fairness.
result Proposes and validates a composite loss function for training models that balance fairness and performance.
Neural networks compress and sample WDN contamination dynamics efficiently.
problem Infrastructure monitoring of complex, networked systems like water distribution networks is expensive and challenging.
method Developed Graph Fourier Transform (GFT) operators and neural networks (NN) for efficient data collection and inference.
result High accuracy reconstruction of contamination dynamics using only 5-10% of the sample set.
Infrastructure monitors AI/ML radiology models across multiple sites.
problem Monitoring and improving AI/ML radiology models across multiple sites.
method Interactive radiology reporting, centralized cloud system, post-marketing surveillance.
result Efficient monitoring and iterative development of AI/ML models without radiologist burden.
Modeling glucose distribution changes over time using neural ODEs.
problem Analyzing how continuous glucose distribution changes over time in diabetic patients.
method Combines Gaussian mixture, MMD, and Neural ODE to model temporal evolution of glucose distribution.
result Highly interpretable model detects subtle distribution shifts and remains computationally efficient.
Optimizes seismic monitoring networks using Bayesian OED.
problem Improve seismic event identification and location.
method Bayesian optimal experimental design (OED) to configure sensor networks.
result Optimized sensor network improves seismic event identification and location.
Framework for monitoring ML model performance in production without labels.
problem Monitoring real-time prediction quality of ML models in production without labels.
method ML Health framework using diagnostic methods to generate alerts for further investigation.
result The method outperforms standard distance metrics at detecting issues with mismatched data sets.
New MMD estimators detect differences in missing paired data.
problem Handling missing data in matched pairs with complex distributions.
method Maximum mean discrepancy (MMD) estimators for complex data with missing values.
result Valid and consistent estimators detect differences in data distributions.
Method monitors neural networks using topological properties.
problem Detecting data shifts and out-of-distribution samples.
method Topological Uncertainty based on activation graphs.
result Validates network predictions without retraining.
A framework monitors and diagnoses concept drift in supervised learning models.
problem Changes in predictive relationships over time render models suboptimal.
method Score vector monitoring using exponentially weighted moving average.
result Score-based approach detects concept drift more effectively than error-based methods.
New method for sensing non-planar surfaces using ERT.
problem Limited computational techniques for planar surfaces in ERT-based sensing skins.
method Generalized ERT to non-planar surfaces using Riemannian geometry.
result Feasibility and applicability of ERT-based sensing skins for non-planar geometries demonstrated.
VarDetect monitors API queries to detect model extraction attacks.
problem Attackers can extract ML models through API queries.
method VarDetect uses a modified variational autoencoder to track and detect attacker samples.
result VarDetect successfully detects and raises alarms for model extraction attacks.
New method detects anomalies in time series data, especially useful for monitoring services.
problem Detecting anomalies in time series data, especially for monitoring services and cloud resources.
method Models time series of probability distributions over real values, scales to millions of time series.
result Outperforms state-of-the-art methods in detecting anomalies on various data sets.
Optimizes state monitoring in Markovian systems with cost constraints.
problem Balancing state queries with prediction costs in Markovian systems.
method Greedy policy and SGD-based learning variant for optimal predict-query tradeoff.
result Greedy policy is suboptimal but performs close to optimal under certain conditions.
Framework monitors insurance pricing models for drift and recalibration.
problem Maintaining predictive performance of pricing models in evolving insurance portfolios.
method Formalizes deviance loss and Murphy's score, studies Gini score, develops monitoring framework.
result Framework guides decisions on refitting or recalibrating pricing models.
Partial monitoring is a general model for sequential learning with limited feedback formalized as a game between two players. In this game, the learner chooses an action and at the same time the opponent chooses an outcome, then the learner suffers a loss and receives a feedback signal. The goal of the learner is to mi…
Work addresses long-term accuracy issues in IoT air quality sensors.
problem Limited accuracy of IoT air quality sensors in long-term field deployments.
method Adaptive machine learning strategies for network calibration.
result Prolongs the validity of multisensor calibration models for continuous learning.
This research develops efficient surrogate models for predicting crack growth in metal structures.
problem Accurately predicting crack growth in metal structures under uncertainty.
method Employing Gaussian Process (GP) regression models for latent variable modeling to create probabilistic surrogate models.
result Surrogate models successfully encode material and load-related uncertainties in stochastic crack growth processes.
We introduce a methodology for efficient monitoring of processes running on hosts in a corporate network. The methodology is based on collecting streams of system calls produced by all or selected processes on the hosts, and sending them over the network to a monitoring server, where machine learning algorithms are use…
AI systems that explain their decisions can be monitored for harmful intentions.
problem Monitoring AI systems' decision-making processes for harmful intentions is imperfect and can miss some misbehavior.
method Monitoring the chain of thought (CoT) of AI systems that communicate in human language.
result CoT monitoring is a promising but fragile approach to AI safety.
Efficiently audits model fairness with continuous monitoring and flexible data collection.
problem Continuous monitoring and flexible data collection for fairness auditing.
method Sequential, anytime-valid inference and game-theoretic statistics.
result Demonstrated efficacy on three fairness datasets.
Study compares statistical properties and power of divergence measures for credit risk monitoring.
problem Detecting distributional shifts in credit risk models.
method Derives statistical properties and chi-square benchmark values for Jensen-Shannon Divergence and Kullback-Leibler Divergence, demonstrating their applicability in credit risk monitoring.
result Jensen-Shannon Divergence and Kullback-Leibler Divergence follow chi-square distributions and reveal practical trade-offs in minimizing false positives vs. detecting changes.
New method for multivariate distribution regression using NPT metric.
problem Regression with multivariate distributional responses and Euclidean predictors.
method Fréchet regression with nonparanormal transport (NPT) metric.
result Efficient estimation and granular interpretation of predictor effects.
Two methods monitor high-dimensional processes via manifold fitting or learning.
problem Monitoring high-dimensional, dynamic industrial processes.
method Manifold fitting and learning approaches for online SPC.
result Manifold-fitting approach achieves performance competitive with classical methods.
Study improves data quality assessment for structural monitoring data.
problem Ensuring reliability of structural health monitoring data.
method Probabilistic data quality assessment using a conditional diffusion model.
result Significantly improves accuracy of data quality assessment.
A drone-based MOT algorithm tracks vehicles using neural network detections and TPMBM filter.
problem Tracking multiple vehicles from drone-mounted cameras.
method Neural network for object detection, TPMBM filter for trajectory estimation, von-Mises Fisher distribution for DOA.
result TPMBM filter optimally estimates vehicle trajectories.
This work bridges outlier and drift detection by comparing inputs to a part of the reference distribution.
problem Monitoring machine learning models to ensure they operate within their validated distribution.
method Comparing a set of inputs to a chosen part of the reference distribution.
result A new approach that bridges outlier detection and drift detection.
Simple online monitor detects unsafe LLM outputs.
problem LLMs generate unsafe outputs despite training.
method Thresholding external verifier signal to decide alarms.
result Simple design competitive with advanced methods.
In this paper, we show an approach to build deep learning algorithms for recognizing signals in distributed fiber optic monitoring and security systems for long perimeters. Synthesizing such detection algorithms poses a non-trivial research and development challenge, because these systems face stringent error (type I a…
Physics-informed model predicts beam stiffness and monitors structural health.
problem Predicting and monitoring the stiffness of Euler-Bernoulli beams.
method Physics-informed Gaussian process model using the Euler-Bernoulli beam equation.
result Model accurately predicts bending stiffness and detects structural damage.
This research tackles monitoring machine learning algorithms post-deployment, addressing performativity issues.
problem Monitoring machine learning algorithms after deployment, especially when they affect their own data-generating process.
method Uses causal inference techniques to navigate performativity and compares different monitoring criteria and data sources.
result Different monitoring systems have varying operating characteristics and implications for ML monitoring design.