Focuses on monitoring and explaining models in real-world applications.
problem Ensuring high quality machine learning services in production environments.
method Statistical techniques for model performance and data monitoring, explanations of predictions.
result Challenges and solutions for implementing monitoring and explanation in production models.
Paper optimizes a big data and ML risk monitoring system for financial markets.
problem Traditional risk monitoring methods are inadequate for modern financial markets due to data complexity and volume.
method Four-layer architecture integrating big data and advanced ML algorithms (LSTM, RF, GB).
result Significantly enhances efficiency and accuracy in risk management, especially in market crash risk detection.
Since 2006, deep learning (DL) has become a rapidly growing research direction, redefining state-of-the-art performances in a wide range of areas such as object recognition, image segmentation, speech recognition and machine translation. In modern manufacturing systems, data-driven machine health monitoring is gaining …
This research tackles monitoring machine learning algorithms post-deployment, addressing performativity issues.
problem Monitoring machine learning algorithms after deployment, especially when they affect their own data-generating process.
method Uses causal inference techniques to navigate performativity and compares different monitoring criteria and data sources.
result Different monitoring systems have varying operating characteristics and implications for ML monitoring design.
Amazon SageMaker Model Monitor detects drift in deployed ML models.
problem Ensuring high performance of ML models in production environments.
method Automatically detects data, concept, bias, and feature attribution drift in real-time.
result Maintains high quality models by providing alerts and corrective actions.
IUPM monitors machine learning models under gradual shifts using optimal transport and active labeling.
problem Gradual distribution shifts lead to unnoticed accuracy declines in machine learning models.
method Incremental Uncertainty-aware Performance Monitoring (IUPM) using optimal transport and active labeling.
result IUPM outperforms existing baselines in gradual shift scenarios and guides label acquisition more effectively.
A monitoring procedure improves machine learning forecasts for digital platforms.
problem Maintaining accurate and stable forecasts for data streams at digital platforms.
method Developed a monitoring procedure to determine when to retrain machine learning algorithms.
result Monitor-based retraining produces accurate forecasts compared to benchmarks.
The paper develops methods for monitoring TPL machine health.
problem Inaccurate and untimely maintenance of TPL systems leads to poor quality and inefficiencies.
method Physics-informed data-driven predictive models integrated with statistical approaches.
result The methods achieve high accuracy across various scenarios and conditions.
Proposes real-time risk monitoring for machine learning systems under unknown shifts.
problem Dynamic distribution shifts challenge real-world machine learning systems' risk assurances.
method Sequential hypothesis testing with 'testing by betting' to detect risk violations.
result Effective real-time risk monitoring under various unknown shifts.
New monitoring method detects ML risk models' performance changes in medical interventions.
problem Monitoring ML risk models in healthcare is complicated by confounding medical interventions.
method Developed a new score-based CUSUM monitoring procedure with dynamic control limits.
result Valid inference is possible if conditional exchangeability or time-constant selection bias hold.
Paper discusses ASD challenge for machine condition monitoring.
problem Detecting unknown anomalous sounds without labeled data.
method Design and evaluation of a large-scale ASD dataset, novel approaches.
result Several novel approaches developed, evaluation results analyzed.
A method to monitor probability predictions for calibration loss in image classification models.
problem Maintaining calibration in machine learning predictions over time.
method Cumulative sum-based approach with dynamic limits for early detection of miscalibration.
result Early detection of operational context changes impacting image classification performance.
Machine failures decrease up-time and can lead to extra repair costs or even to human casualties and environmental pollution. Recent condition monitoring techniques use artificial intelligence in an effort to avoid time-consuming manual analysis and handcrafted feature extraction. Many of these only analyze a single ma…
Study reviews machine learning techniques for stress monitoring.
problem Improving accuracy of stress monitoring devices.
method Reviewed machine learning techniques for various stress indicators.
result Choosing the right classifier depends on multiple factors.
MLDemon monitors ML systems post-deployment, improving reliability with real-time performance estimates and expert labels.
problem Ensuring reliability of machine learning systems post-deployment, especially when user inputs differ from training data.
method Integrates unlabeled and on-demand labeled data to monitor ML model performance in real-time, deciding when to acquire expert labels.
result Outperforms existing approaches in temporal datasets with diverse distribution drifts, providing theoretical optimality for distribution drifts.
The paper proposes a method to create efficient remote monitoring models.
problem Large and complex machine learning models are unsuitable for remote monitoring on edge devices.
method Decompose the model into a simple local monitoring function and a complex correction term evaluated on the server.
result The proposed framework learns monitoring models with significantly reduced complexity that maintain safety.
The significance of air pollution and the problems associated with it are fueling deployments of air quality monitoring stations worldwide. The most common approach for air quality monitoring is to rely on environmental monitoring stations, which unfortunately are very expensive both to acquire and to maintain. Hence e…
The paper monitors model deterioration using uncertainty estimation.
problem Challenges in monitoring deployed machine learning models.
method Non-parametric bootstrapped uncertainty estimates and SHAP values.
result The approach detects model deterioration and uncertainty effectively.
Paper proposes a method to quantify and explain machine learning uncertainty in predictive process monitoring.
problem Neglect of data-driven estimation, point forecasts without model uncertainty, and lack of explanations.
method Quantile Regression Forests for interval predictions and SHapley Additive Explanations for uncertainty.
result Effective handling of model uncertainty in predictive process monitoring.
Deep learning improves material recognition in construction monitoring.
problem Varying illuminations and low accuracy rates in material classification.
method Deep learning methods, including convolutional neural networks, were used to classify and recognize materials in construction sites.
result Achieved 97.35% accuracy rate for material classification.
As the technology is advancing, audio recognition in machine learning is improved as well. Research in audio recognition has traditionally focused on speech. Living creatures (especially the small ones) are part of the whole ecosystem, monitoring as well as maintaining them are important tasks. Species such as animals …
New method uses tensor decomposition to improve noise reduction in machine fault detection.
problem Noise in acoustic signals hinders fault detection in industrial machines.
method Non-negative Canonical Polyadic (CP) decomposition for denoising spectral data.
result Improvement in unsupervised anomaly detection for machine fault detection.
This paper discusses challenges and opportunities in vessel behavior detection using machine and deep learning.
problem Real-time analysis of vessel behaviors is crucial for maritime safety and protection.
method Comparison of classical machine learning and deep learning approaches for vessel event and anomaly detection.
result Novel methods and tools are needed to address challenges in vessel behavior detection.
Machine learning components such as deep neural networks are used extensively in Cyber-Physical Systems (CPS). However, they may introduce new types of hazards that can have disastrous consequences and need to be addressed for engineering trustworthy systems. Although deep neural networks offer advanced capabilities, t…
VarDetect monitors API queries to detect model extraction attacks.
problem Attackers can extract ML models through API queries.
method VarDetect uses a modified variational autoencoder to track and detect attacker samples.
result VarDetect successfully detects and raises alarms for model extraction attacks.
This paper studies an intelligent ultimate technique for health-monitoring and prognostic of common rotary machine components, particularly bearings. During a run-to-failure experiment, rich unsupervised features from vibration sensory data are extracted by a trained sparse auto-encoder. Then, the correlation of the ex…
Increasing population indicates that energy demands need to be managed in the residential sector. Prior studies have reflected that the customers tend to reduce a significant amount of energy consumption if they are provided with appliance-level feedback. This observation has increased the relevance of load monitoring …
DCASE 2022 Task 2 tackles domain shifts in ASD for machine condition monitoring.
problem Domain shifts change acoustic characteristics, affecting ASD performance.
method Domain generalization techniques to detect anomalies across unknown domains.
result Two types of domain generalization techniques were identified and analyzed.
In manufacture, steel and other metals are mainly cut and shaped during the fabrication process by computer numerical control (CNC) machines. To keep high productivity and efficiency of the fabrication process, engineers need to monitor the real-time process of CNC machines, and the lifetime management of machine tools…
Work addresses long-term accuracy issues in IoT air quality sensors.
problem Limited accuracy of IoT air quality sensors in long-term field deployments.
method Adaptive machine learning strategies for network calibration.
result Prolongs the validity of multisensor calibration models for continuous learning.
SafeML monitors ML systems for safety and security risks.
problem Ensuring safety and explainability of ML systems in safety-critical domains.
method Statistical difference measures of ECDF to detect distributional shifts.
result Approach can detect invalid application contexts of ML components.
The worldwide growth of maritime traffic and the development of the Automatic Identification System (AIS) has led to advances in monitoring systems for preventing vessel accidents and detecting illegal activities. In this work, we describe research gaps and challenges in machine learning for vessel behavior change and …
This paper reviews traditional and modern methods for detecting structural damage using vibrations.
problem Early warning of structural damage to maintain civil structures.
method Vibration-based methods and ML/DL algorithms.
result ML and DL algorithms show superior performance in detecting structural damage.
Enhances reward specification in RL with a novel language-based approach.
problem Reward specification in RL can lead to unintended, potentially harmful behaviours.
method Developed a novel class of language-based Reward Machines using RML's built-in memory.
result Can specify non-regular, non-Markovian reward functions for complex tasks.
The paper proposes a method to detect relevant model degradations without over-alerting.
problem Detecting meaningful changes in machine learning model performance over time.
method Sequential monitoring scheme accounting for temporal dependence and multiple testing issues.
result The proposed method outperforms benchmark methods in detecting relevant changes in model quality.
Infrastructure monitors AI/ML radiology models across multiple sites.
problem Monitoring and improving AI/ML radiology models across multiple sites.
method Interactive radiology reporting, centralized cloud system, post-marketing surveillance.
result Efficient monitoring and iterative development of AI/ML models without radiologist burden.
Risk-based active learning improves SHM decision-making.
problem Lack of prior labels for structural health monitoring.
method Risk-based active learning approach to guide data labeling.
result Improves decision-maker's performance in SHM.
A new method detects changes in machine learning models over time.
problem Automatic monitoring of machine learning models trained on evolving data.
method Score-based statistical hypothesis test for change detection.
result The method can detect changes in any number of model components.
Adaptive monitoring for AI systems detects and diagnoses shifts in data distribution.
problem Continuous monitoring of AI systems to detect and address unsafe behavior.
method Weighted-conformal martingales (WCTMs) for online monitoring of AI systems.
result Improved performance over state-of-the-art baselines on real-world datasets.
Dividing deep learning models for consistent anomaly detection in changing log data.
problem Anomaly detection methods fail when log data types change, leading to false negatives.
method Divide deep learning models based on log data correlation and extract correlations.
result Continues anomaly detection accuracy even when log data changes.
Study compares semi-supervised learning methods for anomaly detection in hydraulic systems.
problem Challenges in anomaly detection for hydraulic systems due to scarce labeled data.
method Compared various semi-supervised learning methods including traditional models and deep neural networks.
result Customized extreme learning machine (HELM) model achieved state-of-the-art performance.
Physics-informed model predicts beam stiffness and monitors structural health.
problem Predicting and monitoring the stiffness of Euler-Bernoulli beams.
method Physics-informed Gaussian process model using the Euler-Bernoulli beam equation.
result Model accurately predicts bending stiffness and detects structural damage.
This paper investigates how machine learning APIs change over time and proposes an efficient method to monitor these changes.
problem Understanding and assessing changes in machine learning APIs over time.
method Systematic investigation of ML API shifts, proposing a principled adaptive sampling algorithm (MASA) for efficient estimation of confusion matrix shifts.
result MASA can accurately estimate confusion matrix shifts using up to 90% fewer samples compared to random sampling.
SLIC-UAV monitors forest recovery using UAVs and machine learning.
problem Challenges in monitoring forest recovery, especially in logged tropical forests.
method Novel pipeline for UAV imagery analysis, combining crown labelling, species classification, and superpixel segmentation.
result SLIC-UAV achieves high accuracy in species mapping, from 79.3% to 90.5%.
Machine learning monitors detect motor overheating, adapting to concept drift.
problem Early detection of motor overheating in ships' propulsion systems.
method Machine learning and statistical methods using historical data to adapt to concept drift.
result The proposed monitors provide early detection of overheating during and after concept drifts.
The economic and social impact of poor air quality in towns and cities is increasingly being recognised, together with the need for effective ways of creating awareness of real-time air quality levels and their impact on human health. With local authority maintained monitoring stations being geographically sparse and t…
Study compares geostatistical and machine learning models for PM2.5 prediction.
problem Improving accuracy of hourly PM2.5 maps across California.
method Traditional geostatistical methods (kriging, land use regression) and machine learning models (neural networks, random forests, support vector machines) were evaluated.
result Ensemble model enhanced predictive accuracy of PM2.5 concentration by correcting PurpleAir data bias.
A new method detects anomalous sounds using self-supervised learning.
problem Detecting anomalous sounds in noisy data.
method Flow-based self-supervised density estimation using Normalizing Flows.
result The method improves anomaly detection performance significantly.