A framework monitors and diagnoses concept drift in supervised learning models.
problem Changes in predictive relationships over time render models suboptimal.
method Score vector monitoring using exponentially weighted moving average.
result Score-based approach detects concept drift more effectively than error-based methods.
Proposes and evaluates three diagnostic graphics for probabilistic classifiers.
problem Evaluating and comparing probabilistic classifiers.
method Triptych of diagnostic graphics: reliability diagram, ROC curve, Murphy diagram.
result Visual diagnostics reveal distinct aspects of forecast performance.
It has been argued that in supervised classification tasks, in practice it may be more sensible to perform model selection with respect to some more focused model selection score, like the supervised (conditional) marginal likelihood, than with respect to the standard marginal likelihood criterion. However, for most Ba…
Trimming helps in conformal prediction when it separates anomaly scores.
problem Effectiveness of trimming in conformal prediction under contamination.
method Analyse fixed-threshold trimming as a replacement of the contaminated calibration law with a retained law.
result Trimming helps when it separates anomaly scores, reducing clean-target coverage to a one-dimensional score-CDF transfer problem.
Proposes PRMs for interpreting financial risk concept drift.
problem Concept drift in high-stakes predictions like credit risk.
method Probabilistic Rule Models (PRMs) using Markov Logic Networks.
result Interpretable rules explain borrower risk changes.
Extends dimension reduction to data-driven settings without gradients.
problem Gradient-based dimension reduction limitations in data-driven settings.
method Score ratio matching framework, tailored parameterization, regularization, eigenvalue deflation.
result Outperforms standard score-matching for problems with low-dimensional structure.
Paper develops robust econometric methods for staggered adoption studies.
problem Estimation challenges in event studies with staggered adoption.
method Design-first framework with exact probability limits, diagnostics, and orthogonal score constructions.
result Uniformly valid inference under restricted violations of parallel trends.
The paper breaks down AUC into cluster-level components for better model diagnostics.
problem Global AUC masks weaknesses in specific subpopulations, leading to financial or operational risks.
method Formal decomposition of AUC into intra- and inter-cluster components, comparing with other performance metrics.
result Allows practitioners to evaluate and diagnose model performance within and across clusters.
We study the problem of interpreting trained classification models in the setting of linguistic data sets. Leveraging a parse tree, we propose to assign least-squares based importance scores to each word of an instance by exploiting syntactic constituency structure. We establish an axiomatic characterization of these i…
BoC probe assesses neural network confidence coherence, revealing architecture-specific uncertainty.
problem Poor calibration and OOD detection in neural networks.
method Bag-of-Coins (BoC) probe compares softmax confidence to pairwise dominance probabilities.
result BoC reveals clear ID/OOD separation for some architectures but not others.
ABC improves uncertainty quantification in LLMs for clinical diagnostics.
problem Overconfident and poorly calibrated estimates of LLMs in clinical domains.
method Approximate Bayesian Computation (ABC) for likelihood-free inference.
result Improves accuracy by up to 46.9%, reduces Brier scores by 74.4%, and enhances calibration.
The multi-scale, mutli-physics nature of fusion plasmas makes predicting plasma events challenging. Recent advances in deep convolutional neural network architectures (CNN) utilizing dilated convolutions enable accurate predictions on sequences which have long-range, multi-scale characteristics, such as the time-series…
Language models fail to process hallucinated responses, and this study diagnoses the failure.
problem Language models fail to process hallucinated responses, leading to over-concentration or diffuse attention.
method The study uses forced scoring of benchmark-labeled responses to compute attention shapes and analyze the symmetric component of the degree-normalized attention operator.
result The study proves that every transpose-invariant spectral diagnostic of the attention operator is orientation-blind and bounds the sensitivity of any Lipschitz diagnostic by the asymmetry coefficient \(G\).
Improves visualization of high-dimensional data by correcting misleading artifacts in neighbor embedding methods.
problem Misleading visual artifacts in t-SNE and UMAP due to lack of data-independent manifold learning interpretations.
method LOO-map framework that extends embedding maps to the entire input space, identifying and correcting map discontinuities.
result Developed point-wise diagnostic scores to detect unreliable embedding points and improve hyperparameter selection.
We propose a method for maximizing a partial area under a receiver operating characteristic (ROC) curve (pAUC) for binary classification tasks. In binary classification tasks, accuracy is the most commonly used as a measure of classifier performance. In some applications such as anomaly detection and diagnostic testing…
Critiques binary classification evaluation methods, advocating for proper scoring rules.
problem The dominance of top-K metrics and fixed-threshold evaluations in machine learning.
method Introduces a decision-theoretic framework mapping evaluation metrics to their use cases, and implements a clipped Brier score variant.
result Demonstrates the clinical utility of proper scoring rules through a Python package, exttt{briertools}.
The study examines when to trust confidence thresholding in pseudo-labelling regression.
problem Calibrated probabilities from classifiers used for pseudo-labelling need careful handling to avoid bias in downstream regression.
method Developed a diagnostic apparatus to predict and bound the bias induced by confidence thresholding, derived a closed-form expression for the attenuation bias.
result The bias can be predicted from the residual score variance V∗, motivating a structural separation between classifier features and downstream controls. Active learning aims to obtain a classifier of high accuracy by using fewer label requests in comparison to passive learning by selecting effective queries. Many active learning methods have been developed in the past two decades, which sample queries based on informativeness or representativeness of unlabeled data poi…
SSDMs generate quantum states directly, outperforming classical methods.
problem Generating pure-state quantum representations efficiently.
method Score-based generative model on complex projective manifold.
result SSDMs match target pure-state ensembles by orders of magnitude.
Machine learning models have spread to almost every area of life. They are successfully applied in biology, medicine, finance, physics, and other fields. With modern software it is easy to train even a~complex model that fits the training data and results in high accuracy on the test set. The problem arises when models…
A new framework separates classifier calibration and discrimination.
problem Combining reliability and resolution in probabilistic predictions.
method Manokhin Probability Matrix separates reliability and resolution using Spiegelhalter Z-statistic and AUC-ROC.
result Classifiers are categorized into four archetypes: Eagle, Bull, Sloth, and Mole.
Improves reliability diagrams for probabilistic forecasts.
problem Lack of stability in reliability diagrams hampered their use.
method CORP approach using non-parametric isotonic regression and PAV algorithm.
result Improved reliability diagrams with statistical consistency and reproducibility.
CJE calibrates cheap LLM judges against an oracle, achieving high accuracy at a fraction of the cost.
problem Inexpensive LLM judges can produce biased rankings, leading to unreliable outcomes.
method CJE uses a small oracle to calibrate cheap scores, then evaluates at scale with valid uncertainty.
result CJE achieves 99% pairwise ranking accuracy at 14x lower cost compared to a 16x oracle/judge cost ratio.
Signature kernel scoring rule improves weather forecasting by capturing temporal and spatial dependencies.
problem Lack of suitable scoring rules for probabilistic weather forecasting.
method Reframe weather variables as continuous paths using iterated integrals (signature kernels) to capture temporal and spatial dependencies.
result Signature kernel scoring rule outperforms conventional methods in weather forecasting, especially for long-term forecasts.
Study finds AI can predict diverse cardiac and non-cardiac diagnoses from a single ECG.
problem Narrow focus of ECG analysis models for diverse medical conditions.
method Exploratory study using a single AI model to predict multiple ICD codes.
result Model achieved AUROC scores > 0.8 for 253 cardiac and 172 non-cardiac diagnoses.
GH-PID uses guided harmonic paths for efficient SOT with interpretable diagnostics.
problem Efficiently solving Stochastic Optimal Transport with hard terminal distributions and soft costs.
method Guided Harmonic Path-Integral Diffusion (GH-PID) framework with low-dimensional guidance.
result GH-PID generates geometry-aware, cost-reducing trajectories that match terminal distributions.
Aircraft engine manufacturers collect large amount of engine related data during flights. These data are used to detect anomalies in the engines in order to help companies optimize their maintenance costs. This article introduces and studies a generic methodology that allows one to build automatic early signs of anomal…
AdaPID optimizes diffusion-based samplers by dynamically adjusting schedules.
problem Optimizing the intermediate-time dynamics in diffusion-based samplers.
method Develops a time-varying stiffness schedule using Piece-Wise-Constant (PWC) parametrizations and a hierarchical refinement approach.
result QoS-driven PWC schedules consistently improve sampling fidelity and accuracy.
In medicine, visualizing chromosomes is important for medical diagnostics, drug development, and biomedical research. Unfortunately, chromosomes often overlap and it is necessary to identify and distinguish between the overlapping chromosomes. A segmentation solution that is fast and automated will enable scaling of co…
The conditional-mean barrier helps diagnose deterministic surrogates missing uncertainty.
problem Uncertainty in deterministic surrogates for complex systems.
method Developed diagnostics to locate the conditional-mean barrier and prove its necessity for distributional objectives.
result Crossing the barrier requires a loss that scores distributions, not point predictions.
Bias in training data affects diagnostic algorithms' performance.
problem Bias in training data leads to biased predictions in diagnostic algorithms.
method Survey of MICCAI 2018 proceedings, analysis of a skin lesions dataset, adversarial training setup.
result Classifier performance varies significantly between subgroups based on age and sex.
Improved 3D ECG feature attributions for clinical interpretation.
problem Lack of interpretability in deep learning models for 12-lead ECG analysis.
method Cross-modal mapping of feature attributions from 12-lead ECG models onto CineECG 3D space.
result Mapped feature attributions yield higher Dice scores than standard 12-lead attributions.
Applying machine learning in the health care domain has shown promising results in recent years. Interpretable outputs from learning algorithms are desirable for decision making by health care personnel. In this work, we explore the possibility of utilizing causal relationships to refine diagnostic prediction. We focus…
Establishes statistical and computational bounds for influence diagnostics.
problem Identifying influential datapoints or subsets in machine learning models.
method Finite-sample statistical bounds and computational complexity for influence functions and approximate maximum influence perturbations.
result Established statistical and computational guarantees for influence diagnostics.
The article explains how to estimate confusion matrices for classifiers using unlabeled data.
problem Estimating sensitivity and specificity of binary medical diagnostic tests without gold standard tests.
method Modifying diagnostic test solutions to estimate confusion matrices for classifiers on unlabeled data.
result The approach can be used to estimate accuracy statistics for supervised or unsupervised binary classifiers on unlabeled data.
New score helps choose PIML model parameters, reducing ambiguity in model quality.
problem Ambiguity in measuring model quality in PIML due to multi-objective fitting.
method Introduces Physics-Informed Log Evidence (PILE) score in Gaussian process framework.
result PILE minimizes ambiguity in model selection, improving hyperparameter choices.
The debate of what quantitative risk measure to choose in practice has mainly focused on the dichotomy between Value at Risk (VaR) -- a quantile -- and Expected Shortfall (ES) -- a tail expectation. Range Value at Risk (RVaR) is a natural interpolation between these two prominent risk measures, which constitutes a trad…
TRACE analyzes risk changes in models trained on shifted data.
problem Understanding performance changes when a model trained on shifted data is used.
method TRACE framework decomposes risk change into four factors: generalization gaps, model change penalty, and covariate shift penalty.
result TRACE provides a diagnostic tool to understand and quantify risk changes due to covariate shift.
Celiac Disease (CD) is a chronic autoimmune disease that affects the small intestine in genetically predisposed children and adults. Gluten exposure triggers an inflammatory cascade which leads to compromised intestinal barrier function. If this enteropathy is unrecognized, this can lead to anemia, decreased bone densi…
Deep learning system diagnoses AVNFH from plain radiographs.
problem Challenging AVNFH diagnosis from plain radiographs.
method Deep convolutional neural networks for end-to-end diagnosis.
result AVN-net achieves state-of-the-art AUC of 0.97 in AVNFH detection.
AI enhances cancer diagnostics using spectroscopy.
problem Early and accurate cancer diagnosis.
method Combining AI with spectroscopy-based techniques.
result AI improves cancer diagnostics speed and safety.
Paper introduces a diagnostic for approximate inference methods.
problem Estimating errors in probabilistic inference algorithms, especially for approximate methods.
method Repeatedly simulate datasets from the prior and perform inference on each, estimating a symmetric KL-divergence.
result A diagnostic for approximate inference methods can be estimated using symmetric KL-divergence.
Machine learning improves diagnostic test accuracy for bovine tuberculosis.
problem Improving diagnostic test sensitivity for bovine tuberculosis.
method Machine learning to assess risk landscapes and predict infection.
result Test sensitivity improved, detecting 240 more infected herds per year.
AI predicts medical specialty diagnostic choices from EHR records.
problem Predicting timely medical specialty diagnostic workups for patients.
method Ensemble of feed-forward neural networks trained on EHR data.
result Significantly higher accuracy compared to traditional checklists.
Sparse point observations can provide useful local constraints, but their benefit for radar-like fields depends on the training loss, uncertainty representation, and how observation support is encoded in the model.
problem Improving dense radar-field forecasts with sparse point observations
method A multimodal graph neural network nowcasting system over the Nordic radar domain
result Each source improves a different part of the forecast problem
IGSD separates task-specific content channels in transformer components by comparing activation replacement with zero ablation.
problem Mechanistic interpretability of transformer components
method IGSD: paired-intervention framework for comparing activation replacement with zero ablation
result IGSD identifies an early-layer content channel in transformer components that standard importance methods underestimate.
ReGEN-TAD detects anomalies in financial time series with interpretable models.
problem Detecting anomalies in complex financial time series with high-dimensional data.
method Integrates machine learning with econometric diagnostics in a refined convolutional--transformer architecture.
result Unified anomaly score without labeled data, robust to structured deviations.
This paper argues against using calibration metrics for assessing posterior probabilities and proposes expected proper scoring rules instead.
problem The assessment of posterior probabilities generated by machine learning classifiers using calibration metrics is flawed and should be replaced with expected proper scoring rules.
method The paper reviews proper scoring rules from a practical perspective, explains why expected PSRs are a principled measure of posterior quality, and introduces a new calibration metric called calibration loss.
result Calibration loss is superior to expected calibration error and expected score divergence calibration metrics for assessing posterior probabilities.