This dissertation tackles challenges in reliable machine learning measurement.
problem Challenges in reproducibility, scalability, and uncertainty quantification in machine learning.
method Develops criteria for meaningful metrics and methodologies for scalable, reliable measurement.
result Provides methods for evaluating generative-AI systems and quantifying memorization.
New metrics assess reliability in RL algorithms.
problem Lack of reliability in reinforcement learning algorithms.
method Design and implementation of metrics to quantify reliability.
result Metrics and statistical tests enable rigorous comparisons of RL algorithm reliability.
In many instances, information on engineering systems can be obtained through measurements, monitoring or direct observations of system performances and can be used to update the system reliability estimate. In structural reliability analysis, such information is expressed either by inequalities (e.g. for the observati…
Online knowledge repositories typically rely on their users or dedicated editors to evaluate the reliability of their content. These evaluations can be viewed as noisy measurements of both information reliability and information source trustworthiness. Can we leverage these noisy evaluations, often biased, to distill a…
Paper explores physics-informed deep learning for system reliability assessment.
problem Limited study on deep learning for system reliability assessment.
method Physics-informed deep learning approach for system reliability assessment.
result Physics-informed deep learning can alleviate computational challenges and combine measurement data and mathematical models.
A new score measures data reliability without ground truth.
problem Assessing reliability of datasets without access to ground truth.
method Define ground-truth-based orderings and propose Gram determinant score.
result Gram determinant score effectively captures data quality across diverse observation processes.
BeMF improves recommendation reliability in recommender systems.
problem Improving reliability in recommender systems beyond accuracy.
method Bernoulli Matrix Factorization (BeMF) for model-based collaborative filtering.
result BeMF selects more reliable predictions, improving recommendation quality.
This paper studies the classification of high-dimensional Gaussian signals from low-dimensional noisy, linear measurements. In particular, it provides upper bounds (sufficient conditions) on the number of measurements required to drive the probability of misclassification to zero in the low-noise regime, both for rando…
MAntRA combines machine learning and Bayesian methods for time-dependent reliability analysis of unknown systems.
problem Time-dependent reliability analysis of systems with unknown governing physics.
method Combines machine learning, Bayesian statistics, and stochastic integration to discover and analyze SDEs from data.
result Demonstrates the effectiveness of MAntRA on three numerical examples, indicating its potential for in-situ and heritage structure analysis.
Economics tool predicts failure times in reliability systems.
problem Predicting optimal failure times in weighted k-out-of-n reliability systems with heterogeneous component failure.
method Using rational expectations to analyze and predict failure times in reliability systems with heterogeneous component failure.
result Different measures are optimal for predicting system failure depending on component failure distributions.
The paper improves NN predictions with reliable confidence intervals.
problem Creating accurate prediction intervals for neural network outputs.
method Introduces Conformal Prediction (CP) for NNs, ensuring reliable confidence measures.
result The proposed method produces well-calibrated and tight prediction intervals.
ReaPER improves learning efficiency by prioritizing reliable experiences.
problem Inefficient sampling of past experiences in reinforcement learning.
method Introducing a novel measure of reliability to prioritize experiences in PER.
result ReaPER outperforms PER in various environments, including Atari-10.
With the wide development of black-box machine learning algorithms, particularly deep neural network (DNN), the practical demand for the reliability assessment is rapidly rising. On the basis of the concept that `Bayesian deep learning knows what it does not know,' the uncertainty of DNN outputs has been investigated a…
The paper decomposes probabilistic scores into reliability, uncertainty, and information loss.
problem Understanding the reliability and uncertainty of probabilistic predictions.
method Developed decomposition identities for proper losses, quantifying reliability, residual uncertainty, and information gain.
result A three-term identity for classification scores, revealing miscalibration, grouping term, and feature-level uncertainty.
Measurement noise limits the advantage of nonlinear models over linear models in biomedical prediction
problem Nonlinear models vs. linear models in biomedical prediction
method Measurement reliability
result Measurement noise blurs the population-optimal predictor
This post introduces model calibration and evaluation measures, highlighting issues with a common measure.
problem Ensuring model confidence accurately reflects true outcomes.
method Explains common calibration definition, ECE, and its drawbacks.
result New evaluation measures needed for comprehensive model calibration.
The main aim of this paper is to inspect the properties of survey based on households inflation expectations, conducted by Reserve Bank of India. It is theorized that the respondents answers are exaggerated by extreme response bias. Latent class analysis has been hailed as a promising technique for studying measurement…
Improves reliability diagrams for probabilistic forecasts.
problem Lack of stability in reliability diagrams hampered their use.
method CORP approach using non-parametric isotonic regression and PAV algorithm.
result Improved reliability diagrams with statistical consistency and reproducibility.
We consider the problems of detection and localization of a contiguous block of weak activation in a large matrix, from a small number of noisy, possibly adaptive, compressive (linear) measurements. This is closely related to the problem of compressed sensing, where the task is to estimate a sparse vector using a small…
New metric assesses reliability of AI explanations.
problem Unreliable AI explanations under realistic conditions.
method Explanation Reliability Index (ERI) metrics quantifying stability under four axioms.
result Widespread reliability failures in popular explanation methods.
We proposed a probabilistic approach to joint modeling of participants' reliability and humans' regularity in crowdsourced affective studies. Reliability measures how likely a subject will respond to a question seriously; and regularity measures how often a human will agree with other seriously-entered responses coming…
Proposes CCE to assess point-wise reliability of neural network predictions.
problem Overconfidence and misaligned predictive distributions in neural networks.
method Introduces Conditional Congruence (CCE) metric using conditional kernel mean embeddings.
result CCE exhibits correctness, monotonicity, reliability, and robustness in high-dimensional regression tasks.
In many scientific tasks we are interested in discovering whether there exist any correlations in our data. This raises many questions, such as how to reliably and interpretably measure correlation between a multivariate set of attributes, how to do so without having to make assumptions on distribution of the data or t…
New method improves reliability of LDA topic modeling by assessing stability across replicated runs.
problem LDA's reproducibility issues due to initial values and Gibbs sampling.
method Cluster replicated LDA runs using modified Jaccard coefficient and pruning algorithm.
result New measure S-CLOP quantifies LDA topic stability, improving reproducibility.
Study efficient interactive learning for structured outputs with reliable computation.
problem Interactive learning with noisy labels and structured output spaces.
method Identify and utilize CRISPs (probabilistic models) that guarantee reliable and efficient computation of probabilistic quantities.
result CRISPs enable robust and efficient active and skeptical learning in large structured output spaces.
In the semantic segmentation of street scenes the reliability of the prediction and therefore uncertainty measures are of highest interest. We present a method that generates for each input image a hierarchy of nested crops around the image center and presents these, all re-scaled to the same size, to a neural network …
The paper proposes a method to assess when automated predictions are reliable.
problem Ensuring reliability and safety of automated decision-making in machine learning.
method Clustering to measure distances between outputs and class centroids, defining a safety threshold based on these distances.
result The proposed metric can efficiently determine when automated predictions are acceptable and when they should be deferred.
New method relaxes optimization problems to find solutions more reliably.
problem Optimizing functions with stochastic or non-differentiable elements.
method Using measure theory and Fourier analysis to impose structure on optimization problems.
result Consistency of optimal values, Lipschitzness of gradients, and convexity are key traits for fast and reliable optimization.
New dispersion indices based on inaccuracy and divergence introduced for information measures.
problem Measuring variability in uncertainty measures.
method Introducing new dispersion indices based on Kerridge inaccuracy and Kullback-Leibler divergence.
result Properties, bounds, and examples of new dispersion indices presented.
TCE measures calibration error with a test-based approach.
problem Measuring calibration error of probabilistic binary classifiers.
method TCE uses a novel loss function based on a statistical test.
result TCE offers clear interpretation, consistent scale, and enhanced visual representation.
A new perfectly truthful calibration measure improves prediction reliability.
problem Improving the reliability of predictions by ensuring they are conditionally unbiased.
method Designing a simple, perfectly truthful calibration measure called ATB.
result ATB is the first perfectly truthful calibration measure in the batch setting.
New method evaluates visual explanations of deep models using adversarial perturbations.
problem Lack of objective evaluation of visual explanations of deep models.
method Proposes an adversarial perturbation approach to evaluate visual explanations of deep models.
result Demonstrates the effectiveness of the proposed approach through comparisons with existing methods.
New metrics improve reliability of image generation evaluation.
problem Lack of reliable metrics for evaluating fidelity and diversity in generative models.
method Proposed density and coverage metrics to diagnose fidelity and diversity separately.
result Density and coverage metrics provide more interpretable and reliable signals.
COPP provides reliable intervals for outcomes under a new policy in contextual bandits.
problem Lack of reliable predictive intervals for outcomes under a new policy in contextual bandits.
method Conformal prediction applied to contextual bandits.
result COPP provides finite-sample guarantees without additional assumptions.
Paper introduces conformal prediction for reliable uncertainty quantification in landmark localization.
problem Systematic underestimation of total predictive uncertainty in landmark localization.
method Conformal prediction framework for multi-output regression, generating flexible prediction regions.
result Methods outperform existing approaches in validity and efficiency across 2D and 3D datasets.
Develops methods to improve reliability of deep learning for autonomous driving.
problem Safety concerns in deploying autonomous driving systems.
method Introduces a new criterion (true class probability) for estimating model confidence and learns it from data.
result Proposed method provides better failure prediction than current uncertainty measures.
The aim of the present paper is to develop a strategy for solving reliability-based design optimization (RBDO) problems that remains applicable when the performance models are expensive to evaluate. Starting with the premise that simulation-based approaches are not affordable for such problems, and that the most-probab…
This paper studies graphical model selection, i.e., the problem of estimating a graph of statistical relationships among a collection of random variables. Conventional graphical model selection algorithms are passive, i.e., they require all the measurements to have been collected before processing begins. We propose an…
Feature selection can facilitate the learning of mixtures of discrete random variables as they arise, e.g. in crowdsourcing tasks. Intuitively, not all workers are equally reliable but, if the less reliable ones could be eliminated, then learning should be more robust. By analogy with Gaussian mixture models, we seek a…
The paper introduces recklessness to improve recommendation quality and quantity.
problem The reliability/coverage dilemma in recommender systems limits the number of recommended items.
method Incorporates a new term (recklessness) into matrix factorization-based recommender systems to address the dilemma.
result Recklessness improves the quantity and quality of recommendations by allowing for risk regulation.
Rating platforms enable large-scale collection of user opinion about items (products, other users, etc.). However, many untrustworthy users give fraudulent ratings for excessive monetary gains. In the paper, we present FairJudge, a system to identify such fraudulent users. We propose three metrics: (i) the fairness of …
The paper assesses quality measures for machine learning models using cross-validation.
problem Evaluating the accuracy and robustness of quality measures for machine learning models.
method Cross-validation approach to estimate prediction error and quantify explained variation. Confidence bounds and local quality measures derived from residuals.
result The reliability and robustness of quality measures are assessed through numerical examples and confidence bounds.
The paper proposes a method to improve random forest classification accuracy by weighting trees based on their decision path reliability.
problem Random forests' uniform voting fails to correct errors in regions where incorrect tree representations outnumber correct ones.
method The paper introduces using the structural pattern of each tree's decision path as an instance-adaptive reliability signal to identify and weight more reliable trees.
result Using the proposed method yields a statistically significant accuracy improvement over RF on 36 binary classification benchmarks.
We introduce a measure to quantify ambiguity in deep learning models, improving their reliability.
problem Deep learning models make mistakes on seemingly trivial cases and fail in recognizing what they don't know.
method We define ambiguity based on decision boundaries and convex hulls in feature space, developing a theoretical framework to identify unknowns.
result A single ambiguity measure can detect a significant portion of model mistakes, including adversarial and out-of-distribution inputs.
Paper proposes real-time VaR estimation using quantile regression forest with conformal calibration.
problem Real-time estimation of Value at Risk (VaR) in rapidly changing markets.
method Quantile regression forest trained offline, real-time VaR estimates via observed risk factors, conformalized estimator for reliability.
result The proposed method provides reliable real-time VaR estimates.
Paper proposes MUCS for more reliable TDA in diffusion models.
problem Current TDA approaches lack reliability and robustness.
method Mirrored unlearning and noise-consistent skew (MUCS).
result MUCS outperforms existing methods on three datasets.
This study introduces axioms to assess regression uncertainty measures.
problem Limited formal justification and evaluations of uncertainty measures in regression settings.
method Introduces axioms and analyzes entropy- and variance-based measures in a predictive exponential family context.
result Provides a principled foundation for reliable uncertainty assessment in regression.
Paper introduces variance-based measures for second-order uncertainty quantification in classification problems.
problem Uncertainty in machine learning predictions and decision-making.
method Second-order uncertainty quantification using variance-based measures.
result Variance-based measures effectively quantify uncertainty on a class-based level and are competitive with entropy-based measures.