k-Rater reliability corrects under-reporting of aggregated data reliability.
problem Under-reporting of data reliability in aggregated ratings.
method k-Rater reliability (kRR) as a multi-rater generalization of IRR.
result kRR provides a more accurate measure of reliability for aggregated datasets.
A new score measures data reliability without ground truth.
problem Assessing reliability of datasets without access to ground truth.
method Define ground-truth-based orderings and propose Gram determinant score.
result Gram determinant score effectively captures data quality across diverse observation processes.
Study evaluates machine learning methods for large-scale network reliability, revealing ANN's and PR's performance.
problem Tackles the NP-hard problem of approximating binary-state network reliability for large-scale systems.
method Compares 20 machine learning methods across three reliability regimes and evaluates their performance on large-scale networks.
result Large-scale networks with arc reliability ≥ 0.9 exhibit near-unity system reliability, enabling computational simplifications.
Improves neural network performance by dynamically adjusting model weights based on source reliability.
problem Training neural networks on data from unreliable sources leads to poor performance.
method Dynamic re-weighting strategy using likelihood tempering to adjust model weights based on estimated source reliability.
result Significant improvement in model performance when trained on mixtures of reliable and unreliable data sources.
Paper explores physics-informed deep learning for system reliability assessment.
problem Limited study on deep learning for system reliability assessment.
method Physics-informed deep learning approach for system reliability assessment.
result Physics-informed deep learning can alleviate computational challenges and combine measurement data and mathematical models.
Superpixel-mix enhances reliability in semantic segmentation.
problem Improving reliability in real-world semantic segmentation.
method Superpixel-mix, a new data augmentation method with teacher-student consistency training.
result Superpixel-mix achieves state-of-the-art results in semi-supervised semantic segmentation.
Paper improves ETF tail-risk monitoring reliability.
problem Unreliable ETF risk monitoring under degraded data.
method Combines quality checks, prediction, scoring, and adjustment.
result Improves tail-risk monitoring, especially during stressed periods.
PS-DME evaluates model performance and reliability after data-dependent selection.
problem Evaluating model performance and reliability when data is used for selection and evaluation.
method Post-selection distributional model evaluation (PS-DME) using e-values to control false coverage rate.
result PS-DME provides reliable comparison of model configurations across different reliability levels.
Bayesian framework improves deep classifier reliability.
problem Overconfident models under dataset shift.
method Bayesian inference with out-of-distribution data augmentation.
result Reliable uncertainty estimates for deep classifiers.
MAntRA combines machine learning and Bayesian methods for time-dependent reliability analysis of unknown systems.
problem Time-dependent reliability analysis of systems with unknown governing physics.
method Combines machine learning, Bayesian statistics, and stochastic integration to discover and analyze SDEs from data.
result Demonstrates the effectiveness of MAntRA on three numerical examples, indicating its potential for in-situ and heritage structure analysis.
A new method uses physics-informed neural networks to solve reliability analysis problems without simulations.
problem Solving reliability analysis problems without the need for expensive simulations.
method Physics-informed neural networks to learn directly from problem physics.
result Eliminates the need for expensive simulations and achieves highly accurate results.
Unstructured data refers to information that does not have a predefined data model or is not organized in a pre-defined manner. Loosely speaking, unstructured data refers to text data that is generated by humans. In after-sales service businesses, there are two main sources of unstructured data: customer complaints, wh…
Foundation models improve time series prediction reliability, especially with limited data.
problem Improving time series prediction reliability with limited data.
method Comparison of Time Series Foundation Models (TSFMs) with traditional methods in conformal prediction.
result TSFMs provide more reliable conformalized prediction intervals and more stable calibration with limited data.
New approach to robustly reliable learners against instance-targeted attacks.
problem Addressing data poisoning attacks on robustly reliable learners.
method Defining regularized robustly reliable learners and efficient algorithms.
result Efficient algorithms for robustly reliable learners with sublinear runtime.
Proposes a Koopman operator method for time-dependent reliability analysis of nonlinear systems.
problem Challenges in time-dependent reliability analysis of nonlinear dynamical systems.
method Koopman operator approach for transforming nonlinear systems into linear ones, combined with deep learning for intrinsic coordinates.
result Robust and generalizable approach for time-dependent reliability analysis, superior to purely data-driven methods.
ReaPER improves learning efficiency by prioritizing reliable experiences.
problem Inefficient sampling of past experiences in reinforcement learning.
method Introducing a novel measure of reliability to prioritize experiences in PER.
result ReaPER outperforms PER in various environments, including Atari-10.
The paper improves A/B testing for non-Gaussian data, ensuring reliable results with large sample sizes.
problem Inaccurate A/B testing results due to non-normal data and unequal sample sizes.
method Derives explicit formulas for minimum sample size and introduces an Edgeworth-based correction.
result Corrected method improves reliability of A/B testing in real-world conditions.
Study improves reliability of neural models for virtual screening.
problem Reliability issues in neural models for molecular property prediction.
method Investigated model architectures, regularization, and loss functions.
result Correct choice of regularization and inference methods improves reliability.
Reliability Options are capacity remuneration mechanisms aimed at enhancing security of supply in electricity systems. They can be framed as call options on electricity sold by power producers to System Operators. This paper provides a comprehensive mathematical treatment of Reliability Options. Their value is first de…
Study shows more data improves model explanations, aiding reliable knowledge extraction.
problem Challenges in deriving reliable knowledge from machine learning models due to the Rashōmon effect.
method Examined the influence of sample size on explanations from models in a Rashōmon set using SHAP.
result Explanations from <128 samples are highly variable, but agreement improves with more data.
Intelligent transportation systems (ITSs) will be a major component of tomorrow's smart cities. However, realizing the true potential of ITSs requires ultra-low latency and reliable data analytics solutions that can combine, in real-time, a heterogeneous mix of data stemming from the ITS network and its environment. Su…
R-AutoEval+ improves model evaluation efficiency and reliability using adaptive synthetic data.
problem Accurate model selection from AI candidates using real-world data is costly and impractical at scale.
method R-AutoEval+ uses adaptive prediction-powered inference to correct bias in autoevaluators while maintaining or improving sample efficiency.
result R-AutoEval+ provides finite-sample reliability guarantees and enhanced sample efficiency compared to conventional methods.
In many scientific tasks we are interested in discovering whether there exist any correlations in our data. This raises many questions, such as how to reliably and interpretably measure correlation between a multivariate set of attributes, how to do so without having to make assumptions on distribution of the data or t…
CEDA analyzes large categorical datasets using tree geometry and binary codes.
problem Analyzing large categorical datasets with extreme-K samples. method CEDA uses tree geometry and binary codes to analyze categorical data.
result CEDA discovers patterns and evaluates their reliability in large categorical datasets.
Online knowledge repositories typically rely on their users or dedicated editors to evaluate the reliability of their content. These evaluations can be viewed as noisy measurements of both information reliability and information source trustworthiness. Can we leverage these noisy evaluations, often biased, to distill a…
Research tackles ML failure in non-similar data, introducing reliable algorithms.
problem Machine learning models fail in new data distributions.
method Formal definition, assumptions, and simple algorithms for reliable generalization.
result Introduction of algorithms providing more reliable generalization.
How does one find dimensions in multivariate data that are reliably expressed across repetitions? For example, in a brain imaging study one may want to identify combinations of neural signals that are reliably expressed across multiple trials or subjects. For a behavioral assessment with multiple ratings, one may want …
DeepTrust uses NLP to quickly identify and verify financial anomalies on Twitter.
problem Unreliable information in financial markets leading to unexpected price changes.
method Machine learning for anomaly detection, NLP for information retrieval and reliability assessment.
result DeepTrust outperforms baseline classifiers in identifying financial anomalies.
Proposes a new criterion for reliable uncertainty estimation in deep neural networks.
problem Inability of existing approaches to provide reliable uncertainty estimates for deep neural networks.
method Develops a density uncertainty layer architecture that satisfies the proposed criterion.
result Density uncertainty layers provide more reliable uncertainty estimates and robust out-of-distribution detection.
F-PACOH improves meta-learners' reliability in uncertain regions.
problem Overconfident uncertainty estimates in meta-learning.
method Meta-learning priors as stochastic processes in function space, directly steering predictions towards high epistemic uncertainty.
result Significantly outperforms other meta-learners in Bayesian Optimization.
This dissertation tackles challenges in reliable machine learning measurement.
problem Challenges in reproducibility, scalability, and uncertainty quantification in machine learning.
method Develops criteria for meaningful metrics and methodologies for scalable, reliable measurement.
result Provides methods for evaluating generative-AI systems and quantifying memorization.
Study finds machine learning interpretations are often unstable and unreliable.
problem Reliability of machine learning interpretations in high-stakes domains.
method Stability study on global interpretations using tabular data.
result Popular interpretation methods are frequently unstable, less stable than predictions, and not associated with prediction accuracy.
In this paper, a novel experienced deep reinforcement learning (deep-RL) framework is proposed to provide model-free resource allocation for ultra reliable low latency communication (URLLC). The proposed, experienced deep-RL framework can guarantee high end-to-end reliability and low end-to-end latency, under explicit …
The paper shows how to efficiently generate large Gaussian process samples with reliability guarantees.
problem Generating large-scale Gaussian process samples efficiently and with reliability.
method Demonstrates scaling data generation to large \(n\) while providing high probability guarantees.
result Efficiently generates large Gaussian process samples with reliability guarantees.
The paper examines the reliability of limit order book representations in the face of data perturbation.
problem The reliability of limit order book representations under data perturbation.
method Experimental analysis of existing representations and guidelines for future research.
result Existing representations of limit order book data are vulnerable to data perturbation.
Method cleans noisy training labels for biomedical data.
problem Accurately labeling biomedical data is challenging.
method Reliability-based training data cleaning with inductive conformal prediction.
result Significant enhancements in classification performance across multiple tasks.
The paper certifies AI reliability via sampling and calibration, providing exact guarantees.
problem Ensuring trust in black-box AI systems' outputs.
method Self-consistency sampling and conformal calibration.
result Reliability levels derived from these methods offer finite-sample guarantees.
The paper proposes an AI and IIoT framework for improved maintenance.
problem Current maintenance practices need improvement with AI and IIoT.
method Review of reliability modeling, introduction of Intelligent Maintenance framework, and novel probabilistic deep learning approach.
result Demonstrated novel probabilistic deep learning reliability modelling in Turbofan Engine Degradation Dataset.
COPP provides reliable intervals for outcomes under a new policy in contextual bandits.
problem Lack of reliable predictive intervals for outcomes under a new policy in contextual bandits.
method Conformal prediction applied to contextual bandits.
result COPP provides finite-sample guarantees without additional assumptions.
DW-KNN improves KNN by integrating distance and neighbor reliability for better prediction accuracy.
problem Standard KNN assumes all neighbors are equally reliable, leading to unreliable predictions in heterogeneous feature spaces.
method DW-KNN integrates exponential distance with neighbor validity, providing instance-level interpretability and reducing hyperparameter sensitivity.
result DW-KNN achieves 0.8988 average accuracy, ranks 2nd among six methods, and has the lowest cross-validation variance.
CAT framework improves AI medical screening fairness and reliability.
problem Imbalanced data, varying performance across cohorts, and patient-level inconsistencies in traditional metrics.
method CAT framework introduces patient-level assessment, entropy-based distribution weighting, and cohort-weighted sensitivity and specificity.
result Enhanced predictive reliability, fairness, and interpretability of AI-driven medical screening models.
Paper tackles uncertainty in GNNs for graph data.
problem Uncertainty in GNNs' predictions for graph data.
method CF-T2NN, tensor decomposition, topological learning.
result CF-T2NN improves reliability and interpretability of GNN outcomes.
m-arcsinh improves SVM and MLP reliability and speed in scikit-learn.
problem Improving SVM and MLP reliability and speed in scikit-learn.
method Modified arcsinh function for kernel and activation in SVM and MLP.
result Competitive classification performance and reliability of SVM and MLP with m-arcsinh.
We proposed a probabilistic approach to joint modeling of participants' reliability and humans' regularity in crowdsourced affective studies. Reliability measures how likely a subject will respond to a question seriously; and regularity measures how often a human will agree with other seriously-entered responses coming…
Adaptive BO improves solder joint reliability by 3% with half the computational cost.
problem Improving solder joint reliability under thermomechanical loading.
method Adaptive Bayesian optimization with Gaussian process regression.
result Adaptive BO outperforms regular BO by 3% on average at any given computational budget.
To use machine learning in high stakes applications (e.g. medicine), we need tools for building confidence in the system and evaluating whether it is reliable. Methods to improve model reliability often require new learning algorithms (e.g. using Bayesian inference to obtain uncertainty estimates). An alternative is to…
Paper proposes MUCS for more reliable TDA in diffusion models.
problem Current TDA approaches lack reliability and robustness.
method Mirrored unlearning and noise-consistent skew (MUCS).
result MUCS outperforms existing methods on three datasets.
New test assesses reliability of auto-generated features.
problem Lack of reliable assessment for auto-generated features.
method Selective inference framework for statistical testing.
result Proposes a statistical test for auto-generated features in linear models.