k-Rater reliability corrects under-reporting of aggregated data reliability.
problem Under-reporting of data reliability in aggregated ratings.
method k-Rater reliability (kRR) as a multi-rater generalization of IRR.
result kRR provides a more accurate measure of reliability for aggregated datasets.
Improves reliability of medical diagnosis uncertainty estimates.
problem Label uncertainty in medical diagnosis.
method Post-hoc alpha-calibration method for neural network classifiers. result Significantly enhances reliability of uncertainty estimates.
Framework assesses autograders' reliability and biases.
problem Mixed reliability and biases in autograders for LLM evaluation.
method Bayesian GLMs to model evaluation outcomes.
result Explicit quantification of scoring differences and biases.
ALICE combines feature selection and inter-rater agreeability for ML model insights.
problem Improving interpretability of black box machine learning models.
method Integrates feature selection and inter-rater agreeability into a user-friendly Python library.
result Initial experiments on customer churn modeling show promising insights.
SMART is an open source web application designed to help data scientists and research teams efficiently build labeled training data sets for supervised machine learning tasks. SMART provides users with an intuitive interface for creating labeled data sets, supports active learning to help reduce the required amount of …
Study evaluates consistency of LLMs in binary text classification, providing systematic guidance.
problem Lack of reliable methods for evaluating large language model (LLM) binary text classification.
method Adapting psychometric principles, the study determines sample size requirements, develops metrics for invalid responses, and evaluates intra- and inter-rater reliability.
result LLMs demonstrated high intra-rater consistency, achieving perfect agreement on 90-98% of examples, with smaller models outperforming larger counterparts.
The retinal vascular condition is a reliable biomarker of several ophthalmologic and cardiovascular diseases, so automatic vessel segmentation may be crucial to diagnose and monitor them. In this paper, we propose a novel method that combines the multiscale analysis provided by the Stationary Wavelet Transform with a m…
Study evaluates five LLMs for financial report analysis, revealing performance differences and variability.
problem Lack of understanding in reliability, consistency, and transparency of LLMs in financial analysis.
method Human evaluation, automated similarity metrics, and behavioral diagnostics applied to five transformer-based LLMs over U.S. 10-K filings.
result No single LLM consistently dominates across all evaluation perspectives, highlighting variability and need for interpretability.
Quantifying the degree of atrophy is done clinically by neuroradiologists following established visual rating scales. For these assessments to be reliable the rater requires substantial training and experience, and even then the rating agreement between two radiologists is not perfect. We have developed a model we call…
An automated metric to evaluate dialogue quality is vital for optimizing data driven dialogue management. The common approach of relying on explicit user feedback during a conversation is intrusive and sparse. Current models to estimate user satisfaction use limited feature sets and rely on annotation schemes with low …
Proposes a strategy to train models with minimal labeled data.
problem Scarce and expensive labeled data for medical tasks.
method Recursive training strategy to use image-level annotations for pixel-level segmentation.
result Improved segmentation of intracranial hemorrhage in CT scans.
Study optimizes deep learning models for sleep stage classification.
problem Time-consuming and inconsistent manual sleep stage scoring.
method Investigated architectural choices in encoder-predictor architectures for polysomnography recordings.
result Robust architectures improve sleep stage classification performance.
Generative models have made immense progress in recent years, particularly in their ability to generate high quality images. However, that quality has been difficult to evaluate rigorously, with evaluation dominated by heuristic approaches that do not correlate well with human judgment, such as the Inception Score and …
A new model improves medical image segmentation uncertainty.
problem Uncertainty in medical image segmentation.
method Conditional Normalizing Flow (cFlow) for improved segmentation uncertainty.
result Improved quality and diversity of segmentation samples.
Sleep stage classification constitutes an important element of sleep disorder diagnosis. It relies on the visual inspection of polysomnography records by trained sleep technologists. Automated approaches have been designed to alleviate this resource-intensive task. However, such approaches are usually compared to a sin…
Study evaluates machine learning methods for large-scale network reliability, revealing ANN's and PR's performance.
problem Tackles the NP-hard problem of approximating binary-state network reliability for large-scale systems.
method Compares 20 machine learning methods across three reliability regimes and evaluates their performance on large-scale networks.
result Large-scale networks with arc reliability ≥ 0.9 exhibit near-unity system reliability, enabling computational simplifications.
In many instances, information on engineering systems can be obtained through measurements, monitoring or direct observations of system performances and can be used to update the system reliability estimate. In structural reliability analysis, such information is expressed either by inequalities (e.g. for the observati…
Study integrates reliability constraints into generation planning models.
problem Challenges in integrating reliability constraints with generation planning models.
method Leverages a weighted oblique decision tree (WODT) technique to embed reliability verification constraints.
result Demonstrates effectiveness in achieving reliable and optimal planning solutions.
Paper explores physics-informed deep learning for system reliability assessment.
problem Limited study on deep learning for system reliability assessment.
method Physics-informed deep learning approach for system reliability assessment.
result Physics-informed deep learning can alleviate computational challenges and combine measurement data and mathematical models.
New metric assesses reliability of AI explanations.
problem Unreliable AI explanations under realistic conditions.
method Explanation Reliability Index (ERI) metrics quantifying stability under four axioms.
result Widespread reliability failures in popular explanation methods.
A new score measures data reliability without ground truth.
problem Assessing reliability of datasets without access to ground truth.
method Define ground-truth-based orderings and propose Gram determinant score.
result Gram determinant score effectively captures data quality across diverse observation processes.
We derive the mapping between two of the most pervasive utility functions, the mean square error (MSE) and the concordance correlation coefficient (CCC, ρc). Despite its drawbacks, MSE is one of the most popular performance metrics (and a loss function); along with lately ρc in many of the sequence prediction…
BeMF improves recommendation reliability in recommender systems.
problem Improving reliability in recommender systems beyond accuracy.
method Bernoulli Matrix Factorization (BeMF) for model-based collaborative filtering.
result BeMF selects more reliable predictions, improving recommendation quality.
Examining ESG scoring method for reliability.
problem Reliability of ESG scoring methodology.
method Analyzing Refinitiv's ESG scoring process.
result Methodology needs improvement for trustworthiness.
Improves reliability diagrams for probabilistic forecasts.
problem Lack of stability in reliability diagrams hampered their use.
method CORP approach using non-parametric isotonic regression and PAV algorithm.
result Improved reliability diagrams with statistical consistency and reproducibility.
Improves neural network performance by dynamically adjusting model weights based on source reliability.
problem Training neural networks on data from unreliable sources leads to poor performance.
method Dynamic re-weighting strategy using likelihood tempering to adjust model weights based on estimated source reliability.
result Significant improvement in model performance when trained on mixtures of reliable and unreliable data sources.
New algorithm for reliable learning of Gaussian halfspaces with improved sample and computational complexity.
problem Learning halfspaces under Gaussian marginals with reliable agnostic model.
method Developed a new algorithm for reliable learning of Gaussian halfspaces with specific sample and computational complexity.
result Achieved a new algorithm with improved sample and computational complexity for reliable learning of Gaussian halfspaces.
New method improves reliability of selecting individuals based on predicted treatment effects.
problem Reliability of selecting individuals based on predicted conditional average treatment effects (CATE) is unreliable.
method Denoised Conformal Alignment, combining proxy errors, variance estimation, and Benjamini-Hochberg selection.
result Significantly improved power in selecting individuals while maintaining false discovery rate control.
Superpixel-mix enhances reliability in semantic segmentation.
problem Improving reliability in real-world semantic segmentation.
method Superpixel-mix, a new data augmentation method with teacher-student consistency training.
result Superpixel-mix achieves state-of-the-art results in semi-supervised semantic segmentation.
Survey and framework for efficient active learning in structural reliability.
problem Efficiently solving complex structural reliability problems.
method Generalized modular framework combining surrogate model, reliability estimation algorithm, learning function, and stopping criterion.
result 39 strategies for solving 20 reliability benchmark problems, highlighting the importance of surrogates and algorithms.
Study defines and optimizes bank reliability using LR and PSO.
problem Lack of reliability concept in financial services.
method Logistic Regression (LR) for initial estimation, Particle Swarm Optimization (PSO) for optimization.
result Optimal financial ratios maximize bank reliability.
Paper improves ETF tail-risk monitoring reliability.
problem Unreliable ETF risk monitoring under degraded data.
method Combines quality checks, prediction, scoring, and adjustment.
result Improves tail-risk monitoring, especially during stressed periods.
AL-SPCE improves reliability analysis for complex systems with active learning and SPCE.
problem Efficiently analyzing reliability of complex, computationally expensive models with intrinsic randomness.
method Active learning framework using stochastic polynomial chaos expansions (SPCE) to reduce computational burden.
result AL-SPCE maintains high accuracy in reliability estimates while significantly improving efficiency.
This dissertation tackles challenges in reliable machine learning measurement.
problem Challenges in reproducibility, scalability, and uncertainty quantification in machine learning.
method Develops criteria for meaningful metrics and methodologies for scalable, reliable measurement.
result Provides methods for evaluating generative-AI systems and quantifying memorization.
ReaPER improves learning efficiency by prioritizing reliable experiences.
problem Inefficient sampling of past experiences in reinforcement learning.
method Introducing a novel measure of reliability to prioritize experiences in PER.
result ReaPER outperforms PER in various environments, including Atari-10.
A new method uses physics-informed neural networks to solve reliability analysis problems without simulations.
problem Solving reliability analysis problems without the need for expensive simulations.
method Physics-informed neural networks to learn directly from problem physics.
result Eliminates the need for expensive simulations and achieves highly accurate results.
Proposes a Koopman operator method for time-dependent reliability analysis of nonlinear systems.
problem Challenges in time-dependent reliability analysis of nonlinear dynamical systems.
method Koopman operator approach for transforming nonlinear systems into linear ones, combined with deep learning for intrinsic coordinates.
result Robust and generalizable approach for time-dependent reliability analysis, superior to purely data-driven methods.
The paper proposes a method to calibrate healthcare AI models for reliability and interpretability.
problem Characterizing model reliability and enabling introspection of model behavior in clinical decision making.
method A calibration-driven learning method combined with interpretability techniques based on counterfactual reasoning.
result Demonstrates the effectiveness of the proposed approach using a lesion classification problem with dermoscopy images.
MAntRA combines machine learning and Bayesian methods for time-dependent reliability analysis of unknown systems.
problem Time-dependent reliability analysis of systems with unknown governing physics.
method Combines machine learning, Bayesian statistics, and stochastic integration to discover and analyze SDEs from data.
result Demonstrates the effectiveness of MAntRA on three numerical examples, indicating its potential for in-situ and heritage structure analysis.
Reliability Options are capacity remuneration mechanisms aimed at enhancing security of supply in electricity systems. They can be framed as call options on electricity sold by power producers to System Operators. This paper provides a comprehensive mathematical treatment of Reliability Options. Their value is first de…
Online knowledge repositories typically rely on their users or dedicated editors to evaluate the reliability of their content. These evaluations can be viewed as noisy measurements of both information reliability and information source trustworthiness. Can we leverage these noisy evaluations, often biased, to distill a…
PS-DME evaluates model performance and reliability after data-dependent selection.
problem Evaluating model performance and reliability when data is used for selection and evaluation.
method Post-selection distributional model evaluation (PS-DME) using e-values to control false coverage rate.
result PS-DME provides reliable comparison of model configurations across different reliability levels.
Bayesian framework improves deep classifier reliability.
problem Overconfident models under dataset shift.
method Bayesian inference with out-of-distribution data augmentation.
result Reliable uncertainty estimates for deep classifiers.
CoNBONet improves reliability analysis of complex systems with fast, energy-efficient predictions.
problem Time-dependent reliability analysis of nonlinear systems under stochastic excitations is computationally demanding.
method CoNBONet combines deep operator networks with neuroscience-inspired neuron models for fast, energy-efficient inference.
result CoNBONet provides reliable coverage of failure probabilities with theoretical guarantees.
Bayesian learning improves reliability of molecular predictions for hit compound discovery.
problem Improving reliability of machine learning predictions for virtual screening.
method Bayesian learning algorithms applied to graph neural networks.
result Bayesian learning leads to well-calibrated predictions and higher hit compound success.
Machine learning (ML) methods have the potential to automate clinical EEG analysis. They can be categorized into feature-based (with handcrafted features), and end-to-end approaches (with learned features). Previous studies on EEG pathology decoding have typically analyzed a limited number of features, decoders, or bot…
Plex improves model reliability across vision and language tasks.
problem Improving model reliability in diverse decision-making tasks involving uncertainty and adaptation.
method Developed ViT-Plex and T5-Plex pretrained model extensions to evaluate and improve reliability across 40 datasets.
result Plex greatly improves state-of-the-art across reliability tasks, simplifying evaluation and performance.
Proposes a new criterion for reliable uncertainty estimation in deep neural networks.
problem Inability of existing approaches to provide reliable uncertainty estimates for deep neural networks.
method Develops a density uncertainty layer architecture that satisfies the proposed criterion.
result Density uncertainty layers provide more reliable uncertainty estimates and robust out-of-distribution detection.