Measures policy-violating content prevalence with ML-assisted sampling and LLM labeling.
problem Accurate measurement of content violations that are often rare and costly to label.
method Design-based measurement system using ML-assisted probability sampling and LLM labeling.
result Produces unbiased prevalence estimates with confidence intervals and dashboard drilldowns.
The estimation of class prevalence, i.e., the fraction of a population that belongs to a certain class, is a very useful tool in data analytics and learning, and finds applications in many domains such as sentiment analysis, epidemiology, etc. For example, in sentiment analysis, the objective is often not to estimate w…
The Centers for Disease Control and Prevention (CDC) coordinates a labor-intensive process to measure the prevalence of autism spectrum disorder (ASD) among children in the United States. Random forests methods have shown promise in speeding up this process, but they lag behind human classification accuracy by about 5%…
Develops RES metrics for stable rare-event forecasting evaluation.
problem Challenges in evaluating forecasts of rare events.
method Rare-event-stable (RES) metrics designed to maintain stable thresholds under extreme rarity.
result RES metrics maintain stable thresholds, consistent model rankings, and near-complete prevalence invariance.
Proposes PQ, a more precise Bayesian quantifier for prevalence estimation.
problem Uncertainty quantification in prevalence estimation.
method Bayesian quantification methods, focusing on precision and coverage.
result PQ provides more precise and well-calibrated uncertainty quantification.
The paper discusses thresholds and bounds for accuracy in binary classification systems.
problem The accuracy of binary classification systems and its dependence on prevalence.
method Analyzing the precision-prevalence curve and negative predictive value-prevalence curve to find thresholds and bounds.
result Thresholds (φe and φn) bound various accuracy metrics (Fβ, F1, FM, MCC) and the ratio of maximum accuracy to prevalence. Paper explores how unsupervised learning can be understood through linear algebra concepts.
problem Understanding unsupervised learning through linear algebra concepts.
method Introducing the concept of linearly independent populations and using them to solve for prevalence values.
result Unsupervised learning can be realized as a generalization of supervised learning.
Point estimation of class prevalences in the presence of data set shift has been a popular research topic for more than two decades. Less attention has been paid to the construction of confidence and prediction intervals for estimates of class prevalences. One little considered question is whether or not it is necessar…
Bayesian method corrects bias in imbalanced datasets.
problem Prevalence bias in machine learning datasets.
method Bayesian risk minimization framework, bias-corrected loss function.
result Corrected loss function improves model performance.
This study connects prevalence and machine learning for diagnostic testing.
problem Uncertainty quantification in machine learning for diagnostic tests.
method Developed a numerical homotopy algorithm to estimate classification boundaries and quantify uncertainty.
result The proposed method stabilizes uncertainty quantification in machine learning for diagnostic tests.
Surveying risk measures for handling uncertainty in various fields.
problem Handling uncertainty in engineering and data-driven problems.
method Review of risk measures and their applications.
result Rapid development and widespread use of risk measures.
Over the last decade there has been increasing concern about the biases embodied in traditional evaluation methods for Natural Language Processing/Learning, particularly methods borrowed from Information Retrieval. Without knowledge of the Bias and Prevalence of the contingency being tested, or equivalently the expecta…
New measures capture tail dependence and non-exchangeability in financial data.
problem Underestimation of tail dependence and inability to capture non-exchangeable tail dependence.
method Tail copulas and novel tail dependence measures (MTCM, ATCM) are proposed.
result Captures non-exchangeable tail dependence and provides analytical forms for various copulas.
Study finds AUC is most consistent across different prevalence in binary classification.
problem Consistency of model evaluation metrics across varying prevalence in binary classification.
method Analysis of 156 data scenarios with 18 metrics, 5 models, and a random guess model.
result AUC has the smallest variance in evaluating individual models and ranking of models.
Bayesian methods improve group testing for identifying infected patients.
problem Identifying infected patients from group testing results with false positives.
method Bayesian inference and belief propagation algorithm, combined with expectation-maximization method.
result True-positive rate improved by considering credible intervals.
New measures quantify dependence between variables without distribution estimation.
problem Measuring dependence between variables in arbitrary dimensions.
method Proposed matrix-based normalized total correlation and dual total correlation measures.
result Measures are differentiable and statistically more powerful than existing methods.
Study proposes worst+gap measure for better DG evaluation.
problem Lack of comprehensive exploration of average measure in DG evaluation.
method Introduced worst+gap measure and compared it with average measure.
result Worst+gap measure provides a more accurate approximation of true DG performance.
Extends diffusion models to handle exponential family distributions for inverse problems.
problem Intractability of likelihood score for non-Gaussian observations.
method Evidence trick to approximate likelihood score for exponential family distributions.
result Effective Bayesian inference on complex Poisson processes and malaria prevalence prediction.
Optimizes classification algorithms with bounds on error rates.
problem Bounding uncertainties in classifier outputs for diagnostic testing.
method Set-theoretic and probabilistic arguments to derive uniform error bounds.
result Optimal partition minimizes the largest Gershgorin radius of the confusion matrix.
The paper proves prevalent existence and partially determines moduli space of area-minimizing surfaces with fractal singular sets.
problem Existence and moduli space of area-minimizing surfaces with fractal singular sets.
method Proof of prevalent existence, determination of moduli space, refinement of strata.
result Sharp results on moduli space and refinement of strata, showing fractal singularities do not completely dissolve under generic perturbations.
Proposes a new model to handle noisy data in scientific research.
problem Measurement error in noisy data settings.
method Measurement error BART (meBART) integrates measurement error in Bayesian additive regression trees.
result meBART provides more accurate parameter estimation, robust uncertainty quantification, and superior predictive performance.
Dynamic risk constraints help limit risky behavior in financial portfolios.
problem Static risk measures fail to control tail-risk-seeking traders.
method Introduces dynamic risk constraints applied throughout the trading horizon.
result Dynamic risk constraints can effectively limit risky behavior in portfolios.
This paper explores the possibility that asset prices, especially those traded in large volume on public exchanges, might comply with specific physical laws of motion and probability. The paper first examines the basic dynamics of asset price displacement and finds one can model this dynamic as a harmonic oscillator at…
We explore the loss landscape of fully-connected and convolutional neural networks using random, low-dimensional hyperplanes and hyperspheres. Evaluating the Hessian, H, of the loss function on these hypersurfaces, we observe 1) an unusual excess of the number of positive eigenvalues of H, and 2) a large value of $…
The study evaluates AI model performance measures for medical use.
problem Selecting appropriate performance measures for AI models in medical practice.
method Assessed 32 performance measures across five domains for binary outcomes.
result 17 measures are both proper and reflect decision-analytic performance.
Temporal data are increasingly prevalent in modern data science. A fundamental question is whether two time series are related or not. Existing approaches often have limitations, such as relying on parametric assumptions, detecting only linear associations, and requiring multiple tests and corrections. While many non-p…
Study uses trajectory embedding to measure place function similarity at fine spatial granularity.
problem Measuring place function similarity at fine spatial granularity.
method Trajectory embedding to reduce dimensions and measure similarity of place functions.
result Embedding similarity can be a metric proxy for place functions at fine spatial granularity.
Estimating the dependences between random variables, and ranking them accordingly, is a prevalent problem in machine learning. Pursuing frequentist and information-theoretic approaches, we first show that the p-value and the mutual information can fail even in simplistic situations. We then propose two conditions for r…
In statistical learning, high covariate dimensionality poses challenges for robust prediction and inference. To address this challenge, supervised dimension reduction is often performed, where dependence on the outcome is maximized for a selected covariate subspace with smaller dimensionality. Prevalent dimension reduc…
Various measures can be used to estimate bias or unfairness in a predictor. Previous work has already established that some of these measures are incompatible with each other. Here we show that, when groups differ in prevalence of the predicted event, several intuitive, reasonable measures of fairness (probability of p…
Simplifies VAE for anomaly detection using rate-distortion theory.
problem Anomaly detection in unsupervised learning systems.
method Revisit VAE from information theory, incorporate model uncertainty.
result Competitive performance on benchmark datasets.
AdapDISCOM tackles high-dimensional multimodal data with missingness and errors, improving prediction and biomarker selection.
problem High-dimensional multimodal data with block-wise missingness and measurement errors.
method AdapDISCOM introduces modality-specific weighting schemes to address heterogeneity and error magnitudes.
result AdapDISCOM consistently outperforms existing methods under heterogeneous contamination and heavy-tailed distributions.
In this paper, we provide non-parametric statistical tools to test stationarity of microstructure noise in general hidden Ito semimartingales, and discuss how to measure liquidity risk using high frequency financial data. In particular, we investigate the impact of non-stationary microstructure noise on some volatility…
New method adapts to structural shifts in graph data for better label prevalence estimation.
problem Structural shifts in graph data affect label prevalence estimation.
method Importance sampling variant of KDEy quantification approach.
result Adapts to structural shifts and outperforms standard approaches.
BDC uses Distance Correlation for efficient Bayesian optimization of expensive functions.
problem Efficiently optimizing expensive black-box functions with Bayesian methods.
method Integrates Bayesian optimization with Distance Correlation for automatic exploration and exploitation.
result BDC performs similarly to popular BO methods on benchmark tests and real terrain optimization.
Estimates disease prevalence using non-ignorable missing data in health surveys.
problem Estimating disease prevalence in non-representative samples with non-ignorable missing data.
method Connects auxiliary proxy variable framework to label shift setting, uses high-dimensional covariates without generative models.
result Fails to account for non-ignorable missingness can lead to significant misestimations.
Intensive care clinicians are presented with large quantities of patient information and measurements from a multitude of monitoring systems. The limited ability of humans to process such complex information hinders physicians to readily recognize and act on early signs of patient deterioration. We used machine learnin…
Reduces quantifier variance with accuracy optimization of base classifier.
problem Minimizing quantifier variance under prior probability shift.
method Optimizes the Brier score of a base classifier for training data.
result Optimizing Brier score on training data reduces quantifier variance on test data.
Paper uses inverse optimization to measure risk preference from investment portfolios.
problem Measuring subjective risk preference in investment portfolios.
method Inverse optimization on mean-variance framework.
result Quantified risk preference parameters validated with existing measures.
We consider the problem of demixing a sequence of source signals from the sum of noisy bilinear measurements. It is a generalized mathematical model for blind demixing with blind deconvolution, which is prevalent across the areas of dictionary learning, image processing, and communications. However, state-of- the-art c…
A new method uses deep Gaussian processes to handle missing values in irregularly sampled healthcare data.
problem Missing values and irregular sampling in healthcare data.
method Deep Gaussian process emulation with stochastic imputation.
result The method outperforms conventional imputation methods in clinical datasets.
Paper proposes method for optimal control of unknown systems with latent states.
problem Jointly estimating dynamics and latent states in systems with unmeasurable states.
method Combination of particle Markov chain Monte Carlo methods and scenario theory.
result Probabilistic performance guarantees for optimal input trajectories.
The paper examines the consistency of item embeddings in recommendation systems.
problem The relevance of averaging item embeddings for user or concept representation.
method Proposes an expected precision score to measure consistency and analyzes it theoretically and empirically.
result Real-world averages are less consistent for recommendation compared to theoretical assumptions.
Credit risk prediction is an effective way of evaluating whether a potential borrower will repay a loan, particularly in peer-to-peer lending where class imbalance problems are prevalent. However, few credit risk prediction models for social lending consider imbalanced data and, further, the best resampling technique t…
Machine learning models for medical image analysis often suffer from poor performance on important subsets of a population that are not identified during training or testing. For example, overall performance of a cancer detection model may be high, but the model still consistently misses a rare but aggressive cancer su…
Reconstruction error is a prevalent score used to identify anomalous samples when data are modeled by generative models, such as (variational) auto-encoders or generative adversarial networks. This score relies on the assumption that normal samples are located on a manifold and all anomalous samples are located outside…
HistNetQ improves quantification tasks by optimizing loss functions and eliminating label requirements.
problem Quantification of class prevalence in bags of examples.
method Permutation-invariant Histograms and deep neural networks.
result HistNetQ outperforms other quantification methods and optimizes custom loss functions.
Paper analyzes impact of PRM on binary random variables and distribution shifts.
problem Impact of performative risk minimization on binary random variables and distribution shifts.
method Formulated two measures of impact, derived explicit formulas for full information, and provided estimators for partial information.
result PRM can have amplified side effects compared to methods that do not model data shift.