Study controls error rates of binary classifiers using hypothesis testing.
problem Traditional binary classifiers have uncontrolled error rates.
method Combines binary classification with statistical hypothesis testing.
result Trained classifiers can be made to meet target error rate thresholds.
This research examines how the error rate of nearest neighbor classifiers varies with dataset size.
problem The scaling of classification error rates with dataset size is not uniform.
method Theoretical analysis of nearest neighbor classifiers, focusing on early and late phases of dataset size.
result The error rate of nearest neighbor classifiers can have fine-grained rates depending on the dataset size and data distribution.
We apply multiple testing procedures to the validation of estimated default probabilities in credit rating systems. The goal is to identify rating classes for which the probability of default is estimated inaccurately, while still maintaining a predefined level of committing type I errors as measured by the familywise …
We find a deterministic equivalent for random feature regression's test error, independent of feature map dimension.
problem Understanding the generalization performance of random feature ridge regression.
method We derive a deterministic equivalent for the test error of RFRR under a concentration property, showing it can be approximated by a closed-form expression dependent on feature map eigenvalues.
result Our approximation guarantee is non-asymptotic, multiplicative, and independent of the feature map dimension, providing a tight result for the smallest number of features achieving optimal minimax error rate.
Efficient tests achieve best error rates in high-dimensional hypothesis testing.
problem Achieving optimal error rates in computationally efficient hypothesis testing.
method Linear spectral statistics and low-degree likelihood ratio analysis.
result An efficient test achieves the best possible error rates among all computationally efficient tests.
The repeated community-wide reuse of test sets in popular benchmark problems raises doubts about the credibility of reported test-error rates. Verifying whether a learned model is overfitted to a test set is challenging as independent test sets drawn from the same data distribution are usually unavailable, while other …
Calibration without labels in multiple testing
problem Interpretable error probabilities in large-scale hypothesis testing
method Constructing pseudo-labels from spacings of ordered p-values result Finding that q-value can be severely miscalibrated MOB-dS uses permutation to correct for dependency in discrete survival data.
problem Identifying subgroups in discrete event time data with potential spurious results.
method Model-based recursive partitioning (MOB) with modified data matrix and permutation test.
result MOB-dS controls type I error rate better than standard MOB for discrete survival data.
Proposes selective inference for testing differences in means between clusters.
problem Inflated type I error rate when testing differences in means between clusters.
method Selective inference approach to control selective type I error rate.
result Controls selective type I error rate by accounting for data-driven cluster definition.
We characterize the asymptotic performance of nonparametric one- and two-sample testing. The exponential decay rate or error exponent of the type-II error probability is used as the asymptotic performance metric, and an optimal test achieves the maximum rate subject to a constant level constraint on the type-I error pr…
We address the problem of non-parametric multiple model comparison: given l candidate models, decide whether each candidate is as good as the best one(s) or worse than it. We propose two statistical tests, each controlling a different notion of decision errors. The first test, building on the post selection inference…
We consider binary classification problems with positive definite kernels and square loss, and study the convergence rates of stochastic gradient methods. We show that while the excess testing loss (squared loss) converges slowly to zero as the number of observations (and thus iterations) goes to infinity, the testing …
New method controls false discoveries in real-time data streams.
problem Online testing of hypotheses with strict error constraints and no future data.
method Structure-adaptive sequential testing (SAST) with alpha-investment algorithm.
result Substantial power gain over existing online testing rules.
Max-rank improves multiple testing in conformal prediction.
problem Simultaneous testing of multiple hypotheses in scientific inquiries.
method Introduces max-rank, a novel correction for positive dependencies in simultaneous testing.
result Max-rank efficiently controls family-wise error rate and improves predictive uncertainty estimates.
We characterize the asymptotic performance of nonparametric goodness of fit testing. The exponential decay rate of the type-II error probability is used as the asymptotic performance metric, and a test is optimal if it achieves the maximum rate subject to a constant level constraint on the type-I error probability. We …
A method for safe online classification reduces test costs while maintaining low error rates.
problem Sequential testing for binary disease outcomes with unknown logistic model parameters.
method Joint estimation of logistic parameter and feature distribution with a conservative threshold.
result Achieves target error with high probability and requires minimal excess tests.
This paper tackles worst-class error rate in classification tasks.
problem Minimizing worst-class error rate in classification tasks, especially in medical image classification.
method Designing a boosting approach to bound the worst-class error rate using Deep Neural Networks (DNNs).
result The proposed boosting approach lowers worst-class test error rates while avoiding overfitting.
We propose a two-sample testing procedure based on learned deep neural network representations. To this end, we define two test statistics that perform an asymptotic location test on data samples mapped onto a hidden layer. The tests are consistent and asymptotically control the type-1 error rate. Their test statistics…
We formulate statistical watermarking as hypothesis testing and establish near-optimal bounds.
problem Statistical watermarking in the context of hypothesis testing.
method Formulated as a hypothesis testing problem, using coupling of output tokens and rejection regions.
result Established nearly matching upper and lower bounds on the number of i.i.d. tokens required for small Type I and Type II errors.
Algorithm learns from both labeled and arbitrary test examples, giving guarantees for bounded VC dimension classes.
problem Learning from arbitrary test examples, not just perturbations.
method Selective transductive learning algorithm that outputs abstaining predictions.
result Nontrivial guarantees for bounded VC dimension classes with arbitrary train and test distributions.
Selective inference controls Type I error in k-means clustering tests.
problem Inflated Type I error in classical hypothesis tests for k-means clusters.
method Selective inference approach to control Type I error.
result Proposes a computable finite-sample p-value for selective inference.
Study on error rates for approximating rough volatility models.
problem Simulation of rough volatility models with fractional Brownian motion.
method Analysis of weak error rates for numerical schemes, focusing on fBm and cubic test functions.
result Convergence rates for approximations are (3H+21)∧1 for exact left-point discretization and H+21 for hybrid schemes. SGD-trained models' disagreement predicts test error.
problem Estimating test error of deep networks.
method Empirical testing and theoretical analysis of SGD ensembles.
result SGD ensembles' disagreement correlates with test error.
Successful deployment of machine learning algorithms in healthcare requires careful assessments of their performance and safety. To date, the FDA approves locked algorithms prior to marketing and requires future updates to undergo separate premarket reviews. However, this negates a key feature of machine learning--the …
We revisit resampling procedures for error estimation in binary classification in terms of U-statistics. In particular, we exploit the fact that the error rate estimator involving all learning-testing splits is a U-statistic. Thus, it has minimal variance among all unbiased estimators and is asymptotically normally dis…
The study analyzes how machine learning classifiers' error rates decrease exponentially based on large deviations theory.
problem Understanding the convergence rate of machine learning classifiers' error probabilities.
method Large deviations theory applied to machine learning classification techniques.
result The error probability of ML classifiers converges to zero exponentially, with a rate dependent on the training set size.
New TTP framework fuses control arms while controlling Type-I error.
problem Bias in borrowing control data from previous trials.
method Kernel two-sample testing via MMD and equivalence testing.
result Higher power than standard TTP methods while maintaining error control.
Paper improves power of conditional randomization tests.
problem Improving power of conditional randomization tests.
method Introducing a new cost function to maximize test statistic power.
result Consistently increases the number of correct discoveries.
Equivalent tests for SGD batch size selection found.
problem Finding equivalent tests for adaptive batch size selection in SGD.
method Norm and inner product/orthogonality tests equivalence demonstration.
result Norm and inner product/orthogonality tests are equivalent under specific conditions.
Gaussian graphical model is a graphical representation of the dependence structure for a Gaussian random vector. It is recognized as a powerful tool in different applied fields such as bioinformatics, error-control codes, speech language, information retrieval and others. Gaussian graphical model selection is a statist…
Large speech dataset for commercial use with 9.98% word error rate.
problem Creating a diverse speech recognition dataset for commercial purposes.
method Internet search for licensed audio data with transcriptions, training model on the dataset.
result Model trained on dataset achieves 9.98% word error rate on Librispeech's test-clean test set.
The study examines conditions for achieving a simple lower bound in estimating mean from samples.
problem Achieving a simple lower bound for estimating the mean of a distribution.
method Analyzes conditions for nearly attaining Le Cam's two-point testing lower bound for mean estimation.
result An algorithm nearly attains the two-point testing rate for mixtures of symmetric, log-concave distributions with a common mean.
New private algorithm for sequential hypothesis testing with privacy and error rate guarantees.
problem Privacy protection in sequential hypothesis testing for sensitive data.
method Renyi differential privacy, Wald's Sequential Probability Ratio Test (SPRT).
result Private algorithm with strong privacy guarantees and theoretical performance analysis.
Optimal classification rules control error rates in multiclass mixture models.
problem Classifying observations in multiclass mixture models while controlling error rates.
method Finding optimal classification rules by searching an optimal region in the observation space, using Maximum A Posteriori (MAP) rule and heuristic computation.
result The FDR-like optimal rule can be significantly less conservative than thresholded MAP rules.
Private two-sample tests under LDP achieve minimax rates for multinomial and continuous data.
problem Achieving statistical utility while maintaining privacy in two-sample testing.
method Private permutation tests for multinomial data and adaptive tests for continuous data.
result Minimax optimal tests for private two-sample testing under LDP.
New method selects best HTE estimator without ground-truth treatment effects.
problem Selecting best HTE estimator from multiple candidates.
method Cross-fitted, exponentially weighted test statistic with two-way sample splitting.
result Empirically, reliable error control and reduced false selections.
Biological research often involves testing a growing number of null hypotheses as new data is accumulated over time. We study the problem of online control of the familywise error rate (FWER), that is testing an apriori unbounded sequence of hypotheses (p-values) one by one over time without knowing the future, such th…
SALSA automatically adjusts learning rates in stochastic gradient methods.
problem Automatic adjustment of learning rates in stochastic gradient methods.
method SALSA uses a line-search procedure to gradually increase the learning rate, then a statistical test to decrease it.
result SALSA matches the performance of best hand-tuned learning rate schedules in deep learning tasks.
Sequential hypothesis testing is a desirable decision making strategy in any time sensitive scenario. Compared with fixed sample-size testing, sequential testing is capable of achieving identical probability of error requirements using less samples in average. For a binary detection problem, it is well known that for k…
The paper introduces localized conformal p-values for conditional testing problems.
problem Addressing conditional testing problems in statistics.
method Localized conformal p-values defined by inverting prediction intervals.
result Proposes procedures for conditional outlier detection and label screening with FDR and FWER control.
Hypothesis testing in the linear regression model is a fundamental statistical problem. We consider linear regression in the high-dimensional regime where the number of parameters exceeds the number of samples (p>n). In order to make informative inference, we assume that the model is approximately sparse, that is th…
Study on error probabilities of machine learning classification techniques using large deviations theory.
problem Performance analysis of machine learning binary classification techniques.
method Large deviations theory applied to Data-Driven Decision Function (D3F) for error probability analysis.
result Classification error probabilities vanish exponentially, with an asymptotic formula providing precise error rate estimates.
Minimizes indecisions in selective classification to control misclassification rates.
problem Controlling misclassification rates in high-risk scenarios.
method Using indecisions to control misclassification rates, even below Bayes optimal.
result Control of misclassification rates to any user-specified level, even below Bayes optimal.
Kernel tests assess equivalence between distributions without assuming specific moments.
problem Traditional goodness-of-fit tests fail to detect meaningful distributional differences.
method Proposes kernel-based tests using kernel Stein discrepancy and Maximum Mean Discrepancy.
result Tests assess the absence of meaningful distributional differences under controlled error rates.
We study nonzero-sum hypothesis testing games that arise in the context of adversarial classification, in both the Bayesian as well as the Neyman-Pearson frameworks. We first show that these games admit mixed strategy Nash equilibria, and then we examine some interesting concentration phenomena of these equilibria. Our…
Improved machine learning models outperform their simpler counterparts by using imperfect labels.
problem Improving model performance using imperfect labels.
method Random feature ridge regression (RFRR) with a deterministic equivalent for excess test error.
result The student model can outperform the teacher model regardless of the teacher's scaling law, achieving the minimax optimal rate.
Paper proposes SCQ and P-TAMS for structured OOD testing in high-stakes ML.
problem Difficulty of incorporating auxiliary information in traditional conformal methods.
method Structure-adaptive conformal q-value (SCQ) and pseudo-score-guided transductive automated model selection (P-TAMS).
result Unified framework controls false discovery rate and improves power across diverse settings.
DP-SPRT improves privacy in sequential tests with near-optimal error rates.
problem Privacy constraints in sequential probability ratio tests.
method A wrapper for SPRT that uses a private mechanism to determine when to stop based on predefined intervals.
result DP-SPRT achieves near-optimal error rates and privacy guarantees.