Paper defines and solves a problem in representation learning to ensure fairness with high confidence.
problem Learning fair representations with high confidence guarantees for all downstream tasks.
method Formally defines the problem, introduces FRG framework, proves high probability fairness, and demonstrates effectiveness empirically.
result FRG framework provides high-confidence guarantees for limiting unfairness across all downstream models and tasks.
ReLU networks often give high confidence far from training data, new technique mitigates this.
problem ReLU networks produce high confidence predictions far from training data, which is undesirable.
method Proposed a new robust optimization technique similar to adversarial training to enforce low confidence predictions.
result The technique reduces confidence of predictions far from training data while maintaining test error on the original task.
The paper provides high-confidence error estimates for learned value functions in large state-spaces.
problem Estimating the accuracy of learned value functions in large, continuous state-spaces.
method Developed a high-confidence bound on empirical value error to true value error, and an offline sampling algorithm to repeatedly compute value error estimates.
result Demonstrated that the offline sampling algorithm can provide high-confidence estimates of value error for learned value functions.
Develops a method to find costly high-confidence errors in black box models.
problem Finding rare high-confidence errors missed by random sampling.
method Adversarial perturbation-guided search technique to find errors at rates greater than expected given model confidence.
result Our Adversarial Distance search discovers high-confidence errors at a rate greater than expected given model confidence.
New method gives high confidence bounds for stochastic convex optimization with minimal overhead.
problem Rare high probability guarantees in stochastic convex optimization.
method ProxBoost algorithm combining robust distance estimation and proximal point method.
result Wide class of stochastic optimization algorithms can achieve high confidence bounds with logarithmic and polylogarithmic overhead.
In the field of reinforcement learning there has been recent progress towards safety and high-confidence bounds on policy performance. However, to our knowledge, no practical methods exist for determining high-confidence policy performance bounds in the inverse reinforcement learning setting---where the true reward fun…
Develops a method to define admissible rewards for robust policy evaluation in RL.
problem Defining a reward function for robust off-policy evaluation in RL with limited data.
method Identifies an admissible set of reward functions ensuring policies are close to past behavior and can be evaluated with high confidence.
result Demonstrates the approach on synthetic and real-world domains, including a critical care application.
New method provides reliable high-confidence prediction intervals for high-impact events.
problem High-impact events require very high confidence prediction intervals, but classical methods provide uninformative intervals.
method Bridge extreme value statistics and conformal prediction to provide reliable and informative prediction intervals.
result Provides reliable and informative prediction intervals with high-confidence coverage.
Mutual teaching improves graph models with less labeled data.
problem Training graph models with limited labeled data.
method Dual model training with mutual teaching strategy.
result Significant performance improvement with less labeled data.
New method reduces variance in stochastic optimization with high confidence.
problem Achieving high-probability guarantees in stochastic optimization with weaker noise assumptions.
method Stochastic proximal point method combining proximal subproblem solver and probability booster.
result Demonstrates convergence with low sample complexity under bounded variance assumptions.
Efficient method for high confidence level inference using parallel stochastic optimization.
problem Uncertainty quantification for online estimation.
method Small number of independent multi-runs to construct t-based confidence intervals.
result Rigorous theoretical guarantee for exact coverage of confidence intervals.
Paper tackles fooling deep networks with minimal perturbations.
problem Easily fooling deep neural networks with high confidence predictions.
method Uses integrated adaptive gradients to generate minimal adversarial perturbations.
result Achieves minimal adversarial perturbations for fooling deep networks.
B-REX efficiently learns Atari game policies from pixel inputs using Bayesian methods.
problem Learning reward functions from visual inputs with uncertainty and safety considerations.
method Bayesian Reward Extrapolation (B-REX) using successor features and preferences.
result B-REX generates posterior samples efficiently, enabling high-confidence performance bounds.
Recent work has shown that state-of-the-art classifiers are quite brittle, in the sense that a small adversarial change of an originally with high confidence correctly classified input leads to a wrong classification again with high confidence. This raises concerns that such classifiers are vulnerable to attacks and ca…
It has been suggested that adversarial examples cause deep learning models to make incorrect predictions with high confidence. In this work, we take the opposite stance: an overly confident model is more likely to be vulnerable to adversarial examples. This work is one of the most proactive approaches taken to date, as…
Paper develops a BERT-based classifier to reduce pathology report annotation workload.
problem Manual annotation of pathology reports is labor-intensive and time-consuming.
method Developed an automatic text classifier using BERT and introduced a human-centric metric to identify low-confidence cases.
result The model reduces manual annotation workload by 80% to 98%.
BALLET filters a high-confidence region of interest for Bayesian optimization.
problem High-dimensional and non-stationary Bayesian optimization challenges.
method Adaptive level-set estimation using two probabilistic models.
result Ballets can efficiently shrink the search space and exhibit tighter regret bounds.
Bayes-TrEx finds in-distribution examples for model inspection.
problem Challenges in interpreting neural networks, especially high-confidence failures and ambiguous classifications.
method Bayesian sampling approach to find in-distribution examples with specified prediction confidence.
result Bayes-TrEx enables more flexible holistic model analysis than just inspecting the test set.
New method predicts sets under unknown covariate shift with high confidence.
problem Adapting to unknown covariate shift in prediction sets.
method PredSet-1Step, a flexible distribution-free method.
result Achieves asymptotic probably approximately correct coverage.
The study analyzes group testing algorithms for identifying defective items with high confidence.
problem Identifying defective items from a population using group testing with high confidence.
method Formulated as a function learning problem using the PAC framework, analyzed three algorithms: column matching, combinatorial basis pursuit, and definite defectives.
result Derived bounds on the number of tests needed for approximate set identification, comparing with existing bounds and simulating performance.
Sampling random points can reveal submanifold topology.
problem Estimating the topology of submanifolds in Riemannian manifolds.
method Sampling random points in a neighborhood of the submanifold.
result Topology of the submanifold can be recovered with high confidence.
Researchers show how to secretly train models with hidden data, detect usage with high confidence.
problem Protecting training data from traceability in large language models.
method Gradient-based optimization to learn secret sequences absent from training data.
result Secret sequences can be learned by models without performance degradation, detectable with high confidence.
Bayesian REX learns Atari games from demonstrations efficiently.
problem Bayesian reward learning for complex control problems is computationally intractable.
method Bayesian Reward Extrapolation (Bayesian REX) pre-trains a low-dimensional feature encoding and uses preferences to perform fast Bayesian inference.
result Bayesian REX learns Atari games from demonstrations in 5 minutes, competitive with state-of-the-art methods.
Deep-RBF networks are made robust to adversarial attacks with a reject option.
problem Vulnerability of deep neural networks to adversarial attacks.
method Revisit deep-RBF networks, propose a family of cost functions, add reject option.
result Significant classification accuracy and robustness to adversarial attacks demonstrated.
The paper proves mutual information measurement is statistically limited.
problem Measuring mutual information from finite data is difficult.
method Proves statistical limitations on any method of measuring mutual information.
result Any distribution-free high-confidence lower bound on mutual information estimated from N samples cannot be larger than O(ln N ).
Bayesian approach calibrates DNN confidence for field use.
problem DNN models give false predictions with high confidence in real-world applications.
method Bayesian approach using Gaussian Process Regression to correct confidence with minimal labeled operation data.
result Significantly reduces high-confidence errors with minimal labeled data.
Approximating non-linear kernels using feature maps has gained a lot of interest in recent years due to applications in reducing training and testing times of SVM classifiers and other kernel based learning algorithms. We extend this line of work and present low distortion embeddings for dot product kernels into linear…
We propose a bootstrap-based robust high-confidence level upper bound (Robust H-CLUB) for assessing the risks of large portfolios. The proposed approach exploits rank-based and quantile-based estimators, and can be viewed as a robust extension of the H-CLUB method (Fan et al., 2015). Such an extension allows us to hand…
The application of deep recurrent networks to audio transcription has led to impressive gains in automatic speech recognition (ASR) systems. Many have demonstrated that small adversarial perturbations can fool deep neural networks into incorrectly predicting a specified target with high confidence. Current work on fool…
In the Best-K identification problem (Best-K-Arm), we are given N stochastic bandit arms with unknown reward distributions. Our goal is to identify the K arms with the largest means with high confidence, by drawing samples from the arms adaptively. This problem is motivated by various practical applications and…
New method for predicting paths of unpredictable objects with high confidence.
problem Need for dependable uncertainty estimates in motion planning with diverse unpredictable objects.
method Blend online conformal prediction, multiple time series techniques, and heteroscedasticity addressing.
result Simultaneous forecasting bands that cover entire paths with high probability.
SNPL learns safe policies for multi-objective interventions with high confidence.
problem Designing effective digital interventions balancing multiple objectives with noisy data.
method Leverages algorithmic stability to learn policies with high-confidence guarantees.
result Offers dramatic improvements in safety and policy gains with smaller sample sizes.
Can we make Bayesian posterior MCMC sampling more efficient when faced with very large datasets? We argue that computing the likelihood for N datapoints in the Metropolis-Hastings (MH) test to reach a single binary decision is computationally inefficient. We introduce an approximate MH rule based on a sequential hypoth…
A new metric for stable model selection in CATE prediction.
problem Model selection in conditional average treatment effect (CATE) prediction.
method Analysis of model performance ranking and formulation of a novel metric.
result Our metric outperforms existing metrics in model selection and hyperparameter tuning.
New method speeds up k-means clustering using sketch-and-solve.
problem Efficiently solving k-means clustering for large datasets.
method Sketch-and-solve approach with Peng-Wei semidefinite relaxation.
result Provides high-confidence lower bounds on k-means optimal value.
The easiness at which adversarial instances can be generated in deep neural networks raises some fundamental questions on their functioning and concerns on their use in critical systems. In this paper, we draw a connection between over-generalization and adversaries: a possible cause of adversaries lies in models desig…
CADRO optimizes DRO by reducing conservatism through cost-aware ambiguity sets.
problem Optimizing solutions under uncertainty with reduced conservatism.
method CADRO uses a cost-aware ambiguity set to reduce DRO's conservatism.
result CADRO provides high-confidence upper bounds and consistent estimators of out-of-sample expected cost.
Machine learning models, especially based on deep architectures are used in everyday applications ranging from self driving cars to medical diagnostics. It has been shown that such models are dangerously susceptible to adversarial samples, indistinguishable from real samples to human eye, adversarial samples lead to in…
We propose a framework for verifying data deletion in MLaaS systems.
problem Ensuring compliance with data deletion requests in MLaaS systems.
method Formal framework based on hypothesis testing, novel backdoor-based verification mechanism.
result Demonstrated high confidence in certifying data deletion with minimal impact on ML service accuracy.
Deep Learning models are vulnerable to adversarial examples, i.e.\ images obtained via deliberate imperceptible perturbations, such that the model misclassifies them with high confidence. However, class confidence by itself is an incomplete picture of uncertainty. We therefore use principled Bayesian methods to capture…
Graph conformal prediction predicts future power outages with high confidence.
problem Accurately predicting future power outages to enable rapid recovery.
method Developed a graph conformal prediction method for quarter-hourly outage data.
result Graph conformal prediction method delivers accurate prediction regions for future outage numbers.
Optimal query allocation improves extractive QA efficiency with LLMs.
problem Efficiency and reliability in extractive question answering with LLMs.
method Learning-to-Defer framework that allocates queries to specialized models.
result Enhanced answer reliability with reduced computational overhead.
Machine learning models are often susceptible to adversarial perturbations of their inputs. Even small perturbations can cause state-of-the-art classifiers with high "standard" accuracy to produce an incorrect prediction with high confidence. To better understand this phenomenon, we study adversarially robust learning …
Several machine learning models, including neural networks, consistently misclassify adversarial examples---inputs formed by applying small but intentionally worst-case perturbations to examples from the dataset, such that the perturbed input results in the model outputting an incorrect answer with high confidence. Ear…
New contest evaluates machine learning robustness against unrestricted adversarial examples.
problem Evaluating machine learning robustness against arbitrary adversarial inputs.
method Two-player contest with a prize pool, focusing on unconstrained adversarial examples.
result Demonstrates the need for comprehensive evaluation of machine learning models' worst-case adversarial risk.
Measuring uncertainty is a promising technique for detecting adversarial examples, crafted inputs on which the model predicts an incorrect class with high confidence. But many measures of uncertainty exist, including predictive en- tropy and mutual information, each capturing different types of uncertainty. We study th…
Simple model finds high correlation in retail crypto returns.
problem Discerning correlation in retail cryptocurrency markets without factors.
method Used N*(N) statistic to compare models of daily returns.
result High average pairwise correlation (60%) found, supports isotropic model.
ICT improves semi-supervised learning by making predictions consistent at data interpolations.
problem Improving semi-supervised learning performance with limited labeled data.
method ICT encourages consistent predictions at data interpolations, reducing overfitting.
result ICT achieves state-of-the-art performance on CIFAR-10 and SVHN datasets.