Adversarial training achieves optimal test error for shallow networks.
problem Achieving optimal adversarial test error for general data distributions.
method Applying new Rademacher complexity bounds and properties of optimal adversarial predictors.
result Adversarial training can achieve optimal adversarial test error for general data distributions.
New minimax optimal learner for robust predictors against adversarial examples.
problem Learning robust predictors against adversarial examples.
method Global perspective and new algorithmic ideas.
result Characterizes classes of predictors that are robustly learnable.
New method learns optimal prediction strategies in adversarial games.
problem Learning optimal prediction procedures in uncertain data environments.
method Adversarial Monte Carlo approach with neural network architecture.
result Optimal strategy is equivariant and invariant to various transformations.
While adversarial training can improve robust accuracy (against an adversary), it sometimes hurts standard accuracy (when there is no adversary). Previous work has studied this tradeoff between standard and robust accuracy, but only in the setting where no predictor performs well on both objectives in the infinite data…
This work precisely characterizes and improves the tradeoff between robustness and accuracy in linear regression.
problem Tradeoff between robustness and accuracy in adversarial training.
method Characterizes the effect of augmentation on standard error in linear regression; proves RST improves robust error without sacrificing standard error.
result RST improves both standard and robust error for neural networks under various perturbations.
Study robust learning without knowing perturbation sets, using interactions with attackers.
problem Learning robust predictors against unknown adversarial perturbations.
method Examined different interaction models with adversarial attackers, derived bounds on sample complexity and interactions.
result Upper bounds on sample complexity and lower bounds on interactions in various models.
New trade-off found between accuracy and adversarial robustness in regression.
problem Finding a balance between accuracy and robustness in regression models.
method Deriving a fundamental trade-off between standard and adversarial risk in regression with polynomial ridge functions.
result A necessary condition for achieving adversarial robustness without significant accuracy loss.
New approach improves robustness of deep neural networks without overfitting.
problem Adversarial vulnerability of deep neural networks.
method Non-zero-sum bilevel formulation of adversarial training.
result Algorithm matches and outperforms state-of-the-art attacks, maintains robustness, and avoids overfitting.
New framework improves adversarial robustness in one-stage L2D.
problem Adversarial robustness in one-stage Learning-to-Defer (L2D).
method Formalizes attacks, proposes cost-sensitive adversarial surrogate losses, establishes theoretical guarantees.
result Improves robustness against untargeted and targeted attacks while preserving clean performance.
Paper develops robust Bayesian models for linear regression under adversarial perturbations.
problem Ensuring reliable machine learning models under data perturbations.
method Formulates adversarial Bregman divergence loss, computes adversarial perturbation, introduces adversarially robust posteriors, derives generalization certificates.
result Derives first rigorous generalization certificates for adversarially robust Bayesian linear regression.
This paper proposes a new algorithmic framework, predictor-verifier training, to train neural networks that are verifiable, i.e., networks that provably satisfy some desired input-output properties. The key idea is to simultaneously train two networks: a predictor network that performs the task at hand,e.g., predicting…
Study shows how to improve contextual bandits with loss predictors.
problem Improving minimax regret in contextual bandits with loss predictors.
method Developed novel algorithmic techniques for upper bounds and lower bounds in various settings.
result Optimal regret is O ( min { T , E T 1 4 } ) \mathcal{O}(\min\{\sqrt{T}, \sqrt{\mathcal{E}}T^\frac{1}{4}\}) O ( min { T , E T 4 1 }) when E \mathcal{E} E is known, and O ( E T 1 3 ) \mathcal{O}(\sqrt{\mathcal{E}}T^\frac{1}{3}) O ( E T 3 1 ) if E \mathcal{E} E is unknown. New method minimizes regret in AMDP with high probability.
problem Pessimistic regret analysis in AMDP.
method Developed a new policy search method achieving optimistic regret.
result Achieved sublinear optimistic regret with high probability.
Simple algorithm achieves distance to calibration error of at most 2√T+1.
problem Achieving distance to calibration error of O(√T) in adversarial setting.
method An extremely simple, efficient, deterministic algorithm.
result Obtains distance to calibration error at most 2√T+1.
This work addresses local fairness in machine learning models.
problem Ensuring fairness within subregions of feature space, not just global averages.
method Introduces ROAD, a Distributionally Robust Optimization (DRO) approach with adversarial learning.
result Achieves Pareto dominance in local fairness and accuracy across datasets.
Many predicted structured objects (e.g., sequences, matchings, trees) are evaluated using the F-score, alignment error rate (AER), or other multivariate performance measures. Since inductively optimizing these measures using training data is typically computationally difficult, empirical risk minimization of surrogate …
This thesis explores robust machine learning against adversarial examples.
problem How to create machine learning systems robust to adversarial examples.
method Theoretical exploration and development of new learning algorithms with robustness guarantees.
result Developed new learning algorithms with provable robustness guarantees.
Adversarial training is a useful approach to promote the learning of transferable representations across the source and target domains, which has been widely applied for domain adaptation (DA) tasks based on deep neural networks. Until very recently, existing adversarial domain adaptation (ADA) methods ignore the usefu…
New method finds invariant predictors to improve model robustness.
problem Machine learning models are brittle in new environments with spurious correlations.
method Formulates invariant risk minimization as a Nash equilibrium game.
result Algorithm yields similar or better accuracy with lower variance.
We present a predictor-corrector framework, called PicCoLO, that can transform a first-order model-free reinforcement or imitation learning algorithm into a new hybrid method that leverages predictive models to accelerate policy learning. The new "PicCoLOed" algorithm optimizes a policy by recursively repeating two ste…
We study the question of learning an adversarially robust predictor. We show that any hypothesis class H \mathcal{H} H with finite VC dimension is robustly PAC learnable with an improper learning rule. The requirement of being improper is necessary as we exhibit examples of hypothesis classes H \mathcal{H} H with finite VC…
Generative Adversarial Networks (GAN) (Goodfellow et al., 2014) are an effective method for training generative models of complex data such as natural images. However, they are notoriously hard to train and can suffer from the problem of missing modes where the model is not able to produce examples in certain regions o…
Boosts barely robust learners to be more adversarially robust.
problem Learning predictors robust to small perturbations on a small fraction of data.
method Oracle-efficient algorithm for robustness with larger perturbation set.
result Qualitative and quantitative equivalence between strongly robust and barely robust learning.
Study adversarial perturbations in classification, analyzing learning and certification.
problem Formal study of classification under adversarial perturbations from both learner and third-party perspectives.
method PAC-type semi-supervised learning framework, black-box certification under limited query budget, adversary analysis.
result Existence of a polynomial query complexity adversary implies the existence of a sample efficient robust learner.
Understanding optimal prompts for binary sequence predictors is challenging.
problem Finding good prompts for binary sequence predictors is difficult.
method Viewing prompting as finding the best conditioning sequence on a near-optimal sequence predictor, using empirical and statistical analysis.
result Optimal prompts can be better understood given the pretraining distribution, which is not usually available.
Two approaches scale up DNN optimization for diverse edge devices.
problem Optimizing DNNs for edge devices with varying performance requirements.
method Reuse performance predictors on proxy devices and build scalable predictors.
result Optimized DNN designs for many different edge devices without lengthy optimization.
Proposes BATer for improved adversarial example detection.
problem Detecting adversarial examples in neural networks.
method Introduces a Bayesian adversarial example detector (BATer) using random components in a Bayesian neural network.
result BATer outperforms state-of-the-art detectors in adversarial example detection.
Post-processing predictors reduces calibration errors for decision-making.
problem Predictors with low calibration error for machine learning may have high error for decision-making.
method Post-processing with ε distance to calibration adds noise to make predictions differentially private.
result Post-processing achieves O(√ε) ECE and CDL, asymptotically optimal.
Predict covariance from features using convex optimization.
problem Predicting the covariance of a Gaussian vector from another feature vector.
method A generalized linear model with convex optimization for fitting parameters.
result Predicted covariance matrices are symmetric positive definite.
Optimal trading strategy with predictor and costs, derived equations and shape.
problem Optimal trading strategy in presence of price predictor, costs, and risk control.
method Path-integral method to derive equations for band edges, solved explicitly for Ornstein-Uhlenbeck predictor.
result Explicit equations and shape of the optimal band strategy derived and analyzed.
We resolve the open problem of optimal sample complexity for multicalibration and deterministic predictors.
problem Optimal sample complexity for multicalibration and deterministic predictors
method Minimax-optimal multicalibration algorithm and generalization to OI predictors
result Minimax-optimal multicalibration algorithm and deterministic predictors with optimal sample complexity
We study online linear regression problems in a distributed setting, where the data is spread over a network. In each round, each network node proposes a linear predictor, with the objective of fitting the \emph{network-wide} data. It then updates its predictor for the next round according to the received local feedbac…
MALT improves adversarial attacks by targeting classes more efficiently.
problem Naive targeting of adversarial attacks based on classifier confidence.
method MALT - Mesoscopic Almost Linearity Targeting, based on medium-scale almost linearity assumptions.
result MALT wins over AutoAttack on CIFAR-100 and ImageNet datasets, five times faster.
Optimizing proper loss yields calibrated models under specific conditions.
problem Understanding when optimizing proper loss functions leads to calibrated predictions.
method Local optimality condition and Lipschitz functions.
result Predictors with local optimality are nearly calibrated and nearly locally optimal.
The paper proposes multicalibration to improve matching in graphs with imperfect predictors.
problem Finding the best matching in graphs with imperfect predictors.
method Introduces multicalibration as a fairness notion to ensure unbiasedness on protected sets of contexts.
result Constructing a multicalibrated predictor that outperforms standard optimal rules in matching algorithms.
The paper develops predictors for functional data on manifolds.
problem Functional data prediction on time-varying manifolds.
method Least-squares local linear Fréchet curve predictor and weighted Fréchet mean approach.
result Asymptotical optimality of the proposed predictors.
Study shows interpolating predictor's risk is optimal in low-dimensional factor regression models.
problem Understanding the risk of interpolating predictors in high-dimensional factor regression models.
method Detailed finite-sample analysis of minimum-norm interpolating predictor's risk in factor regression models.
result The risk of the minimum-norm interpolating predictor approaches optimal benchmarks in low-dimensional factor regression models.
New loss function reduces outage probability in ML-assisted resource allocation.
problem Minimizing outage probability in ML-assisted resource allocation systems.
method Developed a novel loss function and trained an ML model to address the outage probability challenge.
result Exact and asymptotic expressions for the system's outage probability were established.
DADI framework dynamically discovers fair information using reinforcement learning.
problem Discovering fair information from third-party features with unknown objectives.
method Adversarial reinforcement learning agent that balances accuracy and fairness.
result Achieves group fairness by rewarding the agent with the adversary's loss.
Ensemble methods improve model performance by averaging over subsampled predictors.
problem Understanding the effect of feature subsampling in ensemble methods.
method Fit linear predictors using ordinary least squares on random submatrices of the data matrix.
result The asymptotic risk of an ensemble is equal to the ridge regression risk, optimal for linear predictors.
Forecasting a time series from multivariate predictors constitutes a challenging problem, especially using model-free approaches. Most techniques, such as nearest-neighbor prediction, quickly suffer from the curse of dimensionality and overfitting for more than a few predictors which has limited their application mostl…
Methods of transfer learning try to combine knowledge from several related tasks (or domains) to improve performance on a test task. Inspired by causal methodology, we relax the usual covariate shift assumption and assume that it holds true for a subset of predictor variables: the conditional distribution of the target…
APQ jointly optimizes neural architecture, pruning, and quantization for efficient inference.
problem Efficient deep learning inference on resource-constrained hardware.
method Joint optimization of neural architecture, pruning, and quantization policy using a quantization-aware accuracy predictor.
result Joint optimization leads to 2.3% higher ImageNet accuracy with reduced latency and energy consumption.
NPENAS improves neural architecture search efficiency and accuracy.
problem Efficient and accurate neural architecture search (NAS) for minimizing search costs.
method Proposes NPENAS, a neural predictor guided evolutionary algorithm that enhances exploration ability of evolutionary algorithms.
result NPENAS-BO and NPENAS-NP outperform existing NAS algorithms on NASBench-201, NASBench-101, and DARTS.
Study shows how pretraining robustness transfers to downstream tasks.
problem Understanding how robustness is transferred from pretraining to downstream tasks.
method Theoretical analysis and practical validation of robustness constraints.
result Robustness of a linear predictor on downstream tasks can be constrained by the robustness of its underlying representation.
BANANAS uses Bayesian optimization with neural predictors to improve neural architecture search.
problem Improving neural architecture search (NAS) performance.
method BANANAS combines Bayesian optimization with neural predictors, focusing on architecture encoding, predictor, uncertainty calibration, acquisition function, and optimization strategy.
result BANANAS achieves state-of-the-art performance on NAS search spaces.
We identify a trade-off between robustness and accuracy that serves as a guiding principle in the design of defenses against adversarial examples. Although this problem has been widely studied empirically, much remains unknown concerning the theory underlying this trade-off. In this work, we decompose the prediction er…
A graph VAE framework optimizes neural architectures in a continuous space.
problem Discovering efficient neural architectures in a discrete space.
method Graph VAE framework with VAE and GNN components, joint learning of predictors and decoders.
result The framework discovers powerful neural architectures with both excellent performance and high computational efficiency.