New regularization method reduces support of empirical risk minimization solutions.
problem Regularization in empirical risk minimization with relative entropy.
method Introduces Type-II regularization, characterizes solutions, analyzes properties of relative entropy.
result Type-II regularization collapses solution support into reference measure's support.
ERM with f-divergence regularization yields unique solution.
problem Optimizing empirical risk with f-divergence. method Mild conditions on f lead to unique optimal measure. result Equivalence of ERM-fDR to different f-divergence regularization. Unified framework for shrinkage, thresholding, and regularization in normal mean estimation and linear regression.
problem Estimation of normal mean in multivariate settings with correlated observations.
method Approximate risk minimization over a functional class of shrinkage-thresholding rules.
result Unified estimator NOMAD for shrinkage, thresholding, and regularization.
Deep neural network with l_1-regularization achieves nearly optimal risk bounds.
problem Achieving optimal risk bounds in deep learning.
method Empirical risk minimization with l_1-regularization.
result Adaptively nearly-minimax risk bound across various function classes.
New geometric insights reveal properties of adversarial training problems.
problem Adversarial training in binary classification.
method Equivalence with regularized risk minimization and convex relaxations.
result Existence of minimal and maximal solutions, and regular solutions.
Many machine learning algorithms minimize a regularized risk, and stochastic optimization is widely used for this task. When working with massive data, it is desirable to perform stochastic optimization in parallel. Unfortunately, many existing stochastic optimization algorithms cannot be parallelized efficiently. In t…
AMP regularization improves deep learning models by favoring flat minima.
problem Improving deep learning model generalization and avoiding overfitting.
method AMP regularization uses adversarial model perturbation to minimize a norm-bounded perturbation of the empirical risk.
result AMP regularization leads to state-of-the-art performance across various deep architectures.
In this paper, we present a simple analysis of {\bf fast rates} with {\it high probability} of {\bf empirical minimization} for {\it stochastic composite optimization} over a finite-dimensional bounded convex set with exponential concave loss functions and an arbitrary convex regularization. To the best of our knowledg…
Differential privacy is concerned about the prediction quality while measuring the privacy impact on individuals whose information is contained in the data. We consider differentially private risk minimization problems with regularizers that induce structured sparsity. These regularizers are known to be convex but they…
A regularized risk minimization procedure for regression function estimation is introduced that achieves near optimal accuracy and confidence under general conditions, including heavy-tailed predictor and response variables. The procedure is based on median-of-means tournaments, introduced by the authors in [8]. It is …
Entropy asymmetry affects regularization in ERM, leading to biased solutions.
problem Analyzing the impact of relative entropy asymmetry in ERM regularization.
method Examined Type-I and Type-II ERM-RER, comparing their solutions and properties.
result Type-II ERM-RER regularization introduces a strong bias against training data.
Dual optimization connects ERM-fDR to normalization function.
problem Empirical risk minimization with f-divergence regularization.
method Dual formulation, Legendre-Fenchel transform, implicit function theorem, nonlinear ODE.
result Computational method to calculate normalization function efficiently.
This work proves L2-regularized ERM controls smCE without post-hoc correction.
problem Calibration of predicted probabilities in machine learning models.
method Canonical L2-regularized empirical risk minimization. result Theoretical proof that smCE is controlled by ERM without post-hoc correction.
In regularized risk minimization, the associated optimization problem becomes particularly difficult when both the loss and regularizer are nonsmooth. Existing approaches either have slow or unclear convergence properties, are restricted to limited problem subclasses, or require careful setting of a smoothing parameter…
We present a Bayesian view of counterfactual risk minimization (CRM) for offline learning from logged bandit feedback. Using PAC-Bayesian analysis, we derive a new generalization bound for the truncated inverse propensity score estimator. We apply the bound to a class of Bayesian policies, which motivates a novel, pote…
This study connects Jacobian regularization to adversarial robustness and improves generalization.
problem Adversarial attacks make deep neural networks vulnerable.
method Developed a connection between Jacobian regularization and adversarial training, and established robust generalization gaps.
result Jacobian norms are related to both standard and robust generalization.
Paper presents ERM with f-divergence regularization and its properties.
problem Minimizing empirical risk with f-divergence constraints. method Introduces normalization function and solves ERM-fDR via ODE. result Characterizes difference between empirical risks and provides numerical algorithm.
We consider a composite convex minimization problem associated with regularized empirical risk minimization, which often arises in machine learning. We propose two new stochastic gradient methods that are based on stochastic dual averaging method with variance reduction. Our methods generate a sparser solution than the…
New method improves adversarial robustness of neural networks.
problem Vulnerability of neural networks to adversarial attacks.
method Integrates φ-divergence regularization into distributionally robust risk function.
result Achieves near-optimal sample complexity with efficient stochastic gradient methods.
Gradient descent implicitly follows regularization for general losses.
problem The implicit bias of gradient descent methods in machine learning.
method Empirical risk minimization over linear predictors with arbitrary convex, strictly decreasing losses.
result Gradient descent and regularization paths converge to the same direction for non-attained risks.
This work characterizes optimal multiclass learning with regularization.
problem The empirical risk minimization (ERM) algorithm fails in multiclass learning settings.
method Using one-inclusion graphs (OIGs), the work introduces optimal learning algorithms that relax structural risk minimization and incorporate unsupervised learning.
result An optimal learner is introduced that uses a local regularization function and an unsupervised learning stage to learn the regularizer.
We develop a family of accelerated stochastic algorithms that minimize sums of convex functions. Our algorithms improve upon the fastest running time for empirical risk minimization (ERM), and in particular linear least-squares regression, across a wide range of problem settings. To achieve this, we establish a framewo…
Research shows minimal communication limits adaptive function estimation rates.
problem Adaptive estimation of a smooth function under minimal communication constraints.
method Investigates the L∞-risk and L2-risk under different numbers of servers. result For L∞-risk, optimal rates cannot be achieved under minimal communication. For L2-risk, adaptivity is possible but depends on server number and sample size. Study improves adversarial classification using distributionally robust models.
problem Improving robustness against adversarial attacks in classification models.
method Distributionally robust chance constraints with Wasserstein ambiguity, reformulated as a regularized ramp loss minimization problem.
result Standard descent methods can converge to the global minimizer for the distributionally robust adversarial classification model.
NAPP-ERM improves ERM with differential privacy guarantees by iteratively achieving target regularization and delivering strong convexity.
problem Over-regularization in privacy-preserving ERM approaches.
method Noise-Augmented Privacy-Preserving Empirical Risk Minimization (NAPP-ERM) with a dual-purpose l2 regularizer and privacy budget retrieval strategy.
result Mitigates over-regularization and achieves strong convexity through a single regularizer.
The study uncovers the breakdown of Gaussian universality in high-dimensional empirical risk minimization.
problem Understanding the breakdown of Gaussian universality in high-dimensional empirical risk minimization.
method Extending the Convex Gaussian Min-Max Theorem to non-Gaussian settings, deriving asymptotic min-max characterizations, and proving asymptotic equivalence of regularizers.
result The projection of the ERM estimator onto a test covariate approximately follows a Gaussian convolution under certain conditions.
Study shows how neural networks generalize with minimal training data.
problem Understanding how neural networks generalize with limited data.
method Mean-field analysis of KL-regularized empirical risk minimization.
result Generalization error rate is O(1/n) for large n. Simplifies risk minimization combining mean and standard deviation.
problem Minimizing mean and standard deviation under heavy-tailed losses.
method Adapting robust mean estimation technique to include standard deviation.
result Simple approach performs as well or better than alternative risk criteria.
The paper analyzes the risk of CV-tuned regularized estimators and connects it to SURE.
problem Understanding the risk of CV-tuned regularized estimators.
method Derives asymptotic risk function of CV-tuned estimators and connects it to SURE.
result The risk function provides a more detailed picture of predictive performance than uniform bounds.
We study the task of learning from non-i.i.d. data. In particular, we aim at learning predictors that minimize the conditional risk for a stochastic process, i.e. the expected loss of the predictor on the next point conditioned on the set of training samples observed so far. For non-i.i.d. data, the training set contai…
We propose a general approach for supervised learning with structured output spaces, such as combinatorial and polyhedral sets, that is based on minimizing estimated conditional risk functions. Given a loss function defined over pairs of output labels, we first estimate the conditional risk function by solving a (possi…
We consider a generic convex optimization problem associated with regularized empirical risk minimization of linear predictors. The problem structure allows us to reformulate it as a convex-concave saddle point problem. We propose a stochastic primal-dual coordinate (SPDC) method, which alternates between maximizing ov…
Adaptive LR improves neural network Lipschitz regularity without slowing convergence.
problem Training neural networks with high Lipschitz constant.
method Adaptive learning rate (decay) in gradient descent for two-layer networks.
result Decaying learning rate maintains convergence rate and Lipschitz regularity.
New method turns optimization algorithms into uniformly stable learning algorithms for non-Euclidean norms.
problem Non-Euclidean norms in binary classification problems.
method Black-box reduction method using uniformly convex regularizers.
result Achieves optimal statistical risk bounds on excess risk for non-Euclidean norms.
Covariance shrinkage via stochastic interpolation
problem High-dimensional covariance estimation
method Recasting shrinkage as empirical risk minimization
result Reduces statistical risk through scheduling, flow maps, and early stopping
New algorithms delete user data from machine learning models efficiently.
problem Deleting user data from machine learning models trained with empirical risk minimization.
method Developed an online unlearning algorithm using the infinitesimal jackknife, targeting non-smooth regularizers.
result Empirically improved runtime while maintaining memory requirements and test accuracy.
Regularized empirical risk minimization including support vector machines plays an important role in machine learning theory. In this paper regularized pairwise learning (RPL) methods based on kernels will be investigated. One example is regularized minimization of the error entropy loss which has recently attracted qu…
This work analyzes how to choose regularization norms for adversarial training in high dimensions.
problem Choosing the right regularization norm for adversarial training in high-dimensional settings.
method Derives asymptotic descriptions and uniform convergence bounds for robust, regularized empirical risk minimizers.
result Characterizes the relationship between perturbation size and optimal regularization choice.
We study Empirical Risk Minimizers (ERM) and Regularized Empirical Risk Minimizers (RERM) for regression problems with convex and L-Lipschitz loss functions. We consider a setting where $|\cO|$ malicious outliers contaminate the labels. In that case, under a local Bernstein condition, we show that the L2-error rat…
New privacy framework tailored to specific data distributions.
problem Protecting individual data points in decision-making processes.
method Introducing tangent differential privacy, a new form of differential privacy.
result Entropic regularization guarantees tangent differential privacy under general conditions.
Proposes a new framework for learning image augmentations to improve classification performance.
problem Improving classification performance with a given class of predictors.
method Transformed Risk Minimization (TRM) framework that optimizes both predictive models and data transformations.
result Performance of TRM with SCALE algorithm compares favorably to prior methods on CIFAR10/100.
This article provides, through theoretical analysis, an in-depth understanding of the classification performance of the empirical risk minimization framework, in both ridge-regularized and unregularized cases, when high dimensional data are considered. Focusing on the fundamental problem of separating a two-class Gauss…
New algorithm reduces online logistic regression regret without exponential constant.
problem Improper learning in online logistic regression with logarithmic regret.
method Regularized empirical risk minimization with surrogate losses.
result Regret scaling as O(B log(Bn)) with low computational complexity.
We develop an approach to risk minimization and stochastic optimization that provides a convex surrogate for variance, allowing near-optimal and computationally efficient trading between approximation and estimation error. Our approach builds off of techniques for distributionally robust optimization and Owen's empiric…
The paper sets fundamental limits for ERM in high dimensions.
problem Understanding statistical accuracy of ERM in high-dimensional settings.
method Sharp performance characterizations and tight lower bounds derived for generalized linear models.
result Optimal tuning of loss function and regularization parameter.
Introduces self-regularization for analyzing learning algorithms.
problem Analyzing and optimizing learning algorithms without explicit regularization.
method Develops a self-regularization framework for learning algorithms.
result Provides statistical analysis and minmax-optimal rates for self-regularized algorithms.
Study characterizes training and test risks for MAP regression with Gaussian priors.
problem Understanding high-dimensional behavior of regularized linear regression with informative priors.
method Maximum a posteriori (MAP) regression with Gaussian priors, using random matrix theory.
result Closed-form risk formulas reveal the bias-variance-prior tradeoff and explain double descent.
We design simple screening tests to automatically discard data samples in empirical risk minimization without losing optimization guarantees. We derive loss functions that produce dual objectives with a sparse solution. We also show how to regularize convex losses to ensure such a dual sparsity-inducing property, and p…