New regularization method reduces support of empirical risk minimization solutions.
problem Regularization in empirical risk minimization with relative entropy.
method Introduces Type-II regularization, characterizes solutions, analyzes properties of relative entropy.
result Type-II regularization collapses solution support into reference measure's support.
ERM with f-divergence regularization yields unique solution.
problem Optimizing empirical risk with f-divergence. method Mild conditions on f lead to unique optimal measure. result Equivalence of ERM-fDR to different f-divergence regularization. In this paper, we present a simple analysis of {\bf fast rates} with {\it high probability} of {\bf empirical minimization} for {\it stochastic composite optimization} over a finite-dimensional bounded convex set with exponential concave loss functions and an arbitrary convex regularization. To the best of our knowledg…
Deep neural network with l_1-regularization achieves nearly optimal risk bounds.
problem Achieving optimal risk bounds in deep learning.
method Empirical risk minimization with l_1-regularization.
result Adaptively nearly-minimax risk bound across various function classes.
AMP regularization improves deep learning models by favoring flat minima.
problem Improving deep learning model generalization and avoiding overfitting.
method AMP regularization uses adversarial model perturbation to minimize a norm-bounded perturbation of the empirical risk.
result AMP regularization leads to state-of-the-art performance across various deep architectures.
Entropy asymmetry affects regularization in ERM, leading to biased solutions.
problem Analyzing the impact of relative entropy asymmetry in ERM regularization.
method Examined Type-I and Type-II ERM-RER, comparing their solutions and properties.
result Type-II ERM-RER regularization introduces a strong bias against training data.
Dual optimization connects ERM-fDR to normalization function.
problem Empirical risk minimization with f-divergence regularization.
method Dual formulation, Legendre-Fenchel transform, implicit function theorem, nonlinear ODE.
result Computational method to calculate normalization function efficiently.
Paper presents ERM with f-divergence regularization and its properties.
problem Minimizing empirical risk with f-divergence constraints. method Introduces normalization function and solves ERM-fDR via ODE. result Characterizes difference between empirical risks and provides numerical algorithm.
This work proves L2-regularized ERM controls smCE without post-hoc correction.
problem Calibration of predicted probabilities in machine learning models.
method Canonical L2-regularized empirical risk minimization. result Theoretical proof that smCE is controlled by ERM without post-hoc correction.
We develop a family of accelerated stochastic algorithms that minimize sums of convex functions. Our algorithms improve upon the fastest running time for empirical risk minimization (ERM), and in particular linear least-squares regression, across a wide range of problem settings. To achieve this, we establish a framewo…
A regularized risk minimization procedure for regression function estimation is introduced that achieves near optimal accuracy and confidence under general conditions, including heavy-tailed predictor and response variables. The procedure is based on median-of-means tournaments, introduced by the authors in [8]. It is …
We consider a composite convex minimization problem associated with regularized empirical risk minimization, which often arises in machine learning. We propose two new stochastic gradient methods that are based on stochastic dual averaging method with variance reduction. Our methods generate a sparser solution than the…
Many machine learning algorithms minimize a regularized risk, and stochastic optimization is widely used for this task. When working with massive data, it is desirable to perform stochastic optimization in parallel. Unfortunately, many existing stochastic optimization algorithms cannot be parallelized efficiently. In t…
The study uncovers the breakdown of Gaussian universality in high-dimensional empirical risk minimization.
problem Understanding the breakdown of Gaussian universality in high-dimensional empirical risk minimization.
method Extending the Convex Gaussian Min-Max Theorem to non-Gaussian settings, deriving asymptotic min-max characterizations, and proving asymptotic equivalence of regularizers.
result The projection of the ERM estimator onto a test covariate approximately follows a Gaussian convolution under certain conditions.
This study connects Jacobian regularization to adversarial robustness and improves generalization.
problem Adversarial attacks make deep neural networks vulnerable.
method Developed a connection between Jacobian regularization and adversarial training, and established robust generalization gaps.
result Jacobian norms are related to both standard and robust generalization.
Gradient descent implicitly follows regularization for general losses.
problem The implicit bias of gradient descent methods in machine learning.
method Empirical risk minimization over linear predictors with arbitrary convex, strictly decreasing losses.
result Gradient descent and regularization paths converge to the same direction for non-attained risks.
Study shows how neural networks generalize with minimal training data.
problem Understanding how neural networks generalize with limited data.
method Mean-field analysis of KL-regularized empirical risk minimization.
result Generalization error rate is O(1/n) for large n. New method improves adversarial robustness of neural networks.
problem Vulnerability of neural networks to adversarial attacks.
method Integrates φ-divergence regularization into distributionally robust risk function.
result Achieves near-optimal sample complexity with efficient stochastic gradient methods.
NAPP-ERM improves ERM with differential privacy guarantees by iteratively achieving target regularization and delivering strong convexity.
problem Over-regularization in privacy-preserving ERM approaches.
method Noise-Augmented Privacy-Preserving Empirical Risk Minimization (NAPP-ERM) with a dual-purpose l2 regularizer and privacy budget retrieval strategy.
result Mitigates over-regularization and achieves strong convexity through a single regularizer.
We consider a generic convex optimization problem associated with regularized empirical risk minimization of linear predictors. The problem structure allows us to reformulate it as a convex-concave saddle point problem. We propose a stochastic primal-dual coordinate (SPDC) method, which alternates between maximizing ov…
Differential privacy is concerned about the prediction quality while measuring the privacy impact on individuals whose information is contained in the data. We consider differentially private risk minimization problems with regularizers that induce structured sparsity. These regularizers are known to be convex but they…
We develop an approach to risk minimization and stochastic optimization that provides a convex surrogate for variance, allowing near-optimal and computationally efficient trading between approximation and estimation error. Our approach builds off of techniques for distributionally robust optimization and Owen's empiric…
New method turns optimization algorithms into uniformly stable learning algorithms for non-Euclidean norms.
problem Non-Euclidean norms in binary classification problems.
method Black-box reduction method using uniformly convex regularizers.
result Achieves optimal statistical risk bounds on excess risk for non-Euclidean norms.
Covariance shrinkage via stochastic interpolation
problem High-dimensional covariance estimation
method Recasting shrinkage as empirical risk minimization
result Reduces statistical risk through scheduling, flow maps, and early stopping
New algorithms delete user data from machine learning models efficiently.
problem Deleting user data from machine learning models trained with empirical risk minimization.
method Developed an online unlearning algorithm using the infinitesimal jackknife, targeting non-smooth regularizers.
result Empirically improved runtime while maintaining memory requirements and test accuracy.
This work characterizes optimal multiclass learning with regularization.
problem The empirical risk minimization (ERM) algorithm fails in multiclass learning settings.
method Using one-inclusion graphs (OIGs), the work introduces optimal learning algorithms that relax structural risk minimization and incorporate unsupervised learning.
result An optimal learner is introduced that uses a local regularization function and an unsupervised learning stage to learn the regularizer.
Adaptive LR improves neural network Lipschitz regularity without slowing convergence.
problem Training neural networks with high Lipschitz constant.
method Adaptive learning rate (decay) in gradient descent for two-layer networks.
result Decaying learning rate maintains convergence rate and Lipschitz regularity.
The paper improves semi-supervised learning using f-divergences and α-Rényi divergences.
problem Improving semi-supervised learning with noisy pseudo-labels.
method Inspired by f-divergences and α-Rényi divergences, the paper develops new empirical risk functions and regularization techniques. result The new methods show better performance than traditional self-training methods, especially in noisy pseudo-label scenarios.
In this paper we study the differentially private Empirical Risk Minimization (ERM) problem in different settings. For smooth (strongly) convex loss function with or without (non)-smooth regularization, we give algorithms that achieve either optimal or near optimal utility bounds with less gradient complexity compared …
We design simple screening tests to automatically discard data samples in empirical risk minimization without losing optimization guarantees. We derive loss functions that produce dual objectives with a sparse solution. We also show how to regularize convex losses to ensure such a dual sparsity-inducing property, and p…
Mirror descent algorithm recovers low-rank matrices in matrix sensing.
problem Matrix sensing with low-rank matrices under certain conditions.
method Discrete-time mirror descent applied to empirical risk with Bregman divergence analysis.
result Mirror descent converges to a matrix minimizing a specific nuclear norm-related quantity.
Mixup improves model accuracy and calibration through data transformation and random perturbation.
problem Improving model accuracy and calibration in machine learning.
method Interprets Mixup as empirical risk minimization with data transformation and random perturbation.
result Mixup induces multiple known regularization schemes that prevent overfitting and overconfident predictions.
Simplifies risk minimization combining mean and standard deviation.
problem Minimizing mean and standard deviation under heavy-tailed losses.
method Adapting robust mean estimation technique to include standard deviation.
result Simple approach performs as well or better than alternative risk criteria.
This article provides, through theoretical analysis, an in-depth understanding of the classification performance of the empirical risk minimization framework, in both ridge-regularized and unregularized cases, when high dimensional data are considered. Focusing on the fundamental problem of separating a two-class Gauss…
New algorithm reduces online logistic regression regret without exponential constant.
problem Improper learning in online logistic regression with logarithmic regret.
method Regularized empirical risk minimization with surrogate losses.
result Regret scaling as O(B log(Bn)) with low computational complexity.
We study Empirical Risk Minimizers (ERM) and Regularized Empirical Risk Minimizers (RERM) for regression problems with convex and L-Lipschitz loss functions. We consider a setting where $|\cO|$ malicious outliers contaminate the labels. In that case, under a local Bernstein condition, we show that the L2-error rat…
The paper sets fundamental limits for ERM in high dimensions.
problem Understanding statistical accuracy of ERM in high-dimensional settings.
method Sharp performance characterizations and tight lower bounds derived for generalized linear models.
result Optimal tuning of loss function and regularization parameter.
This work analyzes how to choose regularization norms for adversarial training in high dimensions.
problem Choosing the right regularization norm for adversarial training in high-dimensional settings.
method Derives asymptotic descriptions and uniform convergence bounds for robust, regularized empirical risk minimizers.
result Characterizes the relationship between perturbation size and optimal regularization choice.
Randomized algorithm solves vector-valued regression problems with low-rank operators.
problem Vector-valued regression problems involving infinite-dimensional spaces.
method Randomized Reduced Rank Regression (R4) using Gaussian sketching for optimization.
result R4 estimators are efficient and accurate, with empirical risk close to optimal.
Introduces self-regularization for analyzing learning algorithms.
problem Analyzing and optimizing learning algorithms without explicit regularization.
method Develops a self-regularization framework for learning algorithms.
result Provides statistical analysis and minmax-optimal rates for self-regularized algorithms.
New algorithm improves robustness by more regularization on less robust samples.
problem Improving robustness against adversarial attacks.
method Proposes a new adversarial training algorithm that applies more regularization to data vulnerable to adversarial attacks.
result Empirically superior to other existing algorithms, achieving state-of-the-art performance in generalization and robustness.
We study the risk performance of distributed learning for the regularization empirical risk minimization with fast convergence rate, substantially improving the error analysis of the existing divide-and-conquer based distributed learning. An interesting theoretical finding is that the larger the diversity of each local…
We propose a general approach for supervised learning with structured output spaces, such as combinatorial and polyhedral sets, that is based on minimizing estimated conditional risk functions. Given a loss function defined over pairs of output labels, we first estimate the conditional risk function by solving a (possi…
Proposes a new framework for learning image augmentations to improve classification performance.
problem Improving classification performance with a given class of predictors.
method Transformed Risk Minimization (TRM) framework that optimizes both predictive models and data transformations.
result Performance of TRM with SCALE algorithm compares favorably to prior methods on CIFAR10/100.
Improved ADMM for convex distributed learning with differential privacy.
problem Privacy concerns in distributed learning with sensitive data.
method Approximate multi-step ADMM with calibrated noise.
result Higher utility and error bounds asymptotic to state-of-the-art.
The goal of regression and classification methods in supervised learning is to minimize the empirical risk, that is, the expectation of some loss function quantifying the prediction error under the empirical distribution. When facing scarce training data, overfitting is typically mitigated by adding regularization term…
This work considers the problem of binary classification: given training data x1,…,xn from a certain population, together with associated labels y1,…,yn∈{0,1}, determine the best label for an element x not among the training data. More specifically, this work considers a variant o…
Regularized empirical risk minimization including support vector machines plays an important role in machine learning theory. In this paper regularized pairwise learning (RPL) methods based on kernels will be investigated. One example is regularized minimization of the error entropy loss which has recently attracted qu…