NAPP-ERM improves ERM with differential privacy guarantees by iteratively achieving target regularization and delivering strong convexity.
problem Over-regularization in privacy-preserving ERM approaches.
method Noise-Augmented Privacy-Preserving Empirical Risk Minimization (NAPP-ERM) with a dual-purpose l2 regularizer and privacy budget retrieval strategy.
result Mitigates over-regularization and achieves strong convexity through a single regularizer.
Entropy asymmetry affects regularization in ERM, leading to biased solutions.
problem Analyzing the impact of relative entropy asymmetry in ERM regularization.
method Examined Type-I and Type-II ERM-RER, comparing their solutions and properties.
result Type-II ERM-RER regularization introduces a strong bias against training data.
Dual optimization connects ERM-fDR to normalization function.
problem Empirical risk minimization with f-divergence regularization.
method Dual formulation, Legendre-Fenchel transform, implicit function theorem, nonlinear ODE.
result Computational method to calculate normalization function efficiently.
New regularization method reduces support of empirical risk minimization solutions.
problem Regularization in empirical risk minimization with relative entropy.
method Introduces Type-II regularization, characterizes solutions, analyzes properties of relative entropy.
result Type-II regularization collapses solution support into reference measure's support.
Paper presents ERM with f-divergence regularization and its properties.
problem Minimizing empirical risk with f-divergence constraints. method Introduces normalization function and solves ERM-fDR via ODE. result Characterizes difference between empirical risks and provides numerical algorithm.
New dual formulation reduces generalization error for ERM-fDR.
problem Generalization error in constrained optimization problems.
method Introduces a dual formulation of ERM-fDR using Legendre-Fenchel transform and implicit function theorem.
result Explicit characterizations of generalization error for algorithms under mild conditions.
This work proves L2-regularized ERM controls smCE without post-hoc correction.
problem Calibration of predicted probabilities in machine learning models.
method Canonical L2-regularized empirical risk minimization. result Theoretical proof that smCE is controlled by ERM without post-hoc correction.
A new distributed optimization method for ERM problems.
problem Efficiently solving ERM problems with nonsmooth regularization in a distributed setting.
method Second-order distributed optimization using successive quadratic approximations and Hessian approximation.
result Global linear convergence for a broad range of non-strongly convex problems.
ERM and RERM minimize error even with malicious label corruptions.
problem Malicious label corruptions in regression problems.
method Empirical Risk Minimizers (ERM) and Regularized Empirical Risk Minimizers (RERM) under a local Bernstein condition.
result The L2-error rate is bounded by $r_N + AL |\cO|/N$ under the local Bernstein condition. ERM with f-divergence regularization yields unique solution.
problem Optimizing empirical risk with f-divergence. method Mild conditions on f lead to unique optimal measure. result Equivalence of ERM-fDR to different f-divergence regularization. Unified view connects CoCoA and ADMM for distributed ERM.
problem Connection between CoCoA and ADMM for distributed ERM.
method Unified primal-dual perspective reformulation.
result Unified ADMM variants perform at least as good as CoCoA in ridge-regularized ERM.
Study shows how bias in optimization affects robustness in adversarial settings.
problem Understanding and mitigating implicit bias in adversarially robust models.
method Analyzes the implicit bias in robust empirical risk minimization and its impact on generalization.
result Implicit bias in optimization can significantly affect robust generalization.
New algorithm reduces ERM problem size while maintaining accuracy.
problem Empirical risk minimization problem size reduction.
method Adaptive Deterministic Uniform-Weight Trimming (ADUWT) algorithm.
result Uniform (1±ε) relative-error approximation for ERM objective. The study uncovers the breakdown of Gaussian universality in high-dimensional empirical risk minimization.
problem Understanding the breakdown of Gaussian universality in high-dimensional empirical risk minimization.
method Extending the Convex Gaussian Min-Max Theorem to non-Gaussian settings, deriving asymptotic min-max characterizations, and proving asymptotic equivalence of regularizers.
result The projection of the ERM estimator onto a test covariate approximately follows a Gaussian convolution under certain conditions.
Decentralized learning achieves centralized performance via Gibbs measures.
problem Achieving centralized performance in decentralized machine learning.
method ERM-RER learning framework with Gibbs measures and relative-entropy regularization.
result Achieving centralized performance with Gibbs measures and specific scaling of regularization factors.
Training deep belief networks (DBNs) requires optimizing a non-convex function with an extremely large number of parameters. Naturally, existing gradient descent (GD) based methods are prone to arbitrarily poor local minima. In this paper, we rigorously show that such local minima can be avoided (upto an approximation …
We propose a communication- and computation-efficient distributed optimization algorithm using second-order information for solving ERM problems with a nonsmooth regularization term. Current second-order and quasi-Newton methods for this problem either do not work well in the distributed setting or work only for specif…
New DP-CD method outperforms DP-SGD in solving composite DP-ERM problems.
problem Privacy-preserving machine learning with differential privacy.
method Differentially Private proximal Coordinate Descent (DP-CD) for composite Empirical Risk Minimization (ERM).
result DP-CD outperforms DP-SGD due to larger step sizes and better gradient exploitation.
The paper sets fundamental limits for ERM in high dimensions.
problem Understanding statistical accuracy of ERM in high-dimensional settings.
method Sharp performance characterizations and tight lower bounds derived for generalized linear models.
result Optimal tuning of loss function and regularization parameter.
Empirical risk minimization (ERM), with proper loss function and regularization, is the common practice of supervised classification. In this paper, we study training arbitrary (from linear to deep) binary classifier from only unlabeled (U) data by ERM. We prove that it is impossible to estimate the risk of an arbitrar…
Recent work shows GRW approaches do not improve over ERM in distributional shift.
problem Improving robustness to distributional shift in machine learning models.
method Generalized Reweighting (GRW) algorithms, which iteratively update model parameters based on reweighting of training samples.
result GRW approaches do not significantly improve over ERM in real applications with distribution shift.
Optimizes differentially private kernel learning with random projection.
problem Privacy-preserving learning algorithms with optimal performance.
method Differentially private kernel ERM algorithm based on random projection in reproducing kernel Hilbert space.
result Achieves minimax-optimal excess risk rates for various loss functions.
Data is continuously generated by modern data sources, and a recent challenge in machine learning has been to develop techniques that perform well in an incremental (streaming) setting. In this paper, we investigate the problem of private machine learning, where as common in practice, the data is not given at once, but…
In this paper we study the differentially private Empirical Risk Minimization (ERM) problem in different settings. For smooth (strongly) convex loss function with or without (non)-smooth regularization, we give algorithms that achieve either optimal or near optimal utility bounds with less gradient complexity compared …
PAIR optimizes machine learning models to generalize better to out-of-distribution data.
problem Optimization of machine learning models for out-of-distribution (OOD) generalization often leads to compromises that weaken robustness.
method Introduces a multi-objective optimization (MOO) perspective and a new optimization scheme called PAreto Invariant Risk Minimization (PAIR).
result PAIR improves robustness of OOD objectives by cooperatively optimizing with other objectives, yielding top OOD performances.
In this work we develop a new algorithm for regularized empirical risk minimization. Our method extends recent techniques of Shalev-Shwartz [02/2015], which enable a dual-free analysis of SDCA, to arbitrary mini-batching schemes. Moreover, our method is able to better utilize the information in the data defining the ER…
Study on Gibbs-ERM learning, focusing on excess risk bounds and effective dimension.
problem Understanding the interplay between data distribution and learning in large hypothesis spaces.
method Distribution-dependent analysis of Gibbs-ERM, focusing on excess risk and effective dimension.
result Distribution-dependent upper bounds on excess risk, showing effective dimension controls risk.
Simple DP algorithms find approximate solutions for nonconvex ERM.
problem Finding approximate solutions to nonconvex ERM problems with privacy.
method Differential privacy, descent directions, line search, mini-batching, two-phase strategy.
result Effective algorithms for nonconvex ERM with privacy guarantees.
Study on Privileged ERM showing limitations and providing capacity analysis.
problem Improving classification accuracy with privileged information.
method Theoretical analysis of Privileged ERM using VC dimension and generalization bounds.
result Worst-case guarantees for Privileged ERM cannot improve over standard ERM unless privileged information capacity is similar or smaller.
The study analyzes multi-class teacher-student perceptron performance and generalization errors.
problem Analyzing multi-class classification with the teacher-student perceptron.
method Deriving asymptotic expressions for Bayes-optimal and empirical risk minimization (ERM) generalization errors.
result Regularised cross-entropy minimization yields close-to-optimal accuracy for multi-class classification.
Paper shows ERM's suboptimality due to bias, not variance.
problem Understanding why ERM fails to achieve optimal rates.
method Probabilistic and admissibility proofs for ERM in various settings.
result ERM's suboptimality is due to bias, not variance.
In many estimation problems, e.g. linear and logistic regression, we wish to minimize an unknown objective given only unbiased samples of the objective function. Furthermore, we aim to achieve this using as few samples as possible. In the absence of computational constraints, the minimizer of a sample average of observ…
Empirical risk minimization (ERM) is ubiquitous in machine learning and underlies most supervised learning methods. While there has been a large body of work on algorithms for various ERM problems, the exact computational complexity of ERM is still not understood. We address this issue for multiple popular ERM problems…
New algorithm for privacy-preserving nonconvex optimization.
problem Privacy-preserving nonconvex empirical risk minimization.
method Differentially private stochastic gradient descent algorithm.
result Achieves strong privacy guarantees efficiently with improved utility.
Paper proposes a new method for SP with covariates using PADR and ERM.
problem Stochastic programming with covariate information.
method Empirical risk minimization (ERM) with nonconvex piecewise affine decision rules (PADR).
result The method provides theoretical consistency and computational tractability for nonconvex SP problems.
In stochastic convex optimization the goal is to minimize a convex function F(x)≐Ef∼D[f(x)] over a convex set K⊂Rd where D is some unknown distribution and each f(⋅) in the support of D is convex over K. The optimization is commonl…
Study on tradeoffs between mistakes and ERM oracle calls in online and transductive learning.
problem Analyzing online and transductive learning with limited ERM and weak consistency oracle access.
method Proves lower bounds and upper bounds on mistakes and oracle calls, considering realizable and agnostic cases.
result Achieves optimal mistake bounds with weak consistency queries for certain concept classes.
This paper studies universal rates of ERM for binary classification under agnostic learning.
problem The challenge of achieving universal rates of ERM for binary classification under agnostic learning.
method The paper explores the agnostic universal rates of ERM for binary classification, revealing three possible rates: e−n, o(n−1/2), or arbitrarily slow. result The paper provides a complete characterization of which concept classes fall into each of the three categories of agnostic universal rates.
Default-ERM shortcut learning persists even without additional information.
problem Default-ERM shortcut learning in perception tasks despite stable feature sufficiency.
method Studied linear perception task; developed margin control (MARG-CTRL) loss functions.
result Margin control mitigates shortcut learning on various tasks.
Sharp asymptotics reveal how network width controls learnability in quadratic neural networks.
problem Understanding learnability in overparameterized quadratic neural networks.
method Mapping ERM to convex matrix sensing with nuclear norm penalization.
result Characterization of global minima and precise generalization thresholds.
This work characterizes optimal multiclass learning with regularization.
problem The empirical risk minimization (ERM) algorithm fails in multiclass learning settings.
method Using one-inclusion graphs (OIGs), the work introduces optimal learning algorithms that relax structural risk minimization and incorporate unsupervised learning.
result An optimal learner is introduced that uses a local regularization function and an unsupervised learning stage to learn the regularizer.
Develops a new learning framework for dynamic data.
problem Poor performance of existing strategies in dynamic data and goals.
method Prospective Learning framework and Prospective ERM algorithm.
result Prospective ERM converges to Bayes risk under certain assumptions.
ERM struggles with synthetic data, but some algorithms can still learn correctly.
problem Learning from a mix of natural and synthetic data.
method Modeling the scenario as a sequence of learning tasks with oblivious algorithms, studying ERM and its limitations.
result ERM converges to the true mean but is outperformed by weighted algorithms. ERM does not always converge in the PAC setting, but there are algorithms capable of learning the correct hypothesis.
New PAC-Bayes training method improves model generalization for unbounded loss.
problem Improving generalization of complex models under unbounded loss.
method Established new PAC-Bayes bound for unbounded loss, jointly training prior and posterior.
result Outperforms existing PAC-Bayes training algorithms and matches ERM accuracy.
Simpler majority vote of three classifiers achieves optimal error bounds.
problem Developing an optimal PAC learning algorithm in the realizable setting.
method Returning the majority vote of three ERM classifiers.
result Achieves optimal in-expectation bound on error.
Study tackles contamination and heterogeneity in multi-task learning, improving robustness and personalization.
problem Challenges in integrating related tasks due to contamination and heterogeneity.
method Proposes a filtering-based robust multi-task gradient descent method to estimate global and clean task-specific minimizers.
result Demonstrates improved robustness and personalization compared to existing methods.
Paper explores universal rates of ERM in machine learning.
problem Understanding universal learning rates for ERM.
method Analyzes realizable concept classes and ERM principles.
result Four possible universal learning rates by ERM.
Algorithm solves online binary classification and infinite games using ERM oracle.
problem Online learning and solving infinite games with computationally inefficient oracles.
method Proposes an algorithm relying solely on ERM oracle calls for online binary classification and nonparametric games.
result Achieves finite and sublinearly growing regret in various settings.