arXiv research

Study risk bounds for distributed ERM with general loss functions and hypothesis spaces.

Distributed machine learning is an approach allowing different parties to learn a model over all data sets without disclosing their own data. In this paper, we propose a weighted distributed differential privacy (WD-DP) empirical risk minimization (ERM) method to train a model in distributed setting, considering differ…

Default-ERM shortcut learning persists even without additional information.

A new distributed optimization method for ERM problems.

This paper studies universal rates of ERM for binary classification under agnostic learning.

Develops a new learning framework for dynamic data.

Recent work shows GRW approaches do not improve over ERM in distributional shift.

Unified view connects CoCoA and ADMM for distributed ERM.

New framework shows ERM is optimal for both interpolation and extrapolation in domain generalization.

FeAT improves OOD generalization by learning richer features.

ERM struggles with synthetic data, but some algorithms can still learn correctly.

PAIR optimizes machine learning models to generalize better to out-of-distribution data.

We analyze Gibbs-based transfer learning algorithms using information theory.

As opposed to standard empirical risk minimization (ERM), distributionally robust optimization aims to minimize the worst-case risk over a larger ambiguity set containing the original empirical distribution of the training data. In this work, we describe a minimax framework for statistical learning with ambiguity sets …

NAPP-ERM improves ERM with differential privacy guarantees by iteratively achieving target regularization and delivering strong convexity.

ERM with square loss achieves sublinear error for learnable function classes with smoothed data.

This work analyzes IRM and ERM from sample complexity perspective, revealing different behaviors under various distribution shifts.

Study tackles criterion collapse in learning criteria, showing conditions for loss minimization.

Entropy asymmetry affects regularization in ERM, leading to biased solutions.

Simple DP algorithms find approximate solutions for nonconvex ERM.

Study on Privileged ERM showing limitations and providing capacity analysis.

New dual formulation reduces generalization error for ERM-fDR.

We propose a communication- and computation-efficient distributed optimization algorithm using second-order information for solving ERM problems with a nonsmooth regularization term. Current second-order and quasi-Newton methods for this problem either do not work well in the distributed setting or work only for specif…

Paper shows ERM's suboptimality due to bias, not variance.

Extends ERM with exponential tilting for improved machine learning performance.

Study improves ERM for heavy-tailed data with dependent inputs.

While many solutions for privacy-preserving convex empirical risk minimization (ERM) have been developed, privacy-preserving nonconvex ERM remains a challenge. We study nonconvex ERM, which takes the form of minimizing a finite-sum of nonconvex loss functions over a training set. We propose a new differentially private…

Gibbs-ERM learning is a natural idealized model of learning with stochastic optimization algorithms (such as Stochastic Gradient Langevin Dynamics and ---to some extent--- Stochastic Gradient Descent), while it also arises in other contexts, including PAC-Bayesian theory, and sampling mechanisms. In this work we study …

Empirical risk minimization (ERM) is ubiquitous in machine learning and underlies most supervised learning methods. While there has been a large body of work on algorithms for various ERM problems, the exact computational complexity of ERM is still not understood. We address this issue for multiple popular ERM problems…

Paper proposes a new method for SP with covariates using PADR and ERM.

Study on tradeoffs between mistakes and ERM oracle calls in online and transductive learning.

The paper analyzes how machine learning models perform under covariate shift, especially when the feature shift in $x$ is larger than that in $y$.

Machine learning models (e.g., speech recognizers) are usually trained to minimize average loss, which results in representation disparity---minority groups (e.g., non-native speakers) contribute less to the training objective and thus tend to suffer higher loss. Worse, as model accuracy affects user retention, a minor…

Dual optimization connects ERM-fDR to normalization function.

Paper presents ERM with $f$-divergence regularization and its properties.

Empirical risk minimization (ERM) is a fundamental learning rule for statistical learning problems where the data is generated according to some unknown distribution $\mathsf{P}$ and returns a hypothesis $f$ chosen from a fixed class $\mathcal{F}$ with small loss $\ell$. In the parametric setting, depending upon $(\ell…

Multiple fairness constraints have been proposed in the literature, motivated by a range of concerns about how demographic groups might be treated unfairly by machine learning classifiers. In this work we consider a different motivation; learning from biased training data. We posit several ways in which training data m…

Simpler majority vote of three classifiers achieves optimal error bounds.

This thesis tackles causality in machine learning, improving OOD generalization and robustness.

Paper explores universal rates of ERM in machine learning.

New method for collaborative learning with multiple distributions, minimizing sample size.

Algorithm solves online binary classification and infinite games using ERM oracle.

Upper bounds and lower bounds show ERM outperforms DG methods in various settings.

Self-adaptive training improves deep learning robustness.

The paper proposes a new method to measure risk with fine-grained tail sensitivity.

The study uncovers the breakdown of Gaussian universality in high-dimensional empirical risk minimization.

Sharp bounds on ERM's minimal error in regression.

In stochastic convex optimization the goal is to minimize a convex function $F(x) \doteq {\mathbf E}_{{\mathbf f}\sim D}[{\mathbf f}(x)]$ over a convex set $\cal K \subset {\mathbb R}^d$ where $D$ is some unknown distribution and each $f(\cdot)$ in the support of $D$ is convex over $\cal K$. The optimization is commonl…