arXiv research

Develops local population-risk certificates for model updates

Develops certificates for local population-risk increments using cross-fitted ridge calibration.

Population risk is always of primary interest in machine learning; however, learning algorithms only have access to the empirical risk. Even for applications with nonconvex nonsmooth losses (such as modern deep networks), the population risk is generally significantly more well-behaved from an optimization point of vie…

We study the Stochastic Gradient Langevin Dynamics (SGLD) algorithm for non-convex optimization. The algorithm performs stochastic gradient descent, where in each step it injects appropriately scaled Gaussian noise to the update. We analyze the algorithm's hitting time to an arbitrary subset of the parameter space. Two…

The landscape of empirical risk has been widely studied in a series of machine learning problems, including low-rank matrix factorization, matrix sensing, matrix completion, and phase retrieval. In this work, we focus on the situation where the corresponding population risk is a degenerate non-convex loss function, nam…

The curse of dimensionality affects neural network optimization, especially with smooth functions.

This work aims to provide understandings on the remarkable success of deep convolutional neural networks (CNNs) by theoretically analyzing their generalization performance and establishing optimization guarantees for gradient descent based training algorithms. Specifically, for a CNN model consisting of $l$ convolution…

We show that model compression can improve the population risk of a pre-trained model, by studying the tradeoff between the decrease in the generalization error and the increase in the empirical risk with model compression. We first prove that model compression reduces an information-theoretic bound on the generalizati…

The classical asymptotic theory for parametric $M$-estimators guarantees that, in the limit of infinite sample size, the excess risk has a chi-square type distribution, even in the misspecified case. We demonstrate how self-concordance of the loss allows to characterize the critical sample size sufficient to guarantee …

New bounds for KANs trained with DP-SGD, addressing correlated noise.

The paper analyzes the maximum margin algorithm's performance on noisy data.

Paper improves privacy-preserving optimization rates for convex functions.

Gradient descent learns a single neuron without knowing the relationship between inputs and labels.

New algorithm corrects bias in LDP-released data for better analysis.

New algorithms for differentially private optimization in convex and non-convex settings with near-optimal rates.

New estimates for the population risk are established for two-layer neural networks. These estimates are nearly optimal in the sense that the error rates scale in the same way as the Monte Carlo error rates. They are equally effective in the over-parametrized regime when the network size is much larger than the size of…

In stochastic optimization, the population risk is generally approximated by the empirical risk. However, in the large-scale setting, minimization of the empirical risk may be computationally restrictive. In this paper, we design an efficient algorithm to approximate the population risk minimizer in generalized linear …

Gradient descent struggles with high-dimensional data fitting.

Novel graph theory for neural networks improves understanding of their structure and performance.

This paper studies the landscape of empirical risk of deep neural networks by theoretically analyzing its convergence behavior to the population risk as well as its stationary points and properties. For an $l$-layer linear neural network, we prove its empirical risk uniformly converges to its population risk at the rat…

Study shows more frequent communication in FL reduces model's generalization power.

The paper addresses differentially private learning for neural networks, focusing on risk bounds and algorithm feasibility.

Paper addresses DP-SCO on heavy-tailed data, providing methods and results.

Optimal a priori estimates are derived for the population risk, also known as the generalization error, of a regularized residual network model. An important part of the regularized model is the usage of a new path norm, called the weighted path norm, as the regularization term. The weighted path norm treats the skip c…

This work compares regularization and constrained inference for label constraints in machine learning.

Gradient descent methods for deep ReLU networks achieve optimal generalization rates.

New study shows deep networks generalize well due to loss surface geometry.

Stochastic Gradient Langevin Dynamics (SGLD) is a popular variant of Stochastic Gradient Descent, where properly scaled isotropic Gaussian noise is added to an unbiased estimate of the gradient at each iteration. This modest change allows SGLD to escape local minima and suffices to guarantee asymptotic convergence to g…

Early stopping improves neural networks' performance on binary classification tasks.

New algorithm finds approximate stationary points faster under differential privacy constraints.

Study optimizes linear regression analysis for high-dimensional settings.

This work establishes uniform convergence of subdifferentials in stochastic optimization.

Paper revisits DP-SCO in Euclidean and $\ell_p^d$ spaces, focusing on constrained and bounded sets.

We provide non-asymptotic excess risk guarantees for statistical learning in a setting where the population risk with respect to which we evaluate the target parameter depends on an unknown nuisance parameter that must be estimated from data. We analyze a two-stage sample splitting meta-algorithm that takes as input ar…

Stacked regressions improve predictive accuracy by combining estimators.

Large stepsizes can accelerate gradient descent for logistic regression.

Stochastic Gradient Descent underperforms on some problems, contrary to expectations.

New stability analysis improves generalization of multipass SGD.

Improved algorithm finds second-order stationary points in non-convex optimization.

We consider the problem of learning a one-hidden-layer neural network: we assume the input $x\in \mathbb{R}^d$ is from Gaussian distribution and the label $y = a^\top σ(Bx) + ξ$, where $a$ is a nonnegative vector in $\mathbb{R}^m$ with $m\le d$, $B\in \mathbb{R}^{m\times d}$ is a full-rank weight matrix, and $ξ$ is a n…

EP-learning framework improves causal contrast estimation efficiency.

For classification problems with significant class imbalance, subsampling can reduce computational costs at the price of inflated variance in estimating model parameters. We propose a method for subsampling efficiently for logistic regression by adjusting the class balance locally in feature space via an accept-reject …

Improved analysis for extreme multi-class CRL with better sample complexity.

Develops asymptotic theory for deep Cox models to enable valid inference.

Gradient descent performs well on weakly convex losses, offering generalization guarantees.

The paper tackles machine unlearning by designing efficient algorithms for adaptive query classes.

Improved privacy bounds for learning linear predictors with convex losses.

Most high-dimensional estimation and prediction methods propose to minimize a cost function (empirical risk) that is written as a sum of losses associated to each data point. In this paper we focus on the case of non-convex losses, which is practically important but still poorly understood. Classical empirical process …