arXiv research

Gradient descent with early stopping achieves optimal sparse recovery.

Early stopping improves logistic regression's calibration and consistency in high dimensions.

Analysis of cross-validation for early-stopped gradient descent in high-dimensional regression.

MGD with early stopping tends to ridge regularization in least squares regression.

Early stopping improves neural networks' performance on binary classification tasks.

Efficiently performs robust and sparse kernel regression.

Study robustness of early-stopping GD for linear regression attacks.

Paper studies early-stopped mirror descent for noisy sparse phase retrieval.

The paper analyzes early stopping in linear regression and shows it's equivalent to ridge regularization.

Gradient descent recovers principal components of overparametrized asymmetric matrices without explicit regularization.

This work examines the convergence of stochastic gradient-based optimization algorithms that use early stopping based on a validation function. The form of early stopping we consider is that optimization terminates when the norm of the gradient of a validation function falls below a threshold. We derive conditions that…

New method estimates variable importance for large models efficiently.

Paper proposes a method for early stopping in regression using reproducing kernels.

In this paper, we propose an adaptive stopping rule for kernel-based gradient descent (KGD) algorithms. We introduce the empirical effective dimension to quantify the increments of iterations in KGD and derive an implementable early stopping strategy. We analyze the performance of the adaptive stopping rule in the fram…

Gradient descent trains neural networks to match kernel regression's sharp generalization rate.

Modern neural networks are typically trained in an over-parameterized regime where the parameters of the model far exceed the size of the training data. Such neural networks in principle have the capacity to (over)fit any set of labels including pure noise. Despite this, somewhat paradoxically, neural network models tr…

GD-trained shallow ReLU nets learn Lipschitz functions with noise.

Early stopping of iterative algorithms is an algorithmic regularization method to avoid over-fitting in estimation and classification. In this paper, we show that early stopping can also be applied to obtain the minimax optimal testing in a general non-parametric setup. Specifically, a Wald-type test statistic is obtai…

GD with early stopping trains shallow neural nets for nonparametric regression robustly.

Random matrix theory explains transient signal detectability in early-stopped gradient flow.

Improves early stopping in deep networks by adjusting stepsizes.

Stochastic Gradient Descent shows directional bias with moderate learning rates, impacting optimization outcomes.

Recently there has been a surge of interest in understanding implicit regularization properties of iterative gradient-based optimization algorithms. In this paper, we study the statistical guarantees on the excess risk achieved by early-stopped unconstrained mirror descent algorithms applied to the unregularized empiri…

The paper analyzes stability and generalization of shallow neural networks using gradient methods.

Neural networks trained with PGD achieve sharp regression rates in interpolation spaces.

The study analyzes how many neurons are needed for two-layer neural networks trained with gradient descent.

The strategy of early stopping is a regularization technique based on choosing a stopping time for an iterative algorithm. Focusing on non-parametric regression in a reproducing kernel Hilbert space, we analyze the early stopping strategy for a form of gradient-descent applied to the least-squares loss function. We pro…

Neural networks learn clean data patterns first, then noisy data, leading to improved performance initially but deteriorating later.

The paper speeds up hyperparameter optimisation in Gaussian processes.

We show that unconverged stochastic gradient descent can be interpreted as a procedure that samples from a nonparametric variational approximate posterior distribution. This distribution is implicitly defined as the transformation of an initial distribution by a sequence of optimization updates. By tracking the change …

Gradient matching method estimates implicit regularization in complex deep learning systems.

Sharp risk bounds for early-stopping in Gaussian linear regression are derived.

Theoretical analysis of t-SNE for visualizing clustered data.

Introduces self-regularization for analyzing learning algorithms.

We study the statistical properties of the iterates generated by gradient descent, applied to the fundamental problem of least squares regression. We take a continuous-time view, i.e., consider infinitesimal step sizes in gradient descent, in which case the iterates form a trajectory called gradient flow. Our primary f…

Analyzes the generalization and training errors of the random feature model over time.

Many statistical estimators for high-dimensional linear regression are M-estimators, formed through minimizing a data-dependent square loss function plus a regularizer. This work considers a new class of estimators implicitly defined through a discretized gradient dynamic system under overparameterization. We show that…

Gradient boosting improved with lassoed trees achieves faster convergence.

Untrained neural networks can recover natural images from few measurements.

Neural operators achieve fast convergence rates for solving PDEs.

Convolutional Neural Networks (CNNs) have emerged as highly successful tools for image generation, recovery, and restoration. A major contributing factor to this success is that convolutional networks impose strong prior assumptions about natural images. A surprising experiment that highlights this architectural bias t…

Paper characterizes gradient descent dynamics for neural networks with finite width.

The behavior of the gradient descent (GD) algorithm is analyzed for a deep neural network model with skip-connections. It is proved that in the over-parametrized regime, for a suitable initialization, with high probability GD can find a global minimum exponentially fast. Generalization error estimates along the GD path…

Paper analyzes iterates in high-dimensional linear models and proposes estimators for their generalization error.

Early stopping is a widely used technique to prevent poor generalization performance when training an over-expressive model by means of gradient-based optimization. To find a good point to halt the optimizer, a common practice is to split the dataset into a training and a smaller validation set to obtain an ongoing est…

Boosting method for causal SEMs from observational data.

ScoreStop uses gradient tests to stop gradient boosting early.

Study on discrepancy principle for learning algorithms in nonparametric regression.