arXiv research

Gradient descent with early stopping achieves optimal sparse recovery.

Early stopping improves logistic regression's calibration and consistency in high dimensions.

Analysis of cross-validation for early-stopped gradient descent in high-dimensional regression.

MGD with early stopping tends to ridge regularization in least squares regression.

This work bounds the run-time of nonconvex optimization with early stopping.

Early stopping improves neural networks' performance on binary classification tasks.

Adaptive rule improves kernel-based gradient descent performance.

Efficiently performs robust and sparse kernel regression.

Study robustness of early-stopping GD for linear regression attacks.

Paper studies early-stopped mirror descent for noisy sparse phase retrieval.

The paper analyzes early stopping in linear regression and shows it's equivalent to ridge regularization.

Gradient descent recovers principal components of overparametrized asymmetric matrices without explicit regularization.

New method estimates variable importance for large models efficiently.

Study mirrors descent's early stopping for linear and kernel models, improving risk guarantees.

Paper proposes a method for early stopping in regression using reproducing kernels.

Gradient descent trains neural networks to match kernel regression's sharp generalization rate.

Modern neural networks are typically trained in an over-parameterized regime where the parameters of the model far exceed the size of the training data. Such neural networks in principle have the capacity to (over)fit any set of labels including pure noise. Despite this, somewhat paradoxically, neural network models tr…

GD-trained shallow ReLU nets learn Lipschitz functions with noise.

Early stopping of iterative algorithms is an algorithmic regularization method to avoid over-fitting in estimation and classification. In this paper, we show that early stopping can also be applied to obtain the minimax optimal testing in a general non-parametric setup. Specifically, a Wald-type test statistic is obtai…

GD with early stopping trains shallow neural nets for nonparametric regression robustly.

Random matrix theory explains transient signal detectability in early-stopped gradient flow.

Stochastic Gradient Descent shows directional bias with moderate learning rates, impacting optimization outcomes.

Improves early stopping in deep networks by adjusting stepsizes.

The paper analyzes stability and generalization of shallow neural networks using gradient methods.

Neural networks trained with PGD achieve sharp regression rates in interpolation spaces.

The study analyzes how many neurons are needed for two-layer neural networks trained with gradient descent.

The strategy of early stopping is a regularization technique based on choosing a stopping time for an iterative algorithm. Focusing on non-parametric regression in a reproducing kernel Hilbert space, we analyze the early stopping strategy for a form of gradient-descent applied to the least-squares loss function. We pro…

Neural networks learn clean data patterns first, then noisy data, leading to improved performance initially but deteriorating later.

The paper speeds up hyperparameter optimisation in Gaussian processes.

We show that unconverged stochastic gradient descent can be interpreted as a procedure that samples from a nonparametric variational approximate posterior distribution. This distribution is implicitly defined as the transformation of an initial distribution by a sequence of optimization updates. By tracking the change …

Gradient matching method estimates implicit regularization in complex deep learning systems.

Sharp risk bounds for early-stopping in Gaussian linear regression are derived.

Theoretical analysis of t-SNE for visualizing clustered data.

Introduces self-regularization for analyzing learning algorithms.

We study the statistical properties of the iterates generated by gradient descent, applied to the fundamental problem of least squares regression. We take a continuous-time view, i.e., consider infinitesimal step sizes in gradient descent, in which case the iterates form a trajectory called gradient flow. Our primary f…

Analyzes the generalization and training errors of the random feature model over time.

Many statistical estimators for high-dimensional linear regression are M-estimators, formed through minimizing a data-dependent square loss function plus a regularizer. This work considers a new class of estimators implicitly defined through a discretized gradient dynamic system under overparameterization. We show that…

Gradient boosting improved with lassoed trees achieves faster convergence.

The paper shows how random ReLU networks converge to smooth splines.

Untrained neural networks can recover natural images from few measurements.

Neural operators achieve fast convergence rates for solving PDEs.

Convolutional Neural Networks (CNNs) have emerged as highly successful tools for image generation, recovery, and restoration. A major contributing factor to this success is that convolutional networks impose strong prior assumptions about natural images. A surprising experiment that highlights this architectural bias t…

Paper characterizes gradient descent dynamics for neural networks with finite width.

The behavior of the gradient descent (GD) algorithm is analyzed for a deep neural network model with skip-connections. It is proved that in the over-parametrized regime, for a suitable initialization, with high probability GD can find a global minimum exponentially fast. Generalization error estimates along the GD path…

Paper analyzes iterates in high-dimensional linear models and proposes estimators for their generalization error.

Early stopping is a widely used technique to prevent poor generalization performance when training an over-expressive model by means of gradient-based optimization. To find a good point to halt the optimizer, a common practice is to split the dataset into a training and a smaller validation set to obtain an ongoing est…

Boosting method for causal SEMs from observational data.

ScoreStop uses gradient tests to stop gradient boosting early.