arXiv research

Unified formula for training dynamics of linear networks combining lazy and balanced regimes.

Gradient descent can find better tensor decompositions than lazy training in over-parameterized settings.

We discuss the approximation of the value function for infinite-horizon discounted Markov Reward Processes (MRP) with nonlinear functions trained with the Temporal-Difference (TD) learning algorithm. We first consider this problem under a certain scaling of the approximating function, leading to a regime called lazy tr…

Study shows optimal model performance at critical level of feature learning.

New private algorithms for online learning improve regret in high privacy regimes.

Two distinct limits for deep learning have been derived as the network width $h\rightarrow \infty$, depending on how the weights of the last layer scale with $h$. In the Neural Tangent Kernel (NTK) limit, the dynamics becomes linear in the weights and is described by a frozen kernel $Θ$. By contrast, in the Mean-Field …

Study on adversarial robustness in neural networks across initialization and training phases.

Deep networks prioritize easier examples over harder ones, leading to faster training.

Study on rich regime training in deep learning, finding active parameters in bottom layers.

Analyzes neural networks using linear models to understand their behavior.

Exact solutions reveal how unbalanced initializations promote rapid feature learning in neural networks.

Quantum machine learning faces 'laziness' and 'barren plateaus', but noise can mitigate the latter.

We explore how neural networks train to zero loss, focusing on initial scale.

Grokking occurs when neural networks transition from lazy to rich training dynamics, fitting initial features before generalizing.

Use simplified layerwise linear models to understand neural dynamics.

Clapping reduces memory usage in distributed optimization by reusing data samples.

Transformers capture combinatorial tasks with bounded error and logarithmic sample dependence.

Adaptive kernels from neural networks improve model performance.

Analysis of deep neural networks under various learning rules reveals dynamics of feature and prediction learning.

Neural networks can learn kernel machines with a data-dependent kernel.

Deep networks generalize well even when they fit training data perfectly, thanks to overparametrization.

Theory captures feature learning effects in finite CNNs.

Neural networks learn task-specific features, influenced by nonlinearity.

A recent line of work studies overparametrized neural networks in the "kernel regime," i.e. when the network behaves during training as a kernelized linear predictor, and thus training with gradient descent has the effect of finding the minimum RKHS norm solution. This stands in contrast to other studies which demonstr…

Neural networks compress uninformative input directions, improving test error.

Mirror flow in shallow neural networks shows similar implicit bias to gradient flow, with key differences in curvature penalties.

Gradient descent in tensor factorization favors low-rank solutions.

Study on how initialization scale affects neural network training regimes.

Counterfactual regret minimization (CFR) is the most popular algorithm on solving two-player zero-sum extensive games with imperfect information and achieves state-of-the-art performance in practice. However, the performance of CFR is not fully understood, since empirical results on the regret are much better than the …

This work challenges the Neural Tangent Kernel's role in overparameterized neural networks, especially with large width and depth.

This work shows linear convergence for two-layer neural networks in mean-field regime.

We consider online learning problems where the aim is to achieve regret which is efficient in the sense that it is the same order as the lowest regret amongst K experts. This is a substantially stronger requirement that achieving $O(\sqrt{n})$ or $O(\log n)$ regret with respect to the best expert and standard algorithm…

We study the supervised learning problem under either of the following two models: (1) Feature vectors ${\boldsymbol x}_i$ are $d$-dimensional Gaussians and responses are $y_i = f_*({\boldsymbol x}_i)$ for $f_*$ an unknown quadratic function; (2) Feature vectors ${\boldsymbol x}_i$ are distributed as a mixture of two $…

Stochastic principal component analysis (SPCA) has become a popular dimensionality reduction strategy for large, high-dimensional datasets. We derive a simplified algorithm, called Lazy SPCA, which has reduced computational complexity and is better suited for large-scale distributed computation. We prove that SPCA and …

Lazy neural networks are vulnerable to adversarial attacks.

Develops a new framework to analyze gradient flow regimes and derive explicit solutions.

A new method simulates a lazy version of a Markov chain for empirical inference.

Study introduces a new investment strategy model using lazy factor and probability weights.

Lazy, perfectly informed investors trade infrequently due to costs.

We prove the precise scaling, at finite depth and width, for the mean and variance of the neural tangent kernel (NTK) in a randomly initialized ReLU network. The standard deviation is exponential in the ratio of network depth to width. Thus, even in the limit of infinite overparameterization, the NTK is not determinist…

Study on generalisation in random feature learning and hidden manifold models.

Deep learning explained through spectral filtering of hierarchical features.

This paper improves adversarial robustness of deep learning models.

The paper explains the richness scale of wide neural networks.

A recent line of work has shown that an overparametrized neural network can perfectly fit the training data, an otherwise often intractable nonconvex optimization problem. For (fully-connected) shallow networks, in the best case scenario, the existing theory requires quadratic over-parametrization as a function of the …

Lazy Gradient Descent outperforms existing polytope algorithms in pseudo-regret.

In this work we introduce a conditional accelerated lazy stochastic gradient descent algorithm with optimal number of calls to a stochastic first-order oracle and convergence rate $O\left(\frac{1}{\varepsilon^2}\right)$ improving over the projection-free, Online Frank-Wolfe based stochastic gradient descent of Hazan an…

Improved generative models using overparametrized shallow neural networks.