A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
Over-parametrization speeds up learning a single neuron model.
The paper shows how gradient flow on over-parametrized tensor decomposition behaves like deflation.
SMD outperforms SGD in over-parametrized linear models for certain data distributions.
Large over-parametrized models learned via stochastic gradient descent (SGD) methods have become a key element in modern machine learning. Although SGD methods are very effective in practice, most theoretical analyses of SGD suggest slower convergence than what is empirically observed. In our recent work [8] we analyze…
The paper shows over-confidence in models isn't just due to over-parametrization.
We improve the over-parametrization size over two beautiful results [Li and Liang' 2018] and [Du, Zhai, Poczos and Singh' 2019] in deep learning theory.
Paper shows faster convergence to local-minimizers in over-parametrized models under interpolation-like conditions.
We study the generalization properties of minimum-norm solutions for three over-parametrized machine learning models including the random feature model, the two-layer neural network model and the residual network model. We proved that for all three models, the generalization error for the minimum-norm solution is compa…
New function class characterizes loss landscape of deep neural networks without over-parametrization.
Dr. of Crosswise proposes a new architecture to reduce over-parametrization in Neural Networks. It introduces an operand for rapid computation in the framework of Deep Learning that leverages learned weights. The formalism is described in detail providing both an accurate elucidation of the mechanics and the theoretica…
Despite existing work on ensuring generalization of neural networks in terms of scale sensitive complexity measures, such as norms, margin and sharpness, these complexity measures do not offer an explanation of why neural networks generalize better with over-parametrization. In this work we suggest a novel complexity m…
A recent line of work has shown that an overparametrized neural network can perfectly fit the training data, an otherwise often intractable nonconvex optimization problem. For (fully-connected) shallow networks, in the best case scenario, the existing theory requires quadratic over-parametrization as a function of the …
We argue that in fully-connected networks a phase transition delimits the over- and under-parametrized regimes where fitting can or cannot be achieved. Under some general conditions, we show that this transition is sharp for the hinge loss. In the whole over-parametrized regime, poor minima of the loss are not encounte…
We provide new theoretical insights on why over-parametrization is effective in learning neural networks. For a hidden node shallow network with quadratic activation and training data points, we show as long as , over-parametrization enables local search algorithms to find a \emph{globally} op…
Study shows neural network parameters converge to ridgelet spectrum.
Improved complexity for machine learning optimization methods.
Gradient descent learns over-param neural nets better than NTK.
New method reduces over-parametrization in neural networks, ensuring sparsity and finite network size.
Convolutional neural networks converge quickly with gradient descent.
Noether's theorem clarifies how symmetries in neural networks influence learning.
ADA augments data using AR replicas for robust regression.
Interpolation improves performance in nearest neighbor algorithms without over-parametrization.
Improved DNN calibration without sacrificing accuracy.
Estimates how much samples inform neural network training and function.
Adaptive gradient methods like AdaGrad are widely used in optimizing neural networks. Yet, existing convergence guarantees for adaptive gradient methods require either convexity or smoothness, and, in the smooth setting, only guarantee convergence to a stationary point. We propose an adaptive gradient method and show t…
New study shows how model complexity affects test risk, challenging classical theory.
Recent theoretical work has established connections between over-parametrized neural networks and linearized models governed by he Neural Tangent Kernels (NTKs). NTK theory leads to concrete convergence and generalization results, yet the empirical performance of neural networks are observed to exceed their linearized …
We analyze ridge interpolators in correlated factor regression models using RDT.
Gradient descent can find better tensor decompositions than lazy training in over-parameterized settings.
Deep neural networks without regularization can achieve consistent estimates with good convergence rates.
Study linear transformations' effects on data augmentation for improved estimation.
Gradient descent trains shallow neural networks to approximate functions in 1D.
New law establishes robustness for neural networks with bounded weights.
Study on neural networks with quadratic activation functions, focusing on optimization and generalization.
Traditionally in regression one minimizes the number of fitting parameters or uses smoothing/regularization to trade training (TE) and generalization error (GE). Driving TE to zero by increasing fitting degrees of freedom (dof) is expected to increase GE. However modern big-data approaches, including deep nets, seem to…
Algorithmic approaches endow deep learning systems with implicit bias that helps them generalize even in over-parametrized settings. In this paper, we focus on understanding such a bias induced in learning through dropout, a popular technique to avoid overfitting in deep learning. For single hidden-layer linear neural …
To better understand the interplay of censoring and sparsity we develop finite sample properties of nonparametric Cox proportional hazard's model. Due to high impact of sequencing data, carrying genetic information of each individual, we work with over-parametrized problem and propose general class of group penalties s…
Gradient descent in neural networks analyzed using RKBS for broader applicability.
Deep neural networks are often trained in the over-parametrized regime (i.e. with far more parameters than training examples), and understanding why the training converges to solutions that generalize remains an open problem. Several studies have highlighted the fact that the training procedure, i.e. mini-batch Stochas…
Lazy neural networks are vulnerable to adversarial attacks.
This paper studies how to capture dependency graph structures from real data which may not be Gaussian. Starting from marginal loss functions not necessarily derived from probability distributions, we utilize an additive over-parametrization with shrinkage to incorporate variable dependencies into the criterion. An ite…
When optimizing over-parameterized models, such as deep neural networks, a large set of parameters can achieve zero training error. In such cases, the choice of the optimization algorithm and its respective hyper-parameters introduces biases that will lead to convergence to specific minimizers of the objective. Consequ…
This paper extends the convergence rate of DEQs with ReLU to any general activation.
Deep neural networks learn spatially heterogeneous patterns from input data.
A fairly comprehensive analysis is presented for the gradient descent dynamics for training two-layer neural network models in the situation when the parameters in both layers are updated. General initialization schemes as well as general regimes for the network width and training data size are considered. In the over-…
New estimates for the population risk are established for two-layer neural networks. These estimates are nearly optimal in the sense that the error rates scale in the same way as the Monte Carlo error rates. They are equally effective in the over-parametrized regime when the network size is much larger than the size of…
The behavior of the gradient descent (GD) algorithm is analyzed for a deep neural network model with skip-connections. It is proved that in the over-parametrized regime, for a suitable initialization, with high probability GD can find a global minimum exponentially fast. Generalization error estimates along the GD path…
In this paper we aim to formally explain the phenomenon of fast convergence of SGD observed in modern machine learning. The key observation is that most modern learning architectures are over-parametrized and are trained to interpolate the data by driving the empirical loss (classification and regression) close to zero…