Neural networks provide a rich class of high-dimensional, non-convex optimization problems. Despite their non-convexity, gradient-descent methods often successfully optimize these models. This has motivated a recent spur in research attempting to characterize properties of their loss surface that may explain such succe…
On-device research index
arXiv research
A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
168,657 papers · 148 categories
Trend · papers per month
12 results for “overparametrised”
Study on how noise and variation-norm regularisation help shallow ReLU networks use fewer neurons.
problem Understanding how shallow ReLU networks use a finite number of neurons in the infinitely wide limit.
method Analysis of two regularisation strategies: noise injection and variation-norm.
result Both regularisation methods minimize functions with a finite number of neurons, regardless of overparametrisation.
This study uses continuous-time analysis to understand how momentum affects the optimisation of diagonal linear networks.
problem The effect of momentum on the optimisation trajectory of gradient descent.
method Leveraging a continuous-time approach to analyze momentum gradient descent with step size γ and momentum parameter β.
result Small values of λ help recover sparse solutions in overparametrised regression settings.
GD converges in unstable regimes, even with oscillatory behavior.
problem Understanding convergence of GD in unstable regimes.
method Analysis of two-step gradient updates.
result Characterization of local conditions for convergence.
Bayesian posterior contraction rates improve with decreasing tails
problem Bayesian posterior contraction in nonparametric settings
method Using -exponential tails for contraction rates
result Improvement in contraction rates with decreasing tails
We provide bounds on control learning error in stochastic systems.
problem Learning optimal controls in stochastic environments with uncontrolled parts.
method Dynamic programming and mean-field interpretation of neural networks.
result Non-asymptotic bounds on generalization error for stable overparametrised settings.
Label noise (stochastic) gradient descent implicitly solves the Lasso for quadratic parametrisationstat.ML
The paper shows how label noise in training can lead to solutions that solve a Lasso program.
problem Understanding the implicit bias of training algorithms in overparametrised models.
method Analyzing the continuous time version of the training dynamics of a quadratically parametrised model.
result The stochastic flow implicitly solves a Lasso program, providing convergence guarantees and support recovery conditions.
Gaussian processes (GPs) are Bayesian nonparametric generative models that provide interpretability of hyperparameters, admit closed-form expressions for training and inference, and are able to accurately represent uncertainty. To model general non-Gaussian data with complex correlation structure, GPs can be paired wit…
(S)GD over Diagonal Linear Networks: Implicit Regularisation, Large Stepsizes and Edge of Stabilitycs.LG
Study on GD and SGD over diagonal networks, focusing on stepsizes and regularisation.
problem Understanding the impact of stochasticity and large stepsizes on gradient descent and SGD solutions.
method Investigation of GD and SGD over diagonal linear networks with macroscopic stepsizes, proving convergence and characterizing solutions.
result Large stepsizes consistently benefit SGD for sparse regression problems, but can hinder GD recovery of sparse solutions, especially in the edge of stability regime.
Develops Bayesian filtering for online learning and related problems.
problem Sequential machine learning challenges, especially non-stationarity, model misspecification, and high dimensionality.
method Modular adaptive framework, provably robust filter, and sequential parameter updates.
result Improved performance in dynamic, high-dimensional, and misspecified models.
The paper analyzes fluctuations in ensemble models in high-dimensional settings.
problem Understanding statistical fluctuations in ensemble models in high-dimensional settings.
method Develops a rigorous theory for the study of fluctuations in ensemble of generalised linear models.
result Provides a complete description of the asymptotic joint distribution of the empirical risk minimizer for convex losses in high-dimensional settings.
A study on a surprising phase transition in model generalization error as parameters approach sample size.
problem Understanding the generalization error of overparametrized ridge models.
method Finite sample analysis using continuous Newton method and -norm solution performance.
result The generalization error decreases after the threshold for ridge models.