Deep ResNets can achieve zero loss with enough layers and weights.
problem Finding parameters in deep ResNets that fit training data perfectly.
method Mean-field analysis and gradient descent flow to a PDE.
result Gradient descent for ResNet training converges to a zero-loss solution in the large-NN limit.
Zero loss is achievable in overparametrized DL networks under specific conditions.
problem Achieving zero loss in overparametrized deep learning networks.
method Determine sufficient conditions for zero loss attainability and present an explicit construction of zero loss minimizers.
result Explicit minimizers for zero loss in overparametrized DL networks are constructed without gradient descent.
Gradient descent struggles to achieve zero loss in deep learning models due to non-generic data distributions.
problem Achieving zero loss minimizers in deep learning networks.
method Analysis of gradient descent algorithm in deep learning, focusing on underparametrized networks.
result Zero loss minimization cannot be achieved generically in deep learning networks.
One-pass SGD dynamics in overparameterized quadratic networks show slow escape from poor solutions.
problem Slow escape from poor generalization solutions in overparameterized neural networks.
method Analysis of one-pass SGD dynamics using ordinary differential equations for overlap matrices.
result Overparameterization only modestly accelerates escape from poor solutions.
Label noise SGD converges to a simple model with a single linear feature.
problem Understanding the simplicity bias in neural network training.
method Analyzing the convergence of label noise SGD on two-layer neural networks.
result Label noise SGD converges to a model with a single linear feature.
In this paper, we present some theoretical work to explain why simple gradient descent methods are so successful in solving non-convex optimization problems in learning large-scale neural networks (NN). After introducing a mathematical tool called canonical space, we have proved that the objective functions in learning…
Representing shapes as level sets of neural networks has been recently proved to be useful for different shape analysis and reconstruction tasks. So far, such representations were computed using either: (i) pre-computed implicit shape representations; or (ii) loss functions explicitly defined over the neural level sets…
Constructs classifiers for neural networks with specific data configurations.
problem Finding global minima of deep ReLU neural networks on sequentially separable data.
method Explicitly constructs zero loss neural network classifiers using cumulative parameters and truncation maps.
result Global minimizers can be described with a limited number of parameters based on the data structure.
SLT reveals how grokking occurs via basin selection in training.
problem Understanding grokking in machine learning models.
method Singular Learning Theory (SLT) to analyze the loss landscape and local learning coefficient (LLC).
result LLC ranks basins by statistical preference, leading to grokking.
Gradient descent converges linearly for neural networks with specific conditions.
problem Optimizing neural networks with fixed width and depth.
method Local Polyak-Lojasiewicz criterion for gradient flow and descent.
result Gradient descent converges to zero-loss solutions under certain conditions.
New DP algorithms achieve near-optimal regret bounds for online learning problems.
problem Online learning problems with zero-loss solutions and differential privacy constraints.
method Developed new Differentially Private algorithms with near-optimal regret bounds.
result Achieved near-optimal regret bounds for various online prediction and convex optimization problems.
We explore how neural networks train to zero loss, focusing on initial scale.
problem Understanding neural network training dynamics and zero loss.
method Macroscopic limits analysis of gradient descent dynamics.
result Gradient descent can drive deep neural networks to zero loss regardless of initialization.
Generalization performance of classifiers in deep learning has recently become a subject of intense study. Deep models, typically over-parametrized, tend to fit the training data exactly. Despite this "overfitting", they perform well on test data, a phenomenon not yet fully understood. The first point of our paper is t…
We analyze multi-layer neural networks in the asymptotic regime of simultaneously (A) large network sizes and (B) large numbers of stochastic gradient descent training iterations. We rigorously establish the limiting behavior of the multi-layer neural network output. The limit procedure is valid for any number of hidde…
This study explains gradient flow dynamics in neural networks for small initialisation.
problem Understanding the training dynamics of neural networks for small initialisation.
method Analysis of gradient flow dynamics for one-hidden layer ReLU networks with orthogonal inputs.
result Gradient flow converges to zero loss and characterizes implicit bias towards minimum variation norm.
Deep ReLU networks generalize well with few parameters.
problem Generalization of overparametrized deep neural networks.
method Explicit bounds on test error independent of overparametrization and VC dimension.
result Generalization error is independent of network architecture and overparametrization.
Automated tool reduces FPGA inference latency to 5 μs for deep neural networks.
problem Deploying ultra low-latency, low-power deep neural networks on FPGAs.
method Extending hls4ml library, using model compression techniques like pruning and quantization-aware training.
result Achieved inference latency of 5 μs with 97% resource reduction.
Wide neural networks converge linearly to zero loss with feature learning.
problem Optimizing wide neural networks with feature learning guarantees.
method Gradient flow analysis for wide shallow and multi-layer NNs.
result Training loss converges linearly to zero for wide NNs under GF, demonstrating feature learning and better generalization.
New method trains shallow neural networks with subquadratic width scaling.
problem Training shallow neural networks with optimal width scaling.
method Polyak-Lojasiewicz condition, smoothness, standard data assumptions, random matrix theory.
result Subquadratic scaling on network width with standard initialization strategies.
We analyze single-layer neural networks with the Xavier initialization in the asymptotic regime of large numbers of hidden units and large numbers of stochastic gradient descent training steps. The evolution of the neural network during training can be viewed as a stochastic system and, using techniques from stochastic…
Augmenting a neural network with memory that can grow without growing the number of trained parameters is a recent powerful concept with many exciting applications. We propose a design of memory augmented neural networks (MANNs) called Labeled Memory Networks (LMNs) suited for tasks requiring online adaptation in class…
This paper explores how train-validation splits help in NAS to prevent overfitting.
problem NAS overfits with train-validation splits and needs better generalization guarantees.
method Established refined properties of validation loss and risk for NAS.
result NAS with train-validation splits can select the most generalizable model.
Newton's method converges faster than gradient descent in overparameterized neural networks.
problem Training neural networks efficiently in the overparameterized limit.
method Developed a convergence analysis for the regularized Newton method in this context.
result The NN training dynamics converge to the solution of a deterministic limit equation involving a Newton neural tangent kernel (NNTK).
Stochastic Gradient Descent (SGD) is a central tool in machine learning. We prove that SGD converges to zero loss, even with a fixed (non-vanishing) learning rate - in the special case of homogeneous linear classifiers with smooth monotone loss functions, optimized on linearly separable data. Previous works assumed eit…
Randomly trained neural networks can generalize well if there's a simpler underlying teacher model.
problem Why randomly trained neural networks generalize well despite interpolating training data.
method Examined a random neural network that interpolates training data and showed it generalizes well if there's a simpler underlying teacher model.
result Randomly trained neural networks can generalize well if there's a simpler underlying teacher model.
Unified framework connects different neural network models.
problem Understanding the geometry of neural network loss landscapes.
method Unified framework capturing four symmetry classes.
result First discovery of low- and zero-barrier linear interpolation paths.
The paper proves ADL mechanisms face a trilemma and optimizes them for fairness, revenue, and exchange solvency.
problem The impossibility of a perpetual futures exchange achieving solvency, revenue, and fairness.
method Formal model of ADL, proving trilemma, and analyzing three ADL mechanisms.
result Optimized ADL mechanisms can reduce trader losses while maintaining exchange solvency.
We present a formal measure-theoretical theory of neural networks (NN) built on probability coupling theory. Our main contributions are summarized as follows. * Built on the formalism of probability coupling theory, we derive an algorithm framework, named Hierarchical Measure Group and Approximate System (HMGAS), nickn…
Study shows how deep residual networks can be analyzed as shallow network ensembles for optimization.
problem Understanding why deep neural networks can be trained to zero loss despite non-convex optimization landscapes.
method Mean-field analysis of deep residual networks, focusing on their continuum limit as a two-layer network.
result Derives the first global convergence result for multilayer neural networks in the mean-field regime.
This paper investigates how large language models achieve neural collapse, a phenomenon linked to generalization.
problem Neural collapse in large language models under imbalanced and token-rich conditions.
method Empirical investigation of scaling and regularization effects on CLMs' progression towards neural collapse.
result Neural collapse properties develop with scale and regularization, linked to generalization in language modeling.
The paper analyzes the implicit bias of SGD near loss manifold and provides new insights.
problem Understanding the implicit bias of SGD near loss manifolds in overparametrized models.
method Adapting ideas from Katzenberger (1991) to analyze SGD dynamics using a stochastic differential equation (SDE).
result SGD with label noise locally decreases the sharpness of loss, leading to a global analysis of implicit bias.
The paper examines partial regularity of Lipschitz solutions to minimal surface system.
problem Understanding the regularity of solutions to the minimal surface system.
method Investigation of stationary, integral weak, and viscosity solutions; interior gradient estimate using maximum principle.
result Partial regularity results for Lipschitz solutions, including interior gradient estimate.
Unique ancient solutions found for anisotropic curve shortening flow.
problem Finding unique solutions for anisotropic curve shortening flow.
method Constructing translating and ancient solutions under given conditions.
result Unique ancient and translating solutions found for anisotropic curve shortening flow.
The paper constructs solutions to a critical Dirac equation on spheres.
problem Solving the critical Dirac equation on spheres with singularities.
method Constructing Delaunay-type solutions and another kind of singular solutions.
result The constructed solutions are building blocks for singular solutions on Spin manifolds.
New findings on κ-solutions with round cylinder as asymptotic shrinker.
problem Characterizing κ-solutions with specific asymptotic behavior. method Analysis of Ricci flow in dimensions n≥4. result Uniformly Positive Isoperimetric Constant (PIC) for κ-solutions. Study higher-dimensional Ricci flow solutions, proving uniqueness.
problem Classifying ancient solutions to the Ricci flow on Sn. method Extending [13] to higher dimensions, proving uniqueness.
result Ancient solutions are either shrinking spheres or Type II solutions.
Ancient solutions of Ricci flow with Type I growth are classified.
problem Understanding ancient solutions of Ricci flow with specific curvature growth.
method Analyzing ancient solutions with Type I curvature growth in arbitrary dimensions.
result Ancient solutions with Type I growth are classified into specific types.
We construct low regularity solutions of the vacuum Einstein constraint equations. In particular, on 3-manifolds we obtain solutions with metrics in $H^s\loc$ with s>23. The theory of maximal asymptotically Euclidean solutions of the constraint equations descends completely the low regularity setting. Moreove…
Researchers find entire solutions to magnetic Ginzburg-Landau equations in 4D.
problem Finding entire solutions to magnetic Ginzburg-Landau equations in 4D.
method Using Lyapunov-Schmidt reduction.
result Existence of entire solutions and saddle type solutions with specific zero sets.
New ancient solutions found for curvature flow in 2D.
problem Ancient solutions for curvature flow in 2D.
method Constructing and classifying convex ancient solutions.
result All convex ancient solutions classified for α∈(32,1). Let n≥3 and m=n+2n−2. We construct 5-parameters, 4-parameters, 3-parameters ancient solutions of the equation vt=(vm)xx+v−vm, v>0, in R×(−∞,T) for some T∈R. This equation arises in the study of Yamabe flow. We obtain various properties of the ancient so…
We construct new ancient compact solutions to the Yamabe flow. Our solutions are rotationally symmetric and converge, as t→−∞, to two self-similar complete non-compact solutions to the Yamabe flow moving in opposite directions. They are type I ancient solutions.
Minimax solutions are weak solutions to Cauchy problems involving Hamilton--Jacobi equations, constructed from generating families quadratic at infinity of their geometric solutions. We give a complete description of minimax solutions and we classify their generic singularities of codimension not greater than 2.
Paper shows how solutions to Allen-Cahn converge to multiphase mean curvature flow.
problem Convergence of Allen-Cahn solutions to multiphase mean curvature flow.
method Conditional convergence result of Allen-Cahn solutions to De Giorgi type BV-solutions of multiphase mean curvature flow.
result De Giorgi type BV-solutions are unique in a weak-strong sense.
We construct new ancient compact solutions to the Yamabe flow. Our solutions are rotationally symmetric and converge, as t→−∞, to two self-similar complete non-compact solutions to the Yamabe flow moving in opposite directions. They are type I ancient solutions.
The paper concerns singular solutions of nonlinear elliptic equations, which include removable singularities for viscosity solutions, a strengthening of the Hopf Lemma including parabolic equations, Strong maximum principle and Hopf Lemma for viscosity solutions including also parabolic equations.
Generic level sets in mean curvature flow are BV solutions.
problem Understanding the behavior of level sets in mean curvature flow.
method Using the framework of sets of finite perimeter and distributional solutions, the paper extends Evans and Spruck's work.
result Generic level sets are distributional solutions with optimal energy dissipation rate.
Proves existence and uniqueness of viscosity solutions to complex Hessian equations on compact Hermitian manifolds.
problem Existence and uniqueness of viscosity solutions to complex Hessian equations.
method Proves existence and uniqueness using viscosity solutions and determinant domination conditions.
result Viscosity solutions exist and are unique under certain conditions.