Probabilistic descent on manifolds using triangular sets and embedding theorems.
problem Descent over manifolds defined by polynomials.
method Triangularization, embedding theorem, numerical continuation.
result Effective numerical method for probabilistic descent.
Stochastic gradient descent's long-term fluctuations are described by a diffusion limit.
problem Long-term behavior of stochastic gradient descent in non-smooth settings.
method Functional central limit theorem applied to rescaled trajectory of SGD.
result Characterization of long-term fluctuations around the minimizer.
Generalizes nonabelian Hodge theory to klt singularities.
problem Applying nonabelian Hodge theory to spaces with klt singularities.
method Uses descent theorems for numerically flat vector bundles and a new restriction theorem for semistable Higgs sheaves.
result Establishes a new restriction theorem for semistable Higgs sheaves.
First-order methods avoid saddle points for most initializations.
problem Avoiding saddle points in optimization problems.
method First-order methods, including gradient descent and variants, analyzed using dynamical systems and the Stable Manifold Theorem.
result First-order methods avoid saddle points for almost all initializations.
Continuous-time SGD converges to optimal parameters via CLT.
problem Learning continuous-time models efficiently.
method Stochastic gradient descent in continuous time (SGDCT).
result Proves a central limit theorem for SGDCT's convergence.
We show that gradient descent converges to a local minimizer, almost surely with random initialization. This is proved by applying the Stable Manifold Theorem from dynamical systems theory.
The paper analyzes gradient descent algorithms using stochastic differential equations.
problem Understanding the asymptotic behaviors of gradient descent algorithms in statistical and computational contexts.
method Modeling gradient descent algorithms as stochastic differential equations and applying gradient flow central limit theorems.
result Identifies four factors affecting the local minima found by stochastic gradient descent.
Study local system points on surfaces using group descent.
problem Understanding integral points on moduli of local systems.
method Mapping class group descent and boundedness results for systoles.
result Established structure theorem for integral points.
Stochastic gradient descent on manifolds improves low-rank approximation.
problem Efficiently approximate large matrices with lower rank.
method Stochastic gradient descent on a manifold.
result Algorithm outperforms Euclidean space methods on Netflix Prize data.
New stochastic gradient descent with random search directions improves efficiency and convergence.
problem Efficiency and convergence of stochastic gradient descent methods.
method Developed a new class of stochastic gradient descent algorithms with random search directions.
result Established almost sure convergence and provided L p \mathbb{L}^p L p rates of convergence. Conservative SPDEs emerge from fluctuating SGD dynamics in neural networks.
problem Understanding the convergence of stochastic gradient descent to SPDEs.
method Mean-field analysis and central limit theorem for SPDEs.
result Optimal convergence rates for SPDEs derived from SGD.
Study on double descent behavior in two-layer neural networks for binary classification.
problem Understanding the double descent phenomenon in model test error.
method Two-layer neural network with ReLU activation for binary classification. Quantified model size by sample-to-dimension ratio. Empirical risk minimization using Convex Gaussian Min Max Theorem.
result Observed and investigated the double descent behavior of model test error.
Proves a central limit theorem for neural networks with hidden layers.
problem Understanding the statistical behavior of neural networks with large numbers of hidden units and training iterations.
method Rigorous mathematical proof using weak convergence methods and stochastic analysis.
result Neural network fluctuations around mean-field limit follow a Gaussian distribution and satisfy a stochastic partial differential equation.
Paper develops a method to construct confidence regions for model parameters using batch means method.
problem Constructing confidence regions for model parameters in stochastic gradient descent.
method Batch means method to cancel out covariance matrix, using Polyak-Ruppert averaging.
result Established process-level functional central limit theorem for stochastic gradient descent estimators.
Although stochastic approximation learning methods have been widely used in the machine learning literature for over 50 years, formal theoretical analyses of specific machine learning algorithms are less common because stochastic approximation theorems typically possess assumptions which are difficult to communicate an…
Developing a singular dimension descent method for positive scalar curvature obstructions
problem Positive scalar curvature obstructions in arbitrary dimensions
method Schoen--Yau type singular dimension descent method
result Proving obstructions to positive scalar curvature on enlargeable manifolds
In this paper we elaborate a general homotopy-theoretic framework in which to study problems of descent and completion and of their duals, codescent and cocompletion. Our approach to homotopic (co)descent and to derived (co)completion can be viewed as ∞ \infty ∞ -category-theoretic, as our framework is constructed in the …
Gradient descent dynamics in wide neural networks are analyzed using a dynamical CLT.
problem Understanding the fluctuations in wide shallow neural networks trained via gradient descent.
method Dynamical Central Limit Theorem (CLT) applied to neural network dynamics.
result Asymptotic fluctuations remain bounded in mean square throughout training.
Completes proof of index theorem using rigorous path integrals for supersymmetric quantum mechanics.
problem Analytic difficulties in generalizing Feynman's path integral to non-quadratic potentials.
method Develops rigorous path integrals for a class of Lagrangians including spinors on Riemannian manifolds.
result Steepest-descent approximation to path integral for twisted N = 1 / 2 N=1/2 N = 1/2 supersymmetric quantum mechanics is provably correct. New method for online inference using SGD with random scaling.
problem Efficient online inference for SGD parameters.
method Asymptotic pivotal statistics via random scaling.
result Robust and efficient online inference without resampling.
New method improves scalability of SGD for large datasets.
problem High variance in stochastic gradient descent.
method Adaptive measure reduction with Carathéodory's theorem.
result Improved scalability to high-dimensional spaces.
Study on privacy leakage in noisy gradient descent algorithms.
problem Information leakage of iterative randomized learning algorithms about training data.
method Analyzes the dynamics of Rényi differential privacy loss in noisy gradient descent algorithms.
result Privacy loss converges exponentially fast for smooth and strongly convex loss functions.
Paper proves CLT for quantile SGD with constant learning rate.
problem Quantile estimation via SGD with non-smooth, non-strongly convex loss.
method Viewed as a Markov chain, derived stationary distribution, analyzed MGF, proved CLT.
result Centered and standardized stationary distribution converges to Gaussian as η i g h t a r r o w 0 η
ightarrow0 η i g h t a r r o w 0 . Let S be a complete surface of constant curvature K = + 1 or -1, i.e. the sphere S^2 or the Lobachevskij plane L^2, and D a bounded convex subset of S. If S = S^2, assume also diameter (D) < pi/2. It is proved that the length of any steepest descent curve of a quasi-convex function in D is less than or equal to the per…
The Conant-Ashby theorem is verified for hypergraph observers, leading to unique learning rules.
problem Verifying conditions for hypergraph observers to maintain internal models.
method Formalizing persistent observers, applying the Conant-Ashby theorem, and using natural gradient descent.
result Natural gradient descent is the unique admissible learning rule for hypergraph observers.
Noether's theorem clarifies how symmetries in neural networks influence learning.
problem Understanding how symmetries in neural networks affect learning.
method Systematic study of symmetry interactions with learning algorithms using Noether's theorem.
result Symmetries impose restrictions on the optimization path, leading to conserved quantities.
Model reveals double descent in binary linear classification.
problem Investigating classification error in high-dimensional binary linear classification.
method Gradient descent on logistic loss, maximum-likelihood, max-margin (SVM) solutions, and convex Gaussian min-max theorem.
result Double descent phenomenon observed in classification error for varying overparameterization ratio.
The paper studies stochastic gradient descent with infinite variance gradients.
problem Theoretical properties of SGD with infinite variance gradients.
method Establish asymptotic behavior of SGD with infinite variance gradients.
result Asymptotic distribution of SGD is characterized as a stationary distribution of an Ornstein-Uhlenbeck process driven by a stable Lévy process.
Extends positive mass theorem to arbitrary dimensions using a new inductive scheme.
problem Overcoming singularities in the Schoen-Yau proof for arbitrary dimensions.
method Inductive scheme combining shielding principle, conformal blow-up, and Cheeger-Naber bound.
result Proof of positive mass theorem in arbitrary dimensions.
Fine-grained analysis of gradient descent with momentum provides modified loss equations.
problem Understanding the dynamics of gradient descent with momentum.
method Fine-grained analysis and derivation of modified loss equations.
result Global approximation bounds and continuous modified equations for HB.
Paper improves confidence set construction for SGD using multiplier bootstrap.
problem Constructing accurate confidence sets for SGD.
method Multiplier bootstrap procedure for non-asymptotic validity.
result Derives approximation rates up to 1 / n 1/\sqrt{n} 1/ n for convex distance. This monograph presents the main complexity theorems in convex optimization and their corresponding algorithms. Starting from the fundamental theory of black-box optimization, the material progresses towards recent advances in structural optimization and stochastic optimization. Our presentation of black-box optimizati…
Continuum transformers learn operators in context via gradient descent.
problem Generalizing transformers to handle infinite-dimensional inputs for in-context learning.
method Gradient descent in an operator RKHS, leveraging generalized representer theorems and gradient flows.
result Operator learned in context is Bayes Optimal Predictor in infinite depth limit.
Paper compares algebraic quantum field theories and factorization algebras on Lorentzian manifolds.
problem Relationship between algebraic quantum field theories and factorization algebras on Lorentzian manifolds.
method Developed functorial constructions under natural hypotheses, including local constancy and descent axioms.
result Equivalence theorem between algebraic quantum field theories and prefactorization algebras.
Paper proposes a federated learning method for quantile inference with local differential privacy.
problem Federated learning of quantile inference under local differential privacy constraints.
method Local stochastic gradient descent with randomized mechanism for privacy and efficiency.
result Asymptotic normality and functional central limit theorem for the proposed estimator.
The paper analyzes SGD in high-dimensional networks, revealing new scaling limits.
problem Understanding SGD dynamics in high-dimensional networks.
method Analyzing the effective dynamics of SGD using recent work on the subject.
result A new correction term emerges at the critical scaling regime, changing the phase diagram.
Study on descent properties of complex affine surfaces under proper morphisms.
problem Understanding descent behavior of homotopy-theoretic properties of smooth affine surfaces.
method Examined Eilenberg-MacLane property and introduced finite homotopy rank-sum property. Proved descent under proper morphisms for surfaces of log Kodaira dimension ≤0.
result Finite homotopy rank-sum property descends under proper morphisms for smooth affine surfaces of log Kodaira dimension ≤0.
Non-asymptotic rates for SGD via martingale CLT.
problem Improving the convergence rates of SGD.
method Combining Stein's method and Lindeberg's argument for multivariate martingale CLT, then applying to SGD.
result Explicit rates for multivariate martingale CLT and SGD convergence.
Deep networks learn hierarchical functions more efficiently than shallow ones.
problem Understanding the advantage of deep neural networks over shallow models.
method Analytical study of learning dynamics and generalization performance of deep networks compared to shallow ones.
result Deep networks reduce effective dimensionality, enabling learning with fewer samples.
The paper proves conditions for the existence of small Urysohn width hypersurfaces in manifolds with positive scalar curvature.
problem Conditions for the existence of small Urysohn width hypersurfaces in manifolds with positive scalar curvature.
method Adaptation of Guth's macroscopic version of the Schoen-Yau descent argument.
result A complete Riemannian manifold with positive macroscopic scalar curvature contains a non-nullhomologous hypersurface of small Urysohn width.
The paper analyzes SGD with dropout regularization in linear models, proving asymptotic properties and providing inference tools.
problem Analyzing the behavior of SGD with dropout regularization in linear models.
method Establishing geometric-moment contraction (GMC) and proving quenched central limit theorems (CLT).
result The existence of a unique stationary distribution and asymptotic normality results for SGD with dropout.
Stochastic Gradient Descent introduces noise in training, affecting model decision boundaries.
problem Understanding the impact of noise in SGD on model decision boundaries.
method Characterized SGD and persistent SGD dynamics in a neural network model, measuring noise magnitude in both under- and over-parametrized regimes.
result Noisier algorithms lead to wider decision boundaries in constraint satisfaction problems.
The paper develops methods for unconstrained optimization on Riemannian manifolds.
problem Optimization on Riemannian manifolds with general functions.
method Developed explicit versions of gradient descent and Newton's method for Riemannian optimization.
result The algorithms either converge to a local minimum or diverge to infinity, depending on the function and manifold properties.
New CLT for SGD in high-dimensional regression provides online inference.
problem Quantifying uncertainty in SGD for high-dimensional regression.
method Established a high-dimensional CLT for online SGD iterates.
result Developed an online approach for estimating variance in CLT.
Paper introduces a neural network training algorithm for noisy data that achieves optimal parameters and replicates real-world behaviors.
problem Theoretical gap between universal approximation theorems and practical machine learning with noisy data.
method Randomized training algorithm for neural networks trained on noisy data samples.
result Trained neural networks achieve optimal parameters and exhibit real-world behaviors like sub-linear complexity and interpolation.
Gradient descent forces neural network eigenvalues to a specific threshold.
problem Understanding why gradient descent drives eigenvalues to a specific threshold.
method Introduced edge coupling, a functional on consecutive iterate pairs, to explain the trajectory towards the eigenvalue threshold.
result Gradient descent forces the Hessian eigenvalue to the threshold 2 / η 2/η 2/ η from arbitrary initialization. High-dimensional SGD limits show surprising dynamics and phase transitions.
problem Understanding SGD in high dimensions and its scaling limits.
method Proving limit theorems for SGD trajectories in high dimensions, choosing summary statistics, initialization, and step-size.
result Critical scaling regime for step-size, new correction term, and complex diffusive limits.
This paper extends mirror descent to Banach spaces with reproducing kernels.
problem Optimizing in Banach spaces with reproducing kernels.
method Mirror descent algorithm adapted for Banach spaces with reproducing kernels.
result Mirror descent achieves linear convergence in certain conditions and standard convergence in a constrained setting.