This paper optimizes functions of probability measures using particle gradient descent for displacement convex functions.
problem Optimizing functions of probability measures with displacement convex properties.
method Particle gradient descent applied to displacement convex functions with theoretical guarantees.
result Finite number of particles and computations are sufficient to find optimal solutions for displacement convex functions.
Improved stochastic gradient descent analysis for non-smooth convex functions.
problem Minimizing non-smooth, non-differentiable convex functions.
method Stochastic gradient descent with suffix averaging method analysis.
result Error rate of final iterate is O(log(T)/T) with high probability. Unified view of GANs, VI, RL through PFD.
problem Unified understanding of GANs, VI, RL.
method Probability Functional Descent (PFD) algorithm.
result PFD unifies GANs, VI, RL.
Paper solves barycenter of probability distributions using Sinkhorn divergence.
problem Computing the barycenter of a set of probability distributions under the Sinkhorn divergence.
method Recast as unconstrained functional optimization and develop Sinkhorn Descent (SD) method.
result SD converges to a stationary point at a sublinear rate and asymptotically finds a global minimizer.
New averaging strategy achieves optimal convergence rate with high probability.
problem Optimizing convergence rate for strongly-convex functions.
method Simple non-uniform averaging strategy combined with Freedman's inequality.
result Achieves optimal O(1/T) convergence rate with high probability. This paper analyzes Stein variational gradient descent for Bayesian inference.
problem Sampling or approximating high-dimensional probability distributions.
method Iterated steepest descent steps with a reproducing kernel Hilbert space norm.
result Performance gains of certain nondifferentiable kernels with adjusted tails.
Gradient descent learns ReLU functions with non-zero bias efficiently.
problem Learning ReLU functions with non-zero bias under Gaussian distributions.
method Gradient descent starting from random initialization.
result Gradient descent achieves near-optimal error with high probability.
SGD converges with positive probability for non-convex deep neural networks under specific conditions.
problem Convergence of SGD for non-convex deep neural networks.
method Established local convergence with positive probability under local Łojasiewicz condition and additional structural assumption.
result SGD converges with positive probability for non-convex deep neural networks under specific conditions.
Accelerates MMLE using SVGD with Nesterov acceleration.
problem Maximum Marginal Likelihood Estimation optimization.
method Stein variational gradient descent with Nesterov acceleration.
result Consistently accelerates convergence across various tasks.
Stochastic gradient methods converge for training wide PINNs.
problem Convergence of stochastic gradient descent in training over-parameterized PINNs.
method Established linear convergence of stochastic gradient descent/flow in training over-parameterized two-layer PINNs.
result Linear convergence with high probability for general activation functions.
KSD Descent uses KSD to sample from a target distribution efficiently.
problem Sampling from complex target distributions efficiently.
method Wasserstein gradient flow of KSD, using L-BFGS optimization.
result KSD Descent can sample from a target distribution using a set of particles.
Study large deviations rates for SGD with strongly convex functions.
problem High probability metrics with SGD.
method Large deviations theory, generic gradient noise, strongly convex functions.
result Upper large deviations bound for SGD with strongly convex functions.
Determining the appropriate batch size for mini-batch gradient descent is always time consuming as it often relies on grid search. This paper considers a resizable mini-batch gradient descent (RMGD) algorithm based on a multi-armed bandit for achieving best performance in grid search by selecting an appropriate batch s…
Gradient descent achieves fast convergence for approximating functions with two-layer neural networks.
problem Approximating continuous functions with two-layer neural networks.
method Gradient descent combined with generic chaining technique from probability theory.
result Gradient descent yields an exponential convergence rate for two-layer neural networks without needing a large width relative to the number of data points.
In reinforcement learning, temporal difference (TD) is the most direct algorithm to learn the value function of a policy. For large or infinite state spaces, exact representations of the value function are usually not available, and it must be approximated by a function in some parametric family. However, with \emph{no…
Gradient descent training of neural networks leads to solutions close to natural cubic splines.
problem Understanding the implicit bias of gradient descent in neural networks.
method Analysis of gradient descent training for wide neural networks, focusing on the curvature penalty and initialization schemes.
result The solutions of gradient descent training are polyharmonic splines for certain initialization schemes.
In this paper we develop a randomized block-coordinate descent method for minimizing the sum of a smooth and a simple nonsmooth block-separable convex function and prove that it obtains an ε-accurate solution with probability at least 1−ρ in at most O(εnlogρ1) iterations, where n is the numbe…
Study on privacy leakage in noisy gradient descent algorithms.
problem Information leakage of iterative randomized learning algorithms about training data.
method Analyzes the dynamics of Rényi differential privacy loss in noisy gradient descent algorithms.
result Privacy loss converges exponentially fast for smooth and strongly convex loss functions.
Derives fluctuation-dissipation relations for SGD, linking hyperparameters and measurable quantities.
problem Understanding and optimizing the training process of machine learning models.
method Derives stationary fluctuation-dissipation relations for stochastic gradient descent.
result Stationary fluctuation-dissipation relations hold for any stationary state and can be used to adaptively set training schedules.
We consider stochastic gradient descent and its averaging variant for binary classification problems in a reproducing kernel Hilbert space. In the traditional analysis using a consistency property of loss functions, it is known that the expected classification error converges more slowly than the expected risk even whe…
New method speeds up optimization over probability measures.
problem High computational overhead in optimizing probability measures.
method Randomized coordinate descent on Wasserstein space.
result Significant speedups over full-gradient methods.
Gradient descent can use larger step sizes to avoid strict saddle points.
problem Avoiding strict saddle points in non-convex optimization.
method Proving that gradient descent with step-size up to 2/L avoids strict saddle points with high probability.
result Gradient descent with step-size up to 2/L almost surely avoids strict saddle points.
A novel approach to computing barycenters on graph-supported probability measures.
problem Computing weighted averages of measures on graphs.
method Dynamic optimal transport formulation on the simplex, gradient descent on the probability simplex.
result Intrinsic gradient descent provides a coherent framework for synthesizing and analyzing measures on graphs.
A new method for efficient sampling from probability measures.
problem Efficiently sampling from complex probability distributions.
method RBM-SVGD, a stochastic version of SVGD using Random Batch Method.
result Reduces computational cost, especially for long-range kernels.
Gradient descent on MMD GAN parameter space converges globally to target distribution.
problem Convergence of gradient descent in Maximum Mean Discrepancy (MMD) GANs.
method Proposes a parametric kernelized gradient flow that mimics the min-max game in gradient regularized MMD GAN.
result Gradient descent on the generator's parameter space in gradient regularized MMD GAN is globally convergent to the target distribution under certain conditions.
Stochastic subgradient descent avoids critical points in definable functions.
problem Finding local minima in definable functions.
method Stochastic subgradient descent with density-like perturbation.
result SGD converges to a local minimum in definable functions.
Gradient descent fails to train two-layer ReLU networks, leading to poor performance.
problem Gradient descent training of two-layer ReLU networks initialized by He et al. (2015) fails to find optimal solutions.
method Gradient descent on a least-squares loss for training two-layer (Leaky)ReLU networks.
result Gradient descent only finds bad local minima, leading to linear regression for non-linear target functions.
Gradient descent finds global minima in deep neural networks with skip-connections.
problem Finding global minima in deep neural networks with skip-connections.
method Analysis of the gradient descent algorithm in over-parametrized deep neural networks with skip-connections.
result Gradient descent can find global minima exponentially fast in the over-parametrized regime.
New method improves scalability of SGD for large datasets.
problem High variance in stochastic gradient descent.
method Adaptive measure reduction with Carathéodory's theorem.
result Improved scalability to high-dimensional spaces.
New algorithms improve computation of optimal transport and Wasserstein barycenter.
problem Computing optimal transport and Wasserstein barycenter for multiple probability distributions.
method Introduced APDRCD and APDGCD algorithms for efficient computation, demonstrating better performance than existing methods.
result New algorithms match or exceed the best known complexities for OT problems and improve practical performance.
A new optimization method for probability simplex problems.
problem Optimizing convex problems over the probability simplex.
method Cauchy-Simplex iteration scheme, mapping to sphere, gradient descent, and back-mapping.
result Convergence results and faster convergence in high dimensions.
One-pass SGD converges in overparametrized neural networks with random data.
problem Understanding convergence of SGD in neural networks with streaming data.
method Overparameterized two-layer neural networks, one-pass SGD, random initialization, NTK eigen-decomposition, VC dimension, McDiarmid's inequality.
result Prediction error converges in expectation under one-pass SGD in overparametrized neural networks.
Paper analyzes SVGD algorithm for non-asymptotic convergence.
problem Optimizing a set of particles to approximate a target probability distribution.
method Finite time analysis of SVGD algorithm, providing descent lemma and convergence rates.
result SVGD algorithm decreases the objective at each iteration and converges to the target distribution.
MWGraD solves multi-objective distributional optimization using particle-based gradient descent.
problem Simultaneously minimize multiple objective functionals over probability distributions.
method Iterative particle-based algorithm MWGraD, estimating and aggregating Wasserstein gradients.
result Demonstrates effectiveness on synthetic and real-world datasets.
Information geometry applies concepts in differential geometry to probability and statistics and is especially useful for parameter estimation in exponential families where parameters are known to lie on a Riemannian manifold. Connections between the geometric properties of the induced manifold and statistical properti…
Gradient descent stagnates in low-precision, but unbiased rounding schemes improve convergence.
problem Stagnation of gradient descent in low-precision computation.
method Proposed unbiased stochastic rounding schemes that trade zero bias for larger probability of preserving small gradients.
result Unbiased rounding methods typically improve convergence rate of gradient descent for convex problems.
This work explores CS-RBFs as novel probability density functions.
problem Underexplored use of CS-RBFs in statistics and probability modeling.
method Derivation of statistical properties of CS-RBFs as univariate and conditional densities, analysis of mixture models, and introduction of an incremental learning algorithm.
result CS-RBF densities provide competitive results in likelihood and model complexity compared to Gaussian mixture models.
SGD optimality proven for convex objectives without smoothness assumptions.
problem Proving optimality of SGD for convex objectives without smoothness assumptions.
method Stochastic Gradient Descent (SGD) for convex objectives without smoothness or strict convexity assumptions.
result With high probability, the objective evaluated at the final candidate minimizer is close to the minimal value of the objective.
Paper analyzes high probability convergence of adaptive SGD with momentum.
problem Theoretical understanding of adaptive SGD with momentum in nonconvex settings is incomplete.
method High probability analysis under weak assumptions.
result First high probability convergence proof for gradients to zero in Delayed AdaGrad with momentum.
Unified approach combining gradient descent and multiplicative updates.
problem Combining gradient descent and multiplicative updates for machine learning.
method Introduces hypentropy and a family of matrix-based updates.
result Derives tight regret bounds for the new family of updates.
Deep ResNets can achieve zero loss with enough layers and weights.
problem Finding parameters in deep ResNets that fit training data perfectly.
method Mean-field analysis and gradient descent flow to a PDE.
result Gradient descent for ResNet training converges to a zero-loss solution in the large-NN limit.
We propose and analyze a new parallel coordinate descent method---`NSync---in which at each iteration a random subset of coordinates is updated, in parallel, allowing for the subsets to be chosen non-uniformly. We derive convergence rates under a strong convexity assumption, and comment on how to assign probabilities t…
New guarantees for SGD in non-convex optimization without strict noise bounds.
problem Efficiently escaping saddle points in non-convex optimization.
method Mean-square arguments and relaxed gradient noise variance bounds.
result Gradient descent can efficiently escape saddle points with a more relaxed gradient noise variance bound.
We consider the problem of learning a one-hidden-layer neural network with non-overlapping convolutional layer and ReLU activation, i.e., f(Z,w,a)=∑jajσ(wTZj), in which both the convolutional weights w and the output weights a are paramete…
New algorithm helps escape saddle points in optimization problems.
problem Optimizing smooth non-convex functions to avoid saddle points.
method Perturbed Saddle-escape Descent (PSD) algorithm with explicit constants.
result PSD finds approximate second-order stationary points efficiently.
New algorithm reduces sample complexity for learning CNNs.
problem Learning one-hidden-layer CNNs with various activation functions.
method Approximate gradient descent algorithm for training CNNs.
result Sample complexity matches information-theoretic lower bound for linear activation functions.
Proves hardness of semi-discrete optimal transport and proposes regularization methods.
problem Computing Wasserstein distance between discrete and non-discrete probability measures.
method Proves hardness, introduces distributionally robust dual optimal transport, regularizes primal objective, uses stochastic gradient descent.
result Regularization schemes and improved convergence guarantees for semi-discrete optimal transport problems.
Randomized SINDy learns dynamic data structures using probabilistic methods.
problem Learning time-dependent data structures in dynamic systems.
method Sequential machine learning with a probabilistic approach, incorporating feature augmentation and Tikhonov regularization.
result Demonstrated effectiveness in regression and binary classification using real-world data.