SGD converges to zero loss for separable data with fixed learning rate.
problem Optimizing homogeneous linear classifiers with SGD on linearly separable data.
method Proved convergence of SGD with fixed learning rate for separable data.
result SGD converges to zero loss for separable data with fixed learning rate.
A new decentralized optimization method with independent step-sizes and separated convergence rates.
problem Decentralized optimization with composite objective terms.
method Proximal-gradient algorithm with uncoordinated step-sizes and separated convergence rates.
result Linear convergence for special case without non-smooth terms under strong convexity.
Gradient descent on logistic loss converges to the maximum-margin separator for separable data.
problem Understanding the convergence of gradient descent on separable datasets with specific loss functions.
method Analysis of gradient descent on linear models with super-polynomially tailed losses.
result For separable datasets, gradient descent converges to the maximum-margin separator for losses with super-polynomial tails, but not for heavier tails.
Study shows LDA topic models converge at rate n^-1/4 without strict topic separability.
problem Convergence rates of Latent Dirichlet Allocation (LDA) topic models.
method Maximum likelihood estimator, Wasserstein's distance metric, without separability or non-degeneracy assumptions.
result Maximum likelihood estimator converges at rate n^-1/4, optimal in worst case.
Understanding separation effects on parameter estimation in finite Gaussian mixtures
problem Minimum component separation impact on convergence rates in finite Gaussian mixtures
method Developing a unified geometric framework using Hellinger lower bounds and specialized moment-extraction test functions
result Separation complexity driven by spatial configuration of mixture components
Gradient descent converges with arbitrary stepsize for separable data under Fenchel-Young losses.
problem Understanding the conditions under which gradient descent converges with arbitrary stepsize.
method Using Fenchel-Young losses and leveraging the classical perceptron argument to derive convergence rates.
result GD converges with arbitrary stepsize for a majority of Fenchel-Young losses, with better rates for specific loss functions.
New algorithm AG-OG optimizes separable convex-concave problems efficiently.
problem Efficiently solving separable convex-concave minimax optimization problems.
method Leverages Nesterov acceleration and optimistic gradient on component and coupling parts of the problem.
result Achieves optimal convergence rate for various settings including bilinearly coupled problems.
Gradient descent-ascent converges to strict local minmax equilibria with a finite timescale separation.
problem Analyzing the convergence of gradient descent-ascent in non-convex, non-concave games with a finite timescale separation.
method Investigates the role of a finite timescale separation parameter τ on gradient descent-ascent in two-player zero-sum games, providing convergence rates and non-convergence results.
result Gradient descent-ascent converges to strict local minmax equilibria for a finite timescale separation parameter τ*.
This paper presents a new method for estimating high dimensional covariance matrices. The method, permuted rank-penalized least-squares (PRLS), is based on a Kronecker product series expansion of the true covariance matrix. Assuming an i.i.d. Gaussian random sample, we establish high dimensional rates of convergence to…
Study shows deep linear networks can converge to flatter minima at large learning rates.
problem Understanding the implicit bias of deep linear networks at large learning rates.
method Characterization of deep linear networks for binary classification using logistic loss in the large learning rate regime.
result Gradient descent iterates converge to a flatter minimum in the catapult phase for certain data separation conditions.
Adam optimizes linear classifiers with separable data.
problem Understanding Adam's implicit bias in linear logistic regression.
method Study of Adam's behavior on linearly separable data.
result Adam converges to a linear classifier with maximum ℓ ∞ \ell_\infty ℓ ∞ -margin. New loss function improves convergence rate for neural networks.
problem Improving convergence speed of neural networks for classification tasks.
method Proposes a modified hinge loss function with gradients to critical points.
result Margin converges to max-margin at O ( 1 / t ) \mathcal{O}(1/t) O ( 1/ t ) rate, faster than exponential losses. Study uniform rates for estimating Gaussian mixtures without separation assumption.
problem Estimating parameters in two-component Gaussian mixtures without separation.
method Uniform convergence rates derived using minimax lower bounds and careful analysis of polynomial equalities.
result Phase transition in optimal estimation rate based on mixture balance.
Last iterate of Extragradient algorithm converges slower than averaged iterates in saddle point problems.
problem Smooth convex-concave saddle point problems
method Analysis of Extragradient (EG) algorithm convergence rates
result The last iterate of EG converges at a rate of O(1/√T), compared to O(1/T) for averaged iterates
EM algorithm quickly finds near-optimal parameters in Gaussian mixtures.
problem Parameter estimation in symmetric two-component Gaussian mixtures.
method Randomly initialized EM algorithm, analyzing convergence rate and optimality.
result EM algorithm converges in O ( n ) O(\sqrt{n}) O ( n ) iterations with high probability. Gradient descent converges to max-margin solution for hinge loss.
problem Applying gradient descent to the hinge loss for linear classifiers.
method Homotopic gradient descent applied to the hinge loss.
result Explicit convergence rates to max-margin solution for separable data.
Gradient descent in logistic regression converges to the maximum margin predictor.
problem Convergence and risk of logistic regression parameters.
method Gradient descent applied to logistic regression.
result Gradient descent iterates converge to the maximum margin predictor at a rate of O ( ln ln t / ln t ) \mathcal{O}(\ln\ln t / \ln t) O ( ln ln t / ln t ) . Softmax policy gradient methods converge at O ( 1 / t ) O(1/t) O ( 1/ t ) rate with constants depending on problem and initialization.
problem Understanding convergence rates of softmax policy gradient methods in tabular settings.
method Analysis of softmax policy gradient and entropy regularized policy gradient methods, using Łojasiewicz inequality and lower bounds.
result Entropy regularization improves convergence rate from O ( 1 / t ) O(1/t) O ( 1/ t ) to O ( e − c ⋅ t ) O(e^{-c \cdot t}) O ( e − c ⋅ t ) . Gradient descent-based adversarial training converges to robust classifiers on linearly separable data.
problem Understanding the inductive bias of adversarial training for robustness.
method Gradient descent on binary classification tasks with linearly separable data, focusing on inductive bias and convergence rates.
result Gradient descent-based adversarial training converges to the maximum margin classifier at a faster rate than clean data training.
Stochastic algorithm achieves sublinear convergence for bi-objective optimization.
problem Optimizing two conflicting functions using gradient or subgradient descent.
method Stochastic alternating algorithm with varying steps for each objective.
result Achieves sublinear convergence rate of O(1/T) under strong convexity.
A lot of effort has been invested into characterizing the convergence rates of gradient based algorithms for non-linear convex optimization. Recently, motivated by large datasets and problems in machine learning, the interest has shifted towards distributed optimization. In this work we present a distributed algorithm …
The paper analyzes reg-SGD for convex problems, proving convergence and quantifying the rate of convergence.
problem Minimizing convex, L-smooth functions in a Hilbert space.
method Regularized stochastic gradient descent with decaying regularization.
result Strong convergence to the minimum-norm solution without boundedness assumptions.
This work analyzes and improves stochastic gradient methods for GAN training.
problem Understanding the training dynamics of GANs, particularly their convergence.
method Continuous-time analysis using differential equations, focusing on simGD and its variants.
result The methods converge under different assumptions, providing new insights into GAN training.
Study separates learning rate effects from adaptive gradient methods.
problem Understanding the impact of learning rates on neural network training.
method Introduced a 'grafting' experiment to isolate learning rate effects.
result Many existing beliefs about adaptive gradient methods may be incorrect.
Deep neural networks outperform traditional methods in high-dimensional classification.
problem Understanding the empirical success of deep neural networks in high-dimensional classification.
method Proposed a teacher-student framework with Bayes classifier as ReLU neural networks, derived convergence rates for 0-1 and hinge losses.
result Sharp rate of convergence for classifiers trained using 0-1 or hinge loss, with O d ( n − 2 / 3 ) O_d(n^{-2/3}) O d ( n − 2/3 ) or O d ( n − 1 ) O_d(n^{-1}) O d ( n − 1 ) under separable data distribution. Paper proposes a quasi-Newton method for nonlinear equations with global convergence guarantees.
problem Solving smooth and monotone nonlinear equations efficiently and globally.
method Hybrid proximal extragradient framework combined with online learning for Jacobian approximation.
result First global convergence results showing quasi-Newton method's advantage over extragradient method.
Unified analysis for decentralized SGD across various topologies and updates.
problem Analysis of decentralized SGD methods with changing topologies and local updates.
method Unified convergence analysis covering local SGD updates and adaptive network topology.
result Universal convergence rates for smooth problems, interpolating between heterogeneous and iid-data settings.
Local Gradient Descent with local steps converges to the centralized model in the interpolation regime.
problem Understanding the implicit bias of Local Gradient Descent in the interpolation regime.
method Analyzing the implicit bias of Local Gradient Descent for classification tasks with linearly separable data.
result The aggregated global model from Local-GD converges exactly to the centralized model in the interpolation regime.
Large stepsize GD for logistic regression converges faster than expected.
problem Optimizing logistic regression with large step sizes.
method Gradient descent with large stepsize applied to logistic regression.
result GD converges to a lower loss in fewer steps than expected.
Unified framework for accelerated Perceptron and related problems.
problem Finding optimal linear threshold functions for classification.
method Modern acceleration techniques, specifically optimistic online learning.
result Improved convergence rates for various Perceptron-related problems.
Study optimal rates for spectral algorithms in Hilbert spaces.
problem Regression problems over separable Hilbert spaces with square loss.
method Investigate spectral/regularized algorithms including ridge, principal component, and gradient methods.
result Prove optimal, high-probability convergence results in terms of norms.
In this paper we analyze the randomized block-coordinate descent (RBCD) methods proposed in [8,11] for minimizing the sum of a smooth convex function and a block-separable convex function. In particular, we extend Nesterov's technique developed in [8] for analyzing the RBCD method for minimizing a smooth convex functio…
Gradient descent reveals the exact implicit bias via dual optimization for linearly separable data.
problem Characterizing the implicit bias of gradient descent on linearly separable data.
method Primal-dual analysis with smoothed margin for general losses, and exponential loss with specific step sizes.
result Proves faster convergence rates for implicit bias and margin maximization.
New algorithm for nonnegative tensor completion with linear convergence rate.
problem Tensor completion without known optimal sample complexity rate.
method Integer optimization using a specific 0-1 polytope gauge norm.
result Achieves information-theoretic rate with linear convergence.
Develops large-sample theory for non-stationary source separation.
problem Lack of large-sample results for non-stationary source separation methods.
method Large-sample theory for NSS-JD method under specific assumptions.
result Consistency of unmixing estimator and its convergence to Gaussian distribution.
This work improves density estimation by characterizing pdf complexity using NL-spectrum.
problem Improving density estimation rates for general probability densities.
method Introducing NL-spectrum to characterize pdf complexity and deriving dimension-independent rates of convergence.
result Dimension-independent rates of convergence for fast density estimation.
We present a stochastic setting for optimization problems with nonsmooth convex separable objective functions over linear equality constraints. To solve such problems, we propose a stochastic Alternating Direction Method of Multipliers (ADMM) algorithm. Our algorithm applies to a more general class of nonsmooth convex …
Study shows directional convergence for neural networks under spherical symmetry.
problem Learning linear predictors with neural networks under spherically symmetric data.
method Analysis of gradient flow and gradient descent for two-layer and deep linear networks.
result Directional convergence guarantees with exact convergence rate for specific network architectures.
Develops a generalized version of Chung's Lemma for stochastic optimization methods.
problem Establishing asymptotic convergence rates for stochastic optimization methods under various step size rules.
method Generalized version of Chung's Lemma for a broader family of step size rules.
result Demonstrates tight non-asymptotic convergence rates for various stochastic methods.
Paper analyzes convergence of decentralized algorithms with noise and bias.
problem Finite time convergence analysis of decentralized stochastic approximation schemes.
method Separated iterates into consensual parts and consensus error; bounded consensus error in terms of stationarity.
result Decentralized SA scheme converges at O ( log T / T ) {\cal O}(\log T/ \sqrt{T} ) O ( log T / T ) rate. EM algorithm's convergence rate changes drastically in over-specified mixtures.
problem Behavior of EM algorithm in over-specified Gaussian mixture models.
method Analysis of Fisher information matrix, localization in empirical process, recursive argument.
result EM algorithm converges at different rates depending on mixture weights.
New conditions ensure MMDs separate and converge to target distributions.
problem Ensuring MMDs separate and converge to target distributions.
method Deriving new sufficient and necessary conditions for MMDs on separable metric spaces.
result First KSDs that exactly metrize weak convergence to P.
A new method reduces computational time for source separation and classification.
problem High computational complexity and unsatisfactory source classification accuracy in MVAE.
method Integrates an auxiliary classifier VAE to reduce computational time and improve classification accuracy.
result fMVAE achieved comparative source separation performance and 80% source classification accuracy while reducing computational time by 93%.
The paper proves a margin inequality for separating hyperplanes, useful for analyzing algorithmic bias.
problem Analyzing the implicit bias of algorithms in machine learning.
method Proves a nonsmooth Kurdyka-Lojasiewicz inequality for margin function.
result The bias of algorithm iterates converges at least as fast as the square-root of the margin convergence rate.
New methods for estimating nested expectations in machine learning.
problem Nested expectations in machine learning and statistics.
method Investigation and analysis of statistical implications of nesting Monte Carlo estimators.
result Established conditions for convergence of nested MC estimators and derived corresponding rates.
Gradient descent with logistic loss can interpolate deep networks with smoothed ReLU activations under certain conditions.
problem Conditions for gradient descent to drive logistic loss to zero in deep networks with smoothed ReLU activations.
method Gradient descent applied to fixed-width deep networks with smoothed ReLU approximations (e.g., Swish, Huberized ReLU).
result Gradient descent can drive logistic loss to zero under specific conditions, providing bounds on convergence rate.
Investigates fund separations and stability for long-term optimal investments.
problem Optimizing long-term investments in an incomplete market with risky and safe assets.
method Analyzes three market models with different state variable processes to find optimal portfolios and prove convergence stability.
result Dynamic optimal portfolios converge to static portfolios over time, with vanishing sensitivities in the long run.
New algorithm quantifies uncertainty in regression models for complex data types.
problem Uncertainty quantification in regression models for complex data types.
method Model-free uncertainty quantification algorithm based on conditional depth measures and kernel mean embeddings.
result Provides faster convergence rates and non-asymptotic guarantees for prediction regions.