Accelerates coordinate descent methods for machine learning problems.
problem Slowness of coordinate descent methods in machine learning.
method Extrapolation-based accelerated coordinate descent.
result Significant speed-up in practice compared to existing methods.
Study on gradient descent in Hilbert spaces with Markov chains, focusing on mixing coefficients.
problem Analyzing convergence of gradient descent in Hilbert spaces with stationary Markov chains.
method Examined strictly stationary Markov chains with φ- and β-mixing coefficients, derived probabilistic upper bounds. result Probabilistic upper bounds on convergence behavior of gradient descent algorithm based on mixing coefficients.
This paper introduces a new method for optimizing large-scale problems using Markov chain block updates.
problem Optimizing large-scale problems with efficient and natural block selection.
method Markov chain block coordinate descent (BCD) for optimization.
result The method converges for minimizing Lipschitz differentiable functions, with sublinear and linear convergence rates for convex and strongly convex functions, respectively.
Develops DP-SCD for stochastic coordinate descent, making it differentially private.
problem Privacy leak in auxiliary information during stochastic coordinate descent training.
method Develops DP-SCD, leveraging independent noise addition and decoupling/parallelizing coordinate updates.
result Demonstrates competitive performance against DP-SGD with less tuning.
Paper introduces robust learning methods using coordinate gradient descent.
problem Supervised learning with corrupted features and labels.
method Coordinate gradient descent combined with robust estimators of partial derivatives.
result Robust learning methods with nearly identical numerical complexity to non-robust ones.
LORIS model estimates main and interaction effects in large data frames.
problem Handling large data frames with missing values and explicit modeling of main effects.
method Low-rank interaction and sparse additive effects (LORIS) model with mixed coordinate gradient descent (MCGD).
result LORIS method provides statistical guarantees and converges efficiently for large data sets.
New method speeds up optimization over probability measures.
problem High computational overhead in optimizing probability measures.
method Randomized coordinate descent on Wasserstein space.
result Significant speedups over full-gradient methods.
New DP-CD method outperforms DP-SGD in solving composite DP-ERM problems.
problem Privacy-preserving machine learning with differential privacy.
method Differentially Private proximal Coordinate Descent (DP-CD) for composite Empirical Risk Minimization (ERM).
result DP-CD outperforms DP-SGD due to larger step sizes and better gradient exploitation.
New stochastic gradient descent with random search directions improves efficiency and convergence.
problem Efficiency and convergence of stochastic gradient descent methods.
method Developed a new class of stochastic gradient descent algorithms with random search directions.
result Established almost sure convergence and provided Lp rates of convergence. We propose accelerated randomized coordinate descent algorithms for stochastic optimization and online learning. Our algorithms have significantly less per-iteration complexity than the known accelerated gradient algorithms. The proposed algorithms for online learning have better regret performance than the known rando…
New algorithm combines gradient and coordinate descent steps for faster convergence.
problem Optimizing smooth convex functions over atom-spans.
method Blended matching pursuit combining coordinate descent and gradient descent.
result Derives linear convergence rates for non-strongly convex functions.
We use differential equations based approaches to provide some {\it \textbf{physics}} insights into analyzing the dynamics of popular optimization algorithms in machine learning. In particular, we study gradient descent, proximal gradient descent, coordinate gradient descent, proximal coordinate gradient, and Newton's …
Proposes blockwise adaptive stepsize for faster training and better generalization in deep learning.
problem Widespread use of coordinate-wise adaptive methods like RMSprop and Adam leads to worse generalization than SGD.
method Splits network parameters into blocks and uses a blockwise adaptive stepsize, balancing adaptivity and generalization.
result Blockwise adaptive gradient descent converges faster and has lower generalization error than coordinate-wise adaptive methods.
Method identifies mixed Nash equilibria in high dimensions for training mixtures of GANs.
problem Finding Nash equilibria in two-player zero-sum continuous games, especially in high dimensions.
method Parametrizing mixed strategies as mixtures of particles, updating their positions and weights using gradient descent-ascent.
result Global convergence to an approximate equilibrium for the related Langevin gradient-ascent dynamic.
New SGD method uses adaptive sampling to converge faster in non-convex problems.
problem Non-convex optimization problems with noisy gradients.
method Adaptive coordinate sampling in stochastic gradient descent (SGD).
result Almost sure convergence and non-asymptotic bounds established.
Differentially private random block coordinate descent improves utility in machine learning.
problem Lack of privacy in classical CD methods when handling sensitive information.
method Proposes a differentially private random block coordinate descent method using sketch matrices and importance sampling.
result Demonstrates improved convergence rates and utility guarantees compared to non-private methods.
New method reduces variance in random coordinate descent for Langevin Monte Carlo.
problem Efficient sampling from log-concave distributions in high dimensions.
method Introduces RCAD, a variance reduction technique for RCD-LMC.
result RCAD-O-LMC and RCAD-U-LMC converge within the same number of iterations as classical LMC methods, saving computational cost.
New method reveals insights about stochastic optimization methods using modified equations.
problem Understanding the qualitative behavior of stochastic optimization algorithms.
method Developed a class of stochastic differential equations to approximate the dynamics of stochastic optimization methods.
result Mean-square stability of the modified equation provides qualitative insights about stochastic coordinate descent.
Paper proposes a mean-field gradient descent for zero-sum games, proving convergence to Nash equilibrium.
problem Finding mixed Nash equilibria in zero-sum games with multiple players.
method Mean-field gradient descent dynamics with time-averaging, incorporating exponentially discounted gradients.
result Exponential convergence rate to mixed Nash equilibrium with respect to total variation metric.
Efficiently performs robust and sparse kernel regression.
problem Robust and sparse kernel regression.
method Sign gradient descent and early stopping.
result Sign gradient descent achieves robust and sparse kernel regression efficiently.
Gradient descent finds better constellations for GMI-based learning.
problem Nonconvexity in end-to-end GMI learning.
method Gradient descent initialized with Gray-labeled APSK constellations.
result State-of-the-art constellations in 2D and 4D provide up to 26% reach increase.
Stochastic gradient methods are the workhorse (algorithms) of large-scale optimization problems in machine learning, signal processing, and other computational sciences and engineering. This paper studies Markov chain gradient descent, a variant of stochastic gradient descent where the random samples are taken on the t…
Accelerated coordinate descent is widely used in optimization due to its cheap per-iteration cost and scalability to large-scale problems. Up to a primal-dual transformation, it is also the same as accelerated stochastic gradient descent that is one of the central methods used in machine learning. In this paper, we imp…
New algorithm improves privacy in high-dimensional machine learning models.
problem Privacy issues in learning large machine learning models.
method Differentially private greedy coordinate descent (DP-GCD) algorithm.
result Achieves logarithmic dependence on dimension for quasi-sparse solutions.
We design a randomised parallel version of Adaboost based on previous studies on parallel coordinate descent. The algorithm uses the fact that the logarithm of the exponential loss is a function with coordinate-wise Lipschitz continuous gradient, in order to define the step lengths. We provide the proof of convergence …
Training deep neural networks (DNNs) efficiently is a challenge due to the associated highly nonconvex optimization. The backpropagation (backprop) algorithm has long been the most widely used algorithm for gradient computation of parameters of DNNs and is used along with gradient descent-type algorithms for this optim…
Paper introduces methods to create fair and accurate regression models.
problem Creating fair and accurate regression models.
method Mixed-integer optimization methods, exact formulations, branch-and-bound algorithm, coordinate descent algorithm.
result Developed methods produce fair and accurate models with reduced training times.
New method selects sparse predictors in large LMMs.
problem Sparse learning for LMMs is computationally infeasible for large datasets.
method Developed an ℓ0 regularized method with coordinate descent and local search algorithms. result Method selects thousands of predictors in seconds to minutes.
In this paper, we propose a low-rank coordinate descent approach to structured semidefinite programming with diagonal constraints. The approach, which we call the Mixing method, is extremely simple to implement, has no free parameters, and typically attains an order of magnitude or better improvement in optimization pe…
Paper studies convergence of Mean-Field GDA dynamics for MNE of continuous games.
problem Finding mixed Nash equilibria in continuous games.
method Two-scale Mean-Field Gradient Descent Ascent dynamics.
result Two-scale Mean-Field GDA converges exponentially to MNE without convexity assumptions.
Accelerates MMLE using SVGD with Nesterov acceleration.
problem Maximum Marginal Likelihood Estimation optimization.
method Stein variational gradient descent with Nesterov acceleration.
result Consistently accelerates convergence across various tasks.
CD methods tackle nonconvex optimization with three terms, achieving critical points.
problem Minimizing nonconvex functions with specific structure.
method Developed randomized CD, randomly permuted CD, and accelerated CD methods.
result CD methods converge to critical points with sublinear complexity.
Paper shows how to use geometric median for robust SGD in high dimensions.
problem Robustifying SGD for high-dimensional optimization problems with gross corruption.
method Applying geometric median to only chosen blocks of coordinates at a time.
result Retains optimal breakdown point of 0.5 for smooth non-convex problems.
In 1963, Polyak proposed a simple condition that is sufficient to show a global linear convergence rate for gradient descent. This condition is a special case of the Łojasiewicz inequality proposed in the same year, and it does not require strong convexity (or even convexity). In this work, we show that this much-older…
Most neural networks are trained using first-order optimization methods, which are sensitive to the parameterization of the model. Natural gradient descent is invariant to smooth reparameterizations because it is defined in a coordinate-free way, but tractable approximations are typically defined in terms of coordinate…
New method improves scalability of SGD for large datasets.
problem High variance in stochastic gradient descent.
method Adaptive measure reduction with Carathéodory's theorem.
result Improved scalability to high-dimensional spaces.
New framework improves variational inference with Markov chain methods.
problem Challenges of minimizing KL divergence with stochastic gradient descent.
method Markov chain score ascent (MCSA) methods, including parallel MCSA (pMCSA).
result Improved theoretical and empirical performance of MCSA methods.
Proposes a new Armijo's condition for coordinate-wise functions.
problem Finding optimal step sizes in coordinate-wise optimization.
method Introduces a new Armijo's condition for functions defined on product spaces.
result Shows the advantage of the new condition over the standard Armijo's condition.
DP-SGD can update fewer coordinates while maintaining privacy.
problem How to update fewer coordinates in DP-SGD without losing optimization signal.
method TP-TopK (Two-Phase TopK DP-SGD), a two-phase method for coordinate-sparse private training.
result Private training can update fewer coordinates without losing optimization signal, scaling noise with active dimension \(k\) instead of full dimension \(d\).
New particle algorithms optimize latent variable models.
problem Optimizing latent variable models for maximum likelihood estimation.
method Identify gradient flows associated with free energy functional and discretize them to create particle-based algorithms.
result Novel particle algorithms scale to high-dimensional settings and perform well in experiments.
A new method using negative-shifted gradient descent improves overparameterized linear regression by avoiding structural limitations of negative ridge endpoints.
problem Structural limitations of negative ridge endpoints in overparameterized linear regression.
method Negative-shifted gradient descent, which avoids the pole constraint of negative ridge endpoints.
result The method improves over all admissible endpoints by a polynomial factor in risk under explicit conditions.
RGBM improves GBM efficiency with randomization.
problem Limited theoretical understanding of Gradient Boosting Machine.
method Randomization scheme to reduce search in weak-learners space.
result RGBM leads to substantial computational gains compared to GBM.
Enhances LMC for log-concave sampling, reducing computational cost.
problem High computational cost of LMC for high-dimensional problems.
method Random coordinate descent (RCD) combined with variance reduction techniques (SAGA, SVRG).
result Achieves computational cost reduction compared to classical LMC, same number of iterations as LMC.
A new method for statistical inference using SGD under φ-mixing data.
problem Valid statistical inference for time series data with general correlation.
method Proposes a mini-batch SGD estimator and associated mini-batch bootstrap procedure for φ-mixing data. result The proposed method constructs valid confidence intervals for φ-mixing data. This paper resolves the Langevin Algorithm's mixing time for log-concave distributions.
problem Resolving the mixing time of the Langevin Algorithm for log-concave sampling.
method Introducing Privacy Amplification by Iteration to analyze Rényi divergence and Optimal Transport smoothing.
result Optimal mixing bounds for the Langevin Algorithm in log-concave sampling settings.
By lifting the ReLU function into a higher dimensional space, we develop a smooth multi-convex formulation for training feed-forward deep neural networks (DNNs). This allows us to develop a block coordinate descent (BCD) training algorithm consisting of a sequence of numerically well-behaved convex optimizations. Using…
A new DP method for deep learning with faster convergence and better privacy.
problem Challenges in differentially private training of deep neural networks.
method Method of auxiliary coordinates with perturbed Taylor expansion for privacy.
result Empirically shows decent trained model quality with modest privacy budget.
A new fast algorithm solves SLOPE optimization problem.
problem Inefficient algorithms for SLOPE optimization in high dimensions.
method Combines proximal gradient descent and proximal coordinate descent steps.
result Our method outperforms competing algorithms in benchmarks.