Improved complexity for machine learning optimization methods.
problem Optimizing over-parametrized models in machine learning.
method Stochastic conditional gradient methods with interpolation-like conditions.
result Improved oracle complexities for finding optimal solutions.
Paper proposes a pre-conditioning technique to speed up gradient-descent convergence in distributed linear least-squares problems.
problem Expediting convergence of gradient-descent method for ill-conditioned distributed linear least-squares problems.
method Iterative pre-conditioning technique to improve convergence rate of gradient-descent method.
result Pre-conditioned gradient-descent achieves superlinear convergence for unique solutions and improved linear convergence otherwise.
We study a hybrid conditional gradient - smoothing algorithm (HCGS) for solving composite convex optimization problems which contain several terms over a bounded set. Examples of these include regularization problems with several norms as penalties and a norm constraint. HCGS extends conditional gradient methods to cas…
New PG methods tackle nonconvex optimization with auto-conditioned stepsizes.
problem Optimizing nonconvex functions over convex sets.
method Auto-conditioned projected gradient (AC-PG) methods and stochastic variants.
result Achieved optimal iteration complexity for finding approximate stationary points.
Paper proposes a pre-conditioning method to speed up gradient descent in multi-agent optimization.
problem Speed up convergence of gradient descent in multi-agent optimization problems.
method Iterative pre-conditioning approach to mitigate the effect of problem conditioning.
result Significant improvement in convergence speed of gradient descent method.
Paper proposes a policy gradient method for confounded POMDPs.
problem Estimating policy gradients for confounded POMDPs with continuous state and observation spaces.
method Developed a novel identification result to estimate policy gradients using offline data, solved conditional moment restrictions, and applied min-max learning with function approximation.
result Showed global convergence of the proposed algorithm in finding the optimal policy.
In this paper, we propose a novel technique to implement stochastic gradient methods, which are beneficial for learning from large datasets, through accelerated stochastic dynamics. A stochastic gradient method is based on mini-batch learning for reducing the computational cost when the amount of data is large. The sto…
New analysis reveals batch size effects on stochastic conditional gradient methods.
problem Understanding the role of batch size in stochastic conditional gradient methods.
method Deriving a new analysis focusing on momentum-based stochastic conditional gradient algorithms (e.g., Scion).
result Increasing batch size initially improves optimization accuracy but can degrade performance beyond a critical threshold.
Generalizes smoothness conditions for optimization methods.
problem Optimization under non-uniform smoothness conditions.
method Develops a new analysis technique for bounding gradients.
result Obtains convergence rates for gradient descent and Nesterov's method.
In 1963, Polyak proposed a simple condition that is sufficient to show a global linear convergence rate for gradient descent. This condition is a special case of the Łojasiewicz inequality proposed in the same year, and it does not require strong convexity (or even convexity). In this work, we show that this much-older…
New gradient methods solve multiscale optimization problems efficiently.
problem Minimizing functions with multiple non-interacting smooth, strongly convex components.
method Big-Step-Little-Step interleaving of standard methods.
result Complexity bound scales as product of square-roots of condition numbers of components, improving on accelerated gradient methods.
We apply stochastic average gradient (SAG) algorithms for training conditional random fields (CRFs). We describe a practical implementation that uses structure in the CRF gradient to reduce the memory requirement of this linearly-convergent stochastic gradient method, propose a non-uniform sampling scheme that substant…
Policy gradients methods apply to complex, poorly understood, control problems by performing stochastic gradient descent over a parameterized class of polices. Unfortunately, even for simple control problems solvable by standard dynamic programming techniques, policy gradient algorithms face non-convex optimization pro…
Stochastic gradient method converges as fast as deterministic for overparametrized models.
problem Convergence rate of stochastic gradient methods in overparametrized models.
method Proposes a regularity condition enabling fast convergence of SGD.
result Stochastic gradient method achieves the same convergence rate as deterministic gradient method.
Large deviations theory applied to policy gradient methods.
problem Understanding convergence of policy gradient methods in reinforcement learning.
method Large deviation rate function and contraction principle from large deviations theory.
result Convergence properties of policy gradient methods can be extended to various policy parametrizations.
Two new methods solve large-scale stochastic convex problems with linear constraints.
problem Solving large-scale stochastic convex optimization problems with many linear constraints.
method Conditional gradient-based methods that process only a subset of constraints at each iteration.
result Rigorous convergence guarantees for the proposed methods.
Gradient descent benefits from tangent kernel advantages under specific conditions.
problem Comparing gradient descent with tangent kernel methods in learning.
method Analysis of gradient descent and tangent kernel methods under different conditions.
result Gradient descent can achieve small error only if tangent kernel methods have a non-trivial advantage, but this advantage can be very small.
Given a convex optimization problem and its dual, there are many possible first-order algorithms. In this paper, we show the equivalence between mirror descent algorithms and algorithms generalizing the conditional gradient method. This is done through convex duality, and implies notably that for certain problems, such…
Paper proves SHB convergence with biased gradients and approximate step sizes.
problem Establishing convergence of SHB with biased gradients and approximate step sizes.
method Generalizes SHB convergence conditions for biased gradients, approximate step sizes, and block updating.
result Proves convergence of SHB with new conditions for biased gradients and approximate step sizes.
Paper proposes a new method for robust modal regression.
problem Estimating the global mode of conditional density functions robustly.
method Directly approximates the gradient of modal regression risk using kernelized and neural-network-based log-density derivative estimators.
result Proposed methods achieve superior performance on various datasets.
New method tackles nonconvex-nonconcave problems with local KL condition.
problem Nonconvex-nonconcave minimax problems under varying KL conditions.
method Inexact proximal gradient method for KL-structured subproblems.
result Complexity guarantees for approximate stationary points.
New unbiased gradient estimators for complex optimization problems.
problem Unbiased and variance-limited gradient estimation for conditional stochastic optimization.
method Developed multilevel Monte Carlo gradient estimators for conditional stochastic optimization problems.
result Unbiased and finite variance gradient estimators for conditional stochastic optimization problems.
Spectral gradient methods outperform Euclidean in certain deep learning scenarios.
problem When do spectral gradient updates outperform Euclidean in deep learning?
method Layerwise condition comparing squared nuclear-to-Frobenius ratio to stable rank of activations.
result Spectral updates can be more effective than Euclidean in deep networks and transformers.
Optimal gradient quantization reduces communication costs in distributed deep learning.
problem High communication costs in distributed training of deep neural networks.
method Deduced optimal gradient quantization conditions for binary and multi-level quantization, developed novel schemes for dynamic quantization levels.
result Demonstrated superior performance of proposed quantization schemes on CIFAR and ImageNet datasets.
Unified algorithm for stochastic optimization with time-varying momentum converges under general conditions.
problem Optimizing functions with time-varying gradients and biases.
method Unified algorithm using a time-varying momentum term.
result Convergence of the unified algorithm under general conditions.
A new biased gradient descent method for conditional stochastic optimization.
problem Challenges in constructing unbiased gradient estimators for conditional stochastic optimization.
method Proposes a biased stochastic gradient descent (BSGD) algorithm and analyzes its sample complexities.
result Establishes sample complexities of BSGD for various objectives and shows that BSpiderBoost matches the lower bound complexity.
Robust CG methods avoid data corruption and solve structured statistical estimation problems.
problem Data corruption and heavy-tailed data in structured statistical estimation.
method Robustification of Conditional Gradient (CG) type methods using Huber's corruption model and robust mean gradient estimation.
result Robust CG methods converge linearly with correct sample complexity, even for high-dimensional problems.
Conditional gradients constitute a class of projection-free first-order algorithms for smooth convex optimization. As such, they are frequently used in solving smooth convex optimization problems over polytopes, for which the computational cost of orthogonal projections would be prohibitive. However, they do not enjoy …
Gradient-free method improves predictive accuracy for probabilistic models.
problem Balancing computational efficiency and robust predictive performance in deep learning.
method CAVI-CMN, a gradient-free variational method for conditional mixture networks.
result CAVI-CMN achieves competitive and often superior predictive accuracy compared to MLE with backpropagation.
Enhances neural network solvers for PDEs with complex boundary conditions.
problem Challenges in solving PDEs with high accuracy and complex boundary conditions.
method Integrates natural gradient optimization with numerical time-stepping schemes to enforce Dirichlet boundary conditions.
result Superior accuracy and computational efficiency of the proposed methods for solving PDEs.
Proposes a new method for posterior sampling using MMD with negative distance kernel.
problem Posterior sampling and conditional generative modeling.
method Approximates joint distribution using discrete Wasserstein gradient flows of MMD with negative distance kernel.
result Establishes an error bound for posterior distributions and proves the method is a Wasserstein gradient flow.
Paper presents a rank-1 approximation method for natural policy gradients in deep RL.
problem Computing natural gradients requires inverting the Fisher Information Matrix, which is computationally expensive.
method Develops a rank-1 approximation to the inverse Fisher Information Matrix for efficient natural policy optimization.
result The rank-1 approximation converges faster and has similar sample complexity to stochastic policy gradient methods.
This work analyzes the gap between off-policy and on-policy policy gradient methods and provides conditions to reduce this gap.
problem The gap between off-policy and on-policy policy gradient methods and conditions to reduce it.
method Theoretical analysis and empirical evidence of conditions to reduce the on-off gap.
result Conditions to reduce the on-off gap between off-policy and on-policy policy gradient methods.
Recent studies have shown that proximal gradient (PG) method and accelerated gradient method (APG) with restarting can enjoy a linear convergence under a weaker condition than strong convexity, namely a quadratic growth condition (QGC). However, the faster convergence of restarting APG method relies on the potentially …
Study on Nesterov's method in stochastic settings, revealing divergence under certain conditions.
problem Understanding Nesterov's method in stochastic settings, especially finite-sum.
method Analysis of Nesterov's accelerated gradient method in stochastic and finite-sum settings.
result Nesterov's method may diverge in finite-sum settings without additional conditions.
New method efficiently computes gradients for stochastic differential equations.
problem Computing gradients for stochastic differential equations efficiently.
method Generalized adjoint sensitivity method to stochastic differential equations.
result Time-efficient and memory-efficient computation of gradients with high-order solvers.
NGBoost boosts probabilistic predictions using natural gradients.
problem Uncertainty estimation in probabilistic predictions.
method Gradient boosting for probabilistic regression with natural gradient correction.
result NGBoost outperforms existing methods for probabilistic prediction.
ScaledGD improves gradient descent for ill-conditioned low-rank matrix estimation.
problem Efficiently solving ill-conditioned low-rank matrix estimation problems.
method Scaled Gradient Descent (ScaledGD) with adaptive pre-conditioners.
result Linear convergence rate independent of condition number, low per-iteration cost.
New methods accelerate gradient descent for convex and strongly convex functions.
problem Improving convergence rates of gradient-based optimization methods.
method Formulated two classes of first-order algorithms with Lyapunov analyses and Hamiltonian assisted gradient method.
result Achieved accelerated convergence rates matching Nesterov's methods in strongly and general convex settings.
A new gradient boosting method improves interpretability of probabilistic models.
problem Learning interpretable yet accurate probabilistic models with limited rule complexity.
method A new objective function that measures the angle between risk gradient and condition output vector projection.
result Significantly improves comprehensibility/accuracy trade-off of fitted ensemble.
A new algorithm improves convergence rates for convex optimization problems.
problem Convex optimization problems with finite-sum structure.
method Nesterov Accelerated Shuffling Gradient (NASG) integrating Nesterov's acceleration with different shuffling schemes.
result Improved convergence rate of O(1/T) for unified shuffling schemes.
New scalable methods for unbalanced optimal transport improve efficiency and applicability.
problem Scalable algorithms for unbalanced optimal transport remain underexplored.
method Analysis of semi-dual formulation and adaptive gradient methods.
result SGD methods achieve a convergence rate of O(n/εT) for large-scale applications.
Treeffuser predicts tabular data distributions using gradient-boosted trees.
problem Probabilistic prediction with flexible, non-parametric models.
method Gradient-boosted trees for score estimation in conditional diffusion model.
result Treeffuser outperforms existing methods in probabilistic prediction tasks.
We consider the problem of minimizing a Lipschitz differentiable function over a class of sparse symmetric sets that has wide applications in engineering and science. For this problem, it is known that any accumulation point of the classical projected gradient (PG) method with a constant stepsize 1/L satisfies the $L…
In this paper, we provide an overview of first-order and second-order variants of the gradient descent method that are commonly used in machine learning. We propose a general framework in which 6 of these variants can be interpreted as different instances of the same approach. They are the vanilla gradient descent, the…
New method solves complex constrained optimization problems.
problem Constrained nonconvex-nonconcave minimax optimization problems.
method Inexact proximal gradient method using sequential convex programming.
result Established complexity guarantees for approximate stationary points.
Gradient methods work well on overparameterized diagonal linear networks.
problem Understanding why gradient-based methods work well in overparameterized models.
method Study of Deep Diagonal Linear Networks with gradient flow analysis.
result Gradient flow on layer parameters induces a mirror-flow dynamic in the effective parameter space, leading to explicit convergence guarantees.
The objectives of this technical report is to provide additional results on the generalized conditional gradient methods introduced by Bredies et al. [BLM05]. Indeed , when the objective function is smooth, we provide a novel certificate of optimality and we show that the algorithm has a linear convergence rate. Applic…