Improved complexity for machine learning optimization methods.
problem Optimizing over-parametrized models in machine learning.
method Stochastic conditional gradient methods with interpolation-like conditions.
result Improved oracle complexities for finding optimal solutions.
The paper investigates geometrical aspects of static spacetime with almost gradient Ricci solitons.
problem Geometrical properties of static spacetime with almost gradient Ricci solitons.
method Analyzing conditions and properties of static spacetime with almost gradient Ricci solitons.
result Conditions and properties of static spacetime with almost gradient Ricci solitons are determined.
Unified probabilistic gradient boosting for entire conditional distribution modeling.
problem Creating accurate probabilistic forecasts from regression tasks.
method Unified probabilistic gradient boosting framework using XGBoost and LightGBM, modeling conditional moments or CDF via Normalizing Flows.
result Achieves state-of-the-art forecast accuracy.
The paper establishes gradient estimates for harmonic and heat equation solutions on manifolds with boundary.
problem Gradient estimates for harmonic and heat equation solutions on manifolds with boundary.
method Yau and Souplet-Zhang type gradient estimates for harmonic and heat equation solutions under Dirichlet boundary condition.
result Established gradient estimates for harmonic and heat equation solutions on manifolds with boundary.
Paper proposes a pre-conditioning technique to speed up gradient-descent convergence in distributed linear least-squares problems.
problem Expediting convergence of gradient-descent method for ill-conditioned distributed linear least-squares problems.
method Iterative pre-conditioning technique to improve convergence rate of gradient-descent method.
result Pre-conditioned gradient-descent achieves superlinear convergence for unique solutions and improved linear convergence otherwise.
Characterizes gradient Yamabe solitons with specific conditions.
problem Understanding properties of gradient Yamabe solitons.
method Proved conditions leading to constant scalar curvature, subharmonicity, and harmonic potential.
result Gradient Yamabe solitons under certain conditions are of constant scalar curvature.
New unbiased gradient estimators for complex optimization problems.
problem Unbiased and variance-limited gradient estimation for conditional stochastic optimization.
method Developed multilevel Monte Carlo gradient estimators for conditional stochastic optimization problems.
result Unbiased and finite variance gradient estimators for conditional stochastic optimization problems.
The study finds a lower bound for the diameter of gradient ρ-Einstein solitons.
problem Estimating the diameter of gradient ρ-Einstein solitons.
method Using mathematical conditions and properties of solitons to derive a lower bound.
result A lower bound for the diameter of gradient ρ-Einstein solitons is established.
Adaptive step-size improves optimization in complex geometries.
problem Optimizing functions with non-Euclidean geometries.
method Adaptive step-size strategy for optimization algorithms.
result Guaranteed convergence for Adaptive Conditional Gradient Descent.
New bounds show BBVI's gradient variance matches SGD conditions, improving parameterization efficiency.
problem Understanding and improving the convergence of black-box variational inference (BBVI).
method Showed BBVI satisfies matching gradient variance bounds corresponding to the ABC condition for smooth and quadratically-growing log-likelihoods.
result Proven BBVI's gradient variance matches SGD conditions, with superior dimensional dependence for mean-field parameterization.
Paper investigates conditions for independence of weak gradients on metric spaces.
problem Dependence of weak gradients on p in arbitrary metric measure spaces. method Investigates the Bounded Interpolation Property to ensure independence of weak gradients.
result Bounded Interpolation Property guarantees independence of weak gradients.
We study a hybrid conditional gradient - smoothing algorithm (HCGS) for solving composite convex optimization problems which contain several terms over a bounded set. Examples of these include regularization problems with several norms as penalties and a norm constraint. HCGS extends conditional gradient methods to cas…
Generalizes smoothness conditions for optimization methods.
problem Optimization under non-uniform smoothness conditions.
method Develops a new analysis technique for bounding gradients.
result Obtains convergence rates for gradient descent and Nesterov's method.
Gradient estimate for harmonic functions with boundary condition proved.
problem Proving gradient estimates for harmonic functions with boundary conditions.
method Using weighted f-harmonic functions and infinite dimensional Bakry-Emery Ricci tensor. result Gradient estimates for positive f-harmonic functions with Dirichlet boundary condition. The paper estimates gradients on graphs under specific conditions and applies these estimates to heat equations.
problem Estimating gradients on graphs with the CDψ(n,−K) condition. method Investigates gradient estimates for positive solutions of heat equations and a heat-type equation.
result Derives heat kernel bounds and Harnack inequalities using gradient estimates.
Paper proposes a policy gradient method for confounded POMDPs.
problem Estimating policy gradients for confounded POMDPs with continuous state and observation spaces.
method Developed a novel identification result to estimate policy gradients using offline data, solved conditional moment restrictions, and applied min-max learning with function approximation.
result Showed global convergence of the proposed algorithm in finding the optimal policy.
New analysis reveals batch size effects on stochastic conditional gradient methods.
problem Understanding the role of batch size in stochastic conditional gradient methods.
method Deriving a new analysis focusing on momentum-based stochastic conditional gradient algorithms (e.g., Scion).
result Increasing batch size initially improves optimization accuracy but can degrade performance beyond a critical threshold.
Policy gradients methods apply to complex, poorly understood, control problems by performing stochastic gradient descent over a parameterized class of polices. Unfortunately, even for simple control problems solvable by standard dynamic programming techniques, policy gradient algorithms face non-convex optimization pro…
SGD and stochastic gradient descent converge at optimal rates for certain non-convex functions.
problem Optimal convergence rates for non-convex functions under gradient noise.
method Geometric interpretation of the PL-condition to analyze convergence rates.
result Convergence rates of SGD and stochastic gradient descent match those of strongly convex quadratics.
This paper classifies solitons under specific tensor conditions.
problem Classifying solitons under vanishing conditions on the Weyl, Cotton, and Cao-Chen tensors.
method Analyzing complete conformal gradient solitons and using tensor conditions.
result Classification of complete nontrivial locally conformally flat conformal gradient solitons.
The paper proves conditions for Einstein solitons to split into line and manifold.
problem Conditions for Einstein solitons to split into line and manifold.
method Weighted Laplacian comparison of distance function and bounded integral condition on Ricci curvature.
result Gradient ρ-Einstein solitons split off a line isometrically under certain conditions.
Gradient descent with logistic loss can interpolate deep networks with smoothed ReLU activations under certain conditions.
problem Conditions for gradient descent to drive logistic loss to zero in deep networks with smoothed ReLU activations.
method Gradient descent applied to fixed-width deep networks with smoothed ReLU approximations (e.g., Swish, Huberized ReLU).
result Gradient descent can drive logistic loss to zero under specific conditions, providing bounds on convergence rate.
The article characterizes gradient ρ-Einstein solitons under specific conditions.
problem Characterizing gradient ρ-Einstein solitons with certain properties.
method Analyzing solitons with vector fields of bounded norm, finite weighted Dirichlet integral, and specific Ricci curvature restrictions.
result Non-trivial complete gradient ρ-Einstein solitons with finite weighted Dirichlet integral and certain Ricci curvature restrictions are of constant scalar curvature and steady.
New PG methods tackle nonconvex optimization with auto-conditioned stepsizes.
problem Optimizing nonconvex functions over convex sets.
method Auto-conditioned projected gradient (AC-PG) methods and stochastic variants.
result Achieved optimal iteration complexity for finding approximate stationary points.
Paper proves SHB convergence with biased gradients and approximate step sizes.
problem Establishing convergence of SHB with biased gradients and approximate step sizes.
method Generalizes SHB convergence conditions for biased gradients, approximate step sizes, and block updating.
result Proves convergence of SHB with new conditions for biased gradients and approximate step sizes.
Paper proposes a pre-conditioning method to speed up gradient descent in multi-agent optimization.
problem Speed up convergence of gradient descent in multi-agent optimization problems.
method Iterative pre-conditioning approach to mitigate the effect of problem conditioning.
result Significant improvement in convergence speed of gradient descent method.
This paper analyzes adaptive gradient algorithms for better performance in ill-conditioned problems.
problem Poor performance of standard stochastic gradient algorithms in ill-conditioned problems.
method Non-asymptotic analysis of adaptive gradient algorithms (Adagrad and Stochastic Newton) for strongly convex objectives.
result Theoretical analysis and adaptation to practical applications like linear regression and regularized GLM.
Large deviations theory applied to policy gradient methods.
problem Understanding convergence of policy gradient methods in reinforcement learning.
method Large deviation rate function and contraction principle from large deviations theory.
result Convergence properties of policy gradient methods can be extended to various policy parametrizations.
Paper shows k-Yamabe solitons have constant curvature under certain conditions.
problem Understanding the properties of k-Yamabe solitons.
method Analyzing the curvature and gradient conditions for k-Yamabe solitons.
result Compact k-Yamabe solitons have constant σk-curvature under certain conditions. Gradient descent benefits from tangent kernel advantages under specific conditions.
problem Comparing gradient descent with tangent kernel methods in learning.
method Analysis of gradient descent and tangent kernel methods under different conditions.
result Gradient descent can achieve small error only if tangent kernel methods have a non-trivial advantage, but this advantage can be very small.
Conditions for trivial gradient hyperbolic Ricci and Yamabe solitons to be Einstein or constant scalar curvature.
problem Characterizing conditions for gradient hyperbolic Ricci and Yamabe solitons to be trivial.
method Analyzing Lie derivatives and divergence conditions.
result Conditions for compact gradient hyperbolic Yamabe solitons to be trivial, leading to constant scalar curvature.
The paper examines how gradient descent stabilizes low-rank matrix factorization in noisy conditions.
problem Stability of low-rank implicit regularization in perturbed deep matrix factorization.
method Derives spectral conditions for gradient descent to exhibit a low-rank phase in noiseless settings and analyzes perturbed dynamics.
result Gradient descent converges to a low-rank solution under perturbation, with explicit dependence on perturbation size.
Paper classifies Einstein-type manifolds with parallel Ricci tensor.
problem Classifying Einstein-type manifolds with specific curvature properties.
method Deduced Bochner-type identity and used it to show rigidity results.
result Found conditions for classifying Einstein-type manifolds with parallel Ricci tensor.
Stochastic gradient method converges as fast as deterministic for overparametrized models.
problem Convergence rate of stochastic gradient methods in overparametrized models.
method Proposes a regularity condition enabling fast convergence of SGD.
result Stochastic gradient method achieves the same convergence rate as deterministic gradient method.
The paper proves gradient estimates for nonlinear parabolic equations on smooth metric measure spaces.
problem Proving gradient estimates for nonlinear parabolic equations on smooth metric measure spaces.
method Using Souplet-Zhang type estimates and properties of Bakry-Emery Ricci tensor and weighted mean curvature.
result Gradient estimates for nonlinear parabolic equations on smooth metric measure spaces with Dirichlet boundary condition.
In 1963, Polyak proposed a simple condition that is sufficient to show a global linear convergence rate for gradient descent. This condition is a special case of the Łojasiewicz inequality proposed in the same year, and it does not require strong convexity (or even convexity). In this work, we show that this much-older…
Studied SGD convergence under weak conditions.
problem Convergence of SGD in nonconvex optimization.
method Analyzed biased nonconvex SGD under mild conditions.
result Provided convergence rates and complexities.
In this paper, we propose a novel technique to implement stochastic gradient methods, which are beneficial for learning from large datasets, through accelerated stochastic dynamics. A stochastic gradient method is based on mini-batch learning for reducing the computational cost when the amount of data is large. The sto…
The communication of gradients is costly for training deep neural networks with multiple devices in computer vision applications. In particular, the growing size of deep learning models leads to higher communication overheads that defy the ideal linear training speedup regarding the number of devices. Gradient quantiza…
In this work we introduce a conditional accelerated lazy stochastic gradient descent algorithm with optimal number of calls to a stochastic first-order oracle and convergence rate O(ε21) improving over the projection-free, Online Frank-Wolfe based stochastic gradient descent of Hazan an…
Spectral gradient methods outperform Euclidean in certain deep learning scenarios.
problem When do spectral gradient updates outperform Euclidean in deep learning?
method Layerwise condition comparing squared nuclear-to-Frobenius ratio to stable rank of activations.
result Spectral updates can be more effective than Euclidean in deep networks and transformers.
ScaledGD improves gradient descent for ill-conditioned low-rank matrix estimation.
problem Efficiently solving ill-conditioned low-rank matrix estimation problems.
method Scaled Gradient Descent (ScaledGD) with adaptive pre-conditioners.
result Linear convergence rate independent of condition number, low per-iteration cost.
The paper examines gradient ρ-Einstein solitons on specific manifolds and spacetimes.
problem Characterizing gradient ρ-Einstein solitons on doubly warped product manifolds.
method Analyzing necessary and sufficient conditions for doubly warped product manifolds to be gradient ρ-Einstein solitons, applying results to specific spacetime models.
result No 3-dimensional essentially conformally symmetric gradient ρ-Einstein soliton exists.
New approach proves convergence of SA and SGD with weaker conditions.
problem Proving convergence of SA and SGD with relaxed noise conditions.
method Introduces GSLLN to decouple function and noise properties.
result Derives sufficient conditions for convergence of SA and SGD.
New gradient methods solve multiscale optimization problems efficiently.
problem Minimizing functions with multiple non-interacting smooth, strongly convex components.
method Big-Step-Little-Step interleaving of standard methods.
result Complexity bound scales as product of square-roots of condition numbers of components, improving on accelerated gradient methods.
Unified algorithm for stochastic optimization with time-varying momentum converges under general conditions.
problem Optimizing functions with time-varying gradients and biases.
method Unified algorithm using a time-varying momentum term.
result Convergence of the unified algorithm under general conditions.
Two new methods solve large-scale stochastic convex problems with linear constraints.
problem Solving large-scale stochastic convex optimization problems with many linear constraints.
method Conditional gradient-based methods that process only a subset of constraints at each iteration.
result Rigorous convergence guarantees for the proposed methods.
We apply stochastic average gradient (SAG) algorithms for training conditional random fields (CRFs). We describe a practical implementation that uses structure in the CRF gradient to reduce the memory requirement of this linearly-convergent stochastic gradient method, propose a non-uniform sampling scheme that substant…