Paper proposes efficient optimizers for large language models with fast convergence and low memory usage.
problem Designing efficient optimizers for large language models with low-memory requirements and fast convergence.
method Structured Fisher information matrix approximation and low-rank extension framework.
result New optimizers (RACS and Alice) achieve better convergence and lower memory usage than existing methods.
Efficient RNN algorithm guarantees convergence in online learning.
problem Online nonlinear regression with RNNs.
method First-order training algorithm with convergence guarantee.
result The algorithm converges to optimum network parameters.
Efficient algorithm converges to Nash equilibrium in bilinear problems with bandit feedback.
problem Learning dynamics in bilinear saddle-point problems with bandit feedback.
method Uncoupled learning algorithm combining experimental design and FTRL with a tailored regularizer.
result Last-iterate convergence rate of ildeO(T−1/4) in high probability. We derive scaling laws for optimizing neural networks in hardware.
problem Optimizing the large parameter space of neural networks in hardware.
method Analytical derivation of scaling laws for Coordinate Descent optimization.
result Convergence is exponential and scales linearly with the number of neurons.
Efficiently estimates Weingarten maps and curvatures from manifold data.
problem Estimating Weingarten maps and curvatures from manifold data.
method Statistical model for Weingarten map estimation; convergence rate analysis.
result Convergence rate of the estimator as sample size increases.
Secure and efficient distributed learning on devices with limited communication.
problem Limited communication and security in distributed on-device learning.
method Proposes SLSGD, a robust distributed optimization algorithm with efficient communication and attack tolerance.
result Stabilizes convergence and tolerates data poisoning on a small number of workers.
Recently, the decentralized optimization problem is attracting growing attention. Most existing methods are deterministic with high per-iteration cost and have a convergence rate quadratically depending on the problem condition number. Besides, the dense communication is necessary to ensure the convergence even if the …
Communication-efficient SGD algorithms, which allow nodes to perform local updates and periodically synchronize local models, are highly effective in improving the speed and scalability of distributed SGD. However, a rigorous convergence analysis and comparative study of different communication-reduction strategies rem…
Momentum improves federated learning convergence.
problem Accelerating convergence in federated learning.
method Integrates momentum into federated learning via momentum gradient descent.
result Momentum federated learning accelerates convergence compared to standard federated learning.
Study guarantees convergence of mean shift mode estimation.
problem Ensuring reliable mode estimation in KDE using mean shift.
method Utilizes Łojasiewicz inequality to prove convergence rate.
result Extends convergence guarantees to biweight kernel.
Paper tackles efficient SVM classification over decentralized networks.
problem Efficiently classifying high-dimensional data over decentralized networks.
method Convolution-based smoothing technique for nonsmooth hinge loss function, combined with an efficient ADMM algorithm.
result Provable linear convergence of the ADMM algorithm and near-optimal statistical convergence of the sparse estimator.
Study efficient convergence of RL algorithm with function approximation.
problem Convergence of actor-critic algorithm with nonlinear function approximation.
method Stochastic gradient descent ascent with adaptive proximal term, Polyak-Łojasiewicz condition.
result First efficient convergence result with rate of O(sqrt{ln(N d G^2) / N}).
Kernel learning FBSDE filter improves nonlinear filtering efficiency.
problem Nonlinear filtering problem in high-dimensional systems.
method Iterative and adaptive meshfree approach using forward backward SDE and KDE.
result Rigorous convergence analysis provided, supporting empirical results.
Efficient algorithms compute lambda quantiles for robust portfolio optimization.
problem Computing lambda quantiles efficiently and robustly.
method Λ-Newton-Bis algorithm combining Newton's method and bisection, interval analysis for multiple roots.
result Demonstrated computational efficiency and practical relevance in portfolio optimization.
APMSqueeze improves Adam for faster training with less communication.
problem Adam's inefficiency in parallel training due to gradient compression limitations.
method Error-compensated gradient compression for Adam-preconditioned Momentum SGD.
result Significantly reduces communication time per epoch without sacrificing accuracy.
AdaLoss optimizes adaptive learning rates for efficient convergence in various models.
problem Efficiently optimizing adaptive learning rates for gradient descent methods.
method AdaLoss uses loss function information to dynamically adjust step sizes.
result AdaLoss achieves linear convergence in linear regression and robust global convergence in neural networks.
The paper analyzes convergence rates of bilevel optimization algorithms and introduces a new stochastic algorithm.
problem Nonconvex-strongly-convex bilevel optimization problems in machine learning.
method Comprehensive convergence rate analysis for deterministic bilevel optimization using AID and ITD, and a novel stochastic algorithm stocBiO.
result Theoretical convergence rates for AID and ITD methods, and stocBiO's superior performance.
Efficiently tunes hyperparameters with dynamic accuracy method.
problem Optimizing machine learning hyperparameters with inexact evaluations.
method Dynamic accuracy derivative-free optimization for hyperparameter tuning.
result Demonstrates robust and efficient hyperparameter tuning compared to fixed accuracy methods.
New bounds on KPCA efficiency reveal conditions for fast convergence.
problem Lack of theoretical understanding of KPCA efficiency.
method Lower and upper bounds on KPCA efficiency involving empirical eigenvalues and new variance quantities.
result Fast convergence rates achievable for certain kernels, highlighting dataset properties.
We consider regret minimization in repeated games with non-convex loss functions. Minimizing the standard notion of regret is computationally intractable. Thus, we define a natural notion of regret which permits efficient optimization and generalizes offline guarantees for convergence to an approximate local optimum. W…
Unified framework for efficient Frank-Wolfe optimization of Dominant Set Clustering.
problem Optimizing Dominant Set Clustering with various Frank-Wolfe algorithms.
method Unified framework for pairwise, standard, and away-steps Frank-Wolfe algorithms, with explicit convergence rates.
result Explicit convergence rates for Frank-Wolfe methods in Dominant Set Clustering.
This paper analyzes and improves convergence in federated learning with biased client selection.
problem Analyzing convergence in federated learning with biased client selection.
method First convergence analysis of federated optimization for biased client selection strategies, proposing Power-of-Choice framework.
result Power-of-Choice strategies converge up to 3 times faster and give 10% higher test accuracy than random selection.
A new topology improves decentralized learning efficiency and accuracy.
problem Finding efficient decentralized learning topologies with fast consensus and low maximum degree.
method Proposed the Base-(k+1) Graph topology for decentralized learning. result The Base-(k+1) Graph enables faster convergence and better communication efficiency than the exponential graph. Paper establishes convergence rates for learning elliptic pseudo-differential operators.
problem Learning elliptic pseudo-differential operators in partial differential equations.
method Wavelet-Galerkin framework, structured infinite-dimensional regression problem, sparse estimator, matrix compression, nested-support strategy.
result Obtained convergence rates for the estimator and efficient Galerkin solver.
Efficient optimization method reduces Full AdaGrad complexity.
problem Computational challenges in Full AdaGrad for large-scale applications.
method Recursive method and streaming variant for parameter updates.
result Significant reduction in complexity and resource demands.
GADMM reduces communication costs in distributed machine learning.
problem Efficiently solving distributed machine learning problems with reduced communication costs.
method Group Alternating Direction Method of Multipliers (GADMM) framework.
result GADMM converges to the optimal solution for convex loss functions and is faster and more communication-efficient than state-of-the-art algorithms.
This paper proposes a communication-efficient distributed algorithm for high-dimensional data mining.
problem Reducing communication time and rounds in distributed data mining.
method Straggler-agnostic and bandwidth-efficient distributed primal-dual algorithm.
result Guaranteed linear convergence rate for convex problems.
New method avoids saddle points without gradients.
problem Optimizing non-convex functions efficiently.
method Zero-order derivative-free algorithm using only function evaluations.
result Converges to second-order stationary points efficiently.
DS-Sync improves distributed DNN training efficiency by 94% with minimal accuracy loss.
problem Network bottlenecks in distributed DNN training.
method Divide workers into non-overlapping groups for independent synchronization, then shuffle workers among groups iteratively.
result DS-Sync achieves up to 94% improvement in training time with minimal accuracy loss.
Enhanced ROOT-SGD optimizes stochastic optimization with diminishing stepsizes.
problem Improving statistical efficiency in stochastic optimization.
method Integrates a diminishing stepsize strategy into ROOT-SGD.
result Achieves optimal convergence rates with improved stability and precision.
Paper proposes a second-order method for faster SVI convergence.
problem Poor convergence rate of first-order SVI algorithms.
method Derives Hessian matrix and implements two numerical schemes for efficient second-order SVI.
result Proposed approach achieves faster convergence compared to first-order SVI.
Unified analysis of efficient local training methods for distributed variational inequalities.
problem Efficient distributed/federated learning for variational inequality problems.
method Unified convergence analysis of communication-efficient local training methods.
result First local gradient descent-accent algorithms with improved communication complexity.
The paper analyzes reinforcement learning methods for estimating weights and quality functions with fast convergence rates.
problem Estimating weights and quality functions in reinforcement learning with function approximation.
method The paper uses minimax methods for estimating marginal importance weights and q-functions.
result The minimax approach enables fast rates of convergence for weights and quality functions, achieving first-order efficiency.
Proposes efficient stochastic algorithms for optimizing NDCG with provable convergence guarantees.
problem Efficient and provable stochastic methods for maximizing NDCG in deep learning models.
method Formulates novel compositional optimization problems, develops efficient stochastic algorithms with provable convergence guarantees, and proposes practical strategies.
result Stochastic algorithms with provable convergence guarantees for optimizing NDCG and its top-K variant. Inference problems in graphical models can be represented as a constrained optimization of a free energy function. It is known that when the Bethe free energy is used, the fixedpoints of the belief propagation (BP) algorithm correspond to the local minima of the free energy. However BP fails to converge in many cases o…
New algorithm improves distributed SGD with random sparsification for better convergence and generalization.
problem Communication bottleneck in distributed deep learning.
method Proposes detached error feedback (DEF) algorithm to improve convergence and generalization of communication-efficient distributed SGD.
result Shows better convergence and generalization bounds than existing methods.
LD-SGD improves communication in decentralized SGD.
problem Efficiently combining local updates and decentralized communication.
method Proposes LD-SGD integrating local updates and decentralized SGD, with a convergence analysis.
result LD-SGD converges to a critical point for non-convex objectives with non-identically distributed data.
Unified parametric assumption improves convergence guarantees for nonconvex optimization.
problem Weak convergence guarantees for nonconvex optimization.
method Introducing a novel unified parametric assumption.
result Unified convergence theorem for gradient-based methods.
A new method reduces variance in PG methods for RL, improving efficiency and convergence.
problem Improving sample efficiency and convergence of policy gradient methods in reinforcement learning.
method Proposes a gradient truncation mechanism and designs TSIVR-PG method to maximize rewards and utility.
result Shows sample complexity of TSIVR-PG to find ε-stationary policy and global ε-optimal policy.
Efficient decentralized learning framework reduces communication costs.
problem Efficiently solve optimization problems in distributed learning networks.
method Censored and Quantized Generalized GADMM (CQ-GGADMM) framework.
result Achieves linear convergence rate under strong convexity assumptions.
Orthogonal initialization speeds up convergence in deep linear networks.
problem The impact of initialization on convergence speed and model performance in deep neural networks.
method Analysis of orthogonal initialization in deep linear networks, proving its superiority over Gaussian initialization.
result Orthogonal initialization speeds up convergence relative to Gaussian initialization in deep networks.
DRL-DPT improves energy efficiency in wireless networks with deterministic power control.
problem Severe performance degradation in traditional ICIC schemes with complex interference patterns.
method Deep Reinforcement Learning with Deterministic Policy and Target (DRL-DPT) framework.
result Consistently outperforms existing schemes in terms of energy efficiency and throughput.
We study streaming principal component analysis (PCA), that is to find, in O(dk) space, the top k eigenvectors of a d×d hidden matrix Σ with online vectors drawn from covariance matrix Σ. We provide global convergence for Oja's algorithm which is popularly used in practice but lacks t…
New analysis shows GD and SGD avoid saddle points efficiently in high dimensions.
problem Gradient descent and SGD struggle with saddle points in high-dimensional nonconvex optimization problems.
method Perturbed versions of GD and SGD analyzed for efficiency in high dimensions.
result Perturbed GD and SGD converge to second-order stationary points efficiently, avoiding saddle points.
Entropy regularization improves MFG learning efficiency and stability.
problem Improving Mean Field Game learning efficiency and stability.
method Entropy regularization applied to MFG with learning.
result Entropy regularization yields time-dependent policies and stabilizes convergence.
FA-HMC improves Bayesian federated learning with rigorous guarantees.
problem Parameter estimation and uncertainty quantification in non-iid distributed data.
method Federated Averaging stochastic Hamiltonian Monte Carlo (FA-HMC) with convergence guarantees.
result FA-HMC achieves better convergence and communication efficiency than existing methods.
New stochastic gradient descent with random search directions improves efficiency and convergence.
problem Efficiency and convergence of stochastic gradient descent methods.
method Developed a new class of stochastic gradient descent algorithms with random search directions.
result Established almost sure convergence and provided Lp rates of convergence. New bounds for agnostic learning with average smoothness.
problem Distribution-free nonparametric regression with average smoothness.
method Distribution-free uniform convergence bounds and agnostic learning algorithm.
result Distribution-free uniform convergence bounds for average-smoothness classes in the agnostic setting.