arXiv research

Momentum SGD fails to track nonstationary optima due to drift amplification.

In this paper, we study the problem of distributed multi-agent optimization over a network, where each agent possesses a local cost function that is smooth and strongly convex. The global objective is to find a common solution that minimizes the average of all cost functions. Assuming agents only have access to unbiase…

A novel decentralized algorithm improves minimax optimization in federated learning.

Gradient filters track moving parameters under noisy data and misspecification.

Many modern large-scale machine learning problems benefit from decentralized and stochastic optimization. Recent works have shown that utilizing both decentralized computing and local stochastic gradient estimates can outperform state-of-the-art centralized algorithms, in applications involving highly non-convex proble…

VRSGT algorithm reduces orthogonality constraints in decentralized optimization.

We present a framework for supervised subspace tracking, when there are two time series $x_t$ and $y_t$, one being the high-dimensional predictors and the other being the response variables and the subspace tracking needs to take into consideration of both sequences. It extends the classic online subspace tracking work…

This paper investigates trajectory tracking problem for a class of underactuated autonomous underwater vehicles (AUVs) with unknown dynamics and constrained inputs. Different from existing policy gradient methods which employ single actor-critic but cannot realize satisfactory tracking control accuracy and stable learn…

Our goal is to improve variance reducing stochastic methods through better control variates. We first propose a modification of SVRG which uses the Hessian to track gradients over time, rather than to recondition, increasing the correlation of the control variates and leading to faster theoretical convergence close to …

This paper studies a decentralized stochastic gradient tracking (DSGT) algorithm for non-convex empirical risk minimization problems over a peer-to-peer network of nodes, which is in sharp contrast to the existing DSGT only for convex problems. To ensure exact convergence and handle the variance among decentralized dat…

New framework for tracking varying bounds in time series forecasting.

New method tracks time-varying parameters in data.

Decentralized solutions to finite-sum minimization are of significant importance in many signal processing, control, and machine learning applications. In such settings, the data is distributed over a network of arbitrarily-connected nodes and raw data sharing is prohibitive often due to communication or privacy constr…

This work presents GROUSE (Grassmanian Rank-One Update Subspace Estimation), an efficient online algorithm for tracking subspaces from highly incomplete observations. GROUSE requires only basic linear algebraic manipulations at each iteration, and each subspace update can be performed in linear time in the dimension of…

Study examines CSO algorithm for 3D swarming and tracking multiple targets.

New algorithm tackles optimization with distributed constraints.

A new method for distributed optimization with noisy function evaluations.

We consider the problem of online subspace tracking of a partially observed high-dimensional data stream corrupted by noise, where we assume that the data lie in a low-dimensional linear subspace. This problem is cast as an online low-rank tensor completion problem. We propose a novel online tensor subspace tracking al…

Transformers learn chain-of-thought reasoning for longer problems, proving length generalization.

Decentralized methods to solve finite-sum minimization problems are important in many signal processing and machine learning tasks where the data is distributed over a network of nodes and raw data sharing is not permitted due to privacy and/or resource constraints. In this article, we review decentralized stochastic f…

Push-SAGA is a decentralized algorithm for directed graphs that converges linearly.

Optimizes decisions in time-varying distributions using online stochastic methods and Wasserstein distance.

The paper analyzes Adam and SGD in nonstationary optimization, revealing tradeoffs between noise and drift.

Gradient descent proves global convergence for deep networks with a single wide layer.

In this paper, we study distributed stochastic optimization to minimize a sum of smooth and strongly-convex local cost functions over a network of agents, communicating over a strongly-connected graph. Assuming that each agent has access to a stochastic first-order oracle ($\mathcal{SFO}$), we propose a novel distribut…

The paper proposes blending gradient boosted trees and neural networks for hierarchical time series forecasting.

A new algorithm for decentralized optimization over directed graphs.

Many applications in data analysis rely on the decomposition of a data matrix into a low-rank and a sparse component. Existing methods that tackle this task use the nuclear norm and L1-cost functions as convex relaxations of the rank constraint and the sparsity measure, respectively, or employ thresholding techniques. …

Masked LARk prevents cross-site tracking while training models.

Unified analysis of federated learning with compression for various data distributions.

Adaptive algorithm improves convergence rate of Langevin dynamics.

A new method reduces communication costs in decentralized optimization.

A new decentralized algorithm DESTINY solves optimization over Stiefel manifold with single communication round.

Contagions such as the spread of popular news stories, or infectious diseases, propagate in cascades over dynamic networks with unobservable topologies. However, "social signals" such as product purchase time, or blog entry timestamps are measurable, and implicitly depend on the underlying topology, making it possible …

GT-SARAH optimizes decentralized non-convex problems with recursive variance reduction.

SONATA algorithm converges to solutions of nonconvex smooth functions with KL property.

A framework for decentralized optimization using first-order methods.

Semiparametric method removes bias in functional bilevel gradient estimation.

Memory-efficient optimizers fail to track a subspace, leading to unpredictable model performance.

Unified view of LR and RP gradients explained via divergence theorem.

RL approach for target tracking with unknown dynamics and sensor control.

Improved convergence analysis for decentralized non-convex optimization.

This work uses SVM to identify track component failures in AC Track Circuits.

Reparameterizes mirror descent as gradient descent for efficient sparse learning.

New train tracks for complex homeomorphisms found.

New method uses cluster shapes to improve track finding in particle collisions.

Stochastic gradient descent (SGD) on a low-rank factorization is commonly employed to speed up matrix problems including matrix completion, subspace tracking, and SDP relaxation. In this paper, we exhibit a step size scheme for SGD on a low-rank least-squares problem, and we prove that, under broad sampling conditions,…

New algorithms reduce online learning regret by tracking gradient variation.