arXiv research

EAGC boosts GCD by regulating gradient entanglement, improving known and novel category separability.

EDCompress optimizes energy efficiency of CNN models on edge devices.

Energy savings for DNN inference on resource-constrained devices.

A new energy-efficient pruning method for federated learning.

Accelerates coordinate descent methods for machine learning problems.

Energy-efficient DL inference for IoT devices reduces power consumption and improves performance.

DP-SGD can update fewer coordinates while maintaining privacy.

New SGD method uses adaptive sampling to converge faster in non-convex problems.

Paper introduces robust learning methods using coordinate gradient descent.

In this paper, we analyze energy-harvesting adaptive diffusion networks for a distributed estimation problem. In order to wisely manage the available energy resources, we propose a scheme where a censoring algorithm is jointly applied over the diffusion strategy. An energy-aware variation of a diffusion algorithm is us…

New method speeds up optimization over probability measures.

The paper studies steady solitons with curvature decay and proves their smoothness.

New DP-CD method outperforms DP-SGD in solving composite DP-ERM problems.

Stochastic methods with coordinate-wise adaptive stepsize (such as RMSprop and Adam) have been widely used in training deep neural networks. Despite their fast convergence, they can generalize worse than stochastic gradient descent. In this paper, by revisiting the design of Adagrad, we propose to split the network par…

Most neural networks are trained using first-order optimization methods, which are sensitive to the parameterization of the model. Natural gradient descent is invariant to smooth reparameterizations because it is defined in a coordinate-free way, but tractable approximations are typically defined in terms of coordinate…

Develops DP-SCD for stochastic coordinate descent, making it differentially private.

Current state-of-the-art discrete optimization methods struggle behind when it comes to challenging contrast-enhancing discrete energies (i.e., favoring different labels for neighboring variables). This work suggests a multiscale approach for these challenging problems. Deriving an algebraic representation allows us to…

A new sampling method, RC-LMC, reduces computational cost for high-dimensional log-concave distributions.

New stochastic gradient descent with random search directions improves efficiency and convergence.

We use differential equations based approaches to provide some {\it \textbf{physics}} insights into analyzing the dynamics of popular optimization algorithms in machine learning. In particular, we study gradient descent, proximal gradient descent, coordinate gradient descent, proximal coordinate gradient, and Newton's …

We consider (locally) energy finite coordinates associated with a strongly local regular Dirichlet form on a metric measure space. We give coordinate formulas for substitutes of tangent spaces, for gradient and divergence operators and for the infinitesimal generator. As examples we discuss Euclidean spaces, Riemannian…

We propose accelerated randomized coordinate descent algorithms for stochastic optimization and online learning. Our algorithms have significantly less per-iteration complexity than the known accelerated gradient algorithms. The proposed algorithms for online learning have better regret performance than the known rando…

New method reduces variance in random coordinate descent for Langevin Monte Carlo.

Poisson variational autoencoders introduce a metabolic cost term that penalizes high baseline activity.

New method reveals insights about stochastic optimization methods using modified equations.

New algorithm improves privacy in high-dimensional machine learning models.

Designing energy-efficient networks is of critical importance for enabling state-of-the-art deep learning in mobile and edge settings where the computation and energy budgets are highly limited. Recently, Liu et al. (2019) framed the search of efficient neural architectures into a continuous splitting process: it itera…

This work improves communication efficiency in federated learning over wireless networks by optimizing energy consumption.

Develops new algorithms for solving root-finding problems in large-scale settings.

We design a randomised parallel version of Adaboost based on previous studies on parallel coordinate descent. The algorithm uses the fact that the logarithm of the exponential loss is a function with coordinate-wise Lipschitz continuous gradient, in order to define the step lengths. We provide the proof of convergence …

Formula for Laplace-Beltrami on orthogonal group in Euclidean coords.

Stochastic Gradient Descent (SGD) has become popular for solving large scale supervised machine learning optimization problems such as SVM, due to their strong theoretical guarantees. While the closely related Dual Coordinate Ascent (DCA) method has been implemented in various software packages, it has so far lacked go…

Differentially private random block coordinate descent improves utility in machine learning.

Let $z=(x,y)$ be coordinates for the product space $\mathbb{R}^{m_1}\times \mathbb{R}^{m_2}$. Let $f:\mathbb{R}^{m_1}\times \mathbb{R}^{m_2}\rightarrow \mathbb{R}$ be a $C^1$ function, and $\nabla f=(\partial _xf,\partial _yf)$ its gradient. Fix $0<α<1$. For a point $(x,y) \in \mathbb{R}^{m_1}\times \mathbb{R}^{m_2}$, …

Stochastic dual coordinate ascent (SDCA) is an effective technique for solving regularized loss minimization problems in machine learning. This paper considers an extension of SDCA under the mini-batch setting that is often used in practice. Our main contribution is to introduce an accelerated mini-batch version of SDC…

Gradient-based optimization improves variational empirical Bayes regression.

Nonconvex optimization problems arise in different research fields and arouse lots of attention in signal processing, statistics and machine learning. In this work, we explore the accelerated proximal gradient method and some of its variants which have been shown to converge under nonconvex context recently. We show th…

In 1963, Polyak proposed a simple condition that is sufficient to show a global linear convergence rate for gradient descent. This condition is a special case of the Łojasiewicz inequality proposed in the same year, and it does not require strong convexity (or even convexity). In this work, we show that this much-older…

A new sampling method reduces computational cost for high-dimensional log-concave distributions.

This paper aims at achieving a "good" estimator for the gradient of a function on a high-dimensional space. Often such functions are not sensitive in all coordinates and the gradient of the function is almost sparse. We propose a method for gradient estimation that combines ideas from Spall's Simultaneous Perturbation …

Novel coordinate descent (CD) methods are proposed for minimizing nonconvex functions consisting of three terms: (i) a continuously differentiable term, (ii) a simple convex term, and (iii) a concave and continuous term. First, by extending randomized CD to nonsmooth nonconvex settings, we develop a coordinate subgradi…

In this work, a unified framework for gradient-free Multidimensional Scaling (MDS) based on Coordinate Search (CS) is proposed. This family of algorithms is an instance of General Pattern Search (GPS) methods which avoid the explicit computation of derivatives but instead evaluate the objective function while searching…

Training deep neural networks (DNNs) efficiently is a challenge due to the associated highly nonconvex optimization. The backpropagation (backprop) algorithm has long been the most widely used algorithm for gradient computation of parameters of DNNs and is used along with gradient descent-type algorithms for this optim…

Enhances LMC for log-concave sampling, reducing computational cost.

A new method for optimizing functions without gradients, improving efficiency and convergence.

Paper optimizes approximating high-dimensional diffusions by independent coordinates.

Accelerated coordinate descent is widely used in optimization due to its cheap per-iteration cost and scalability to large-scale problems. Up to a primal-dual transformation, it is also the same as accelerated stochastic gradient descent that is one of the central methods used in machine learning. In this paper, we imp…

New particle algorithms optimize latent variable models.