arXiv research

This paper studies communication efficiency in federated learning by optimizing the sum-rate-distortion function for indirect multiterminal source coding.

We propose the stochastic average gradient (SAG) method for optimizing the sum of a finite number of smooth convex functions. Like stochastic gradient (SG) methods, the SAG method's iteration cost is independent of the number of terms in the sum. However, by incorporating a memory of previous gradient values the SAG me…

Paper solves outlier robust mean estimation near breakdown point.

Gradient methods converge exponentially in concave network games.

Paper closes convergence gap for SGD without replacement.

SignSVRG improves SignSGD by reducing variance, achieving similar convergence rates.

Optimizes convergence rate of stochastic proximal algorithms for composite convex problems.

New algorithms converge faster to Nash equilibrium in zero-sum games with bandit feedback.

GradaGrad adapts learning rate non-monotonically, overcoming AdaGrad's step size decrease.

It seems to be a pearl of conventional wisdom that parameter learning in deep sum-product networks is surprisingly fast compared to shallow mixture models. This paper examines the effects of overparameterization in sum-product networks on the speed of parameter optimisation. Using theoretical analysis and empirical exp…

Optimal SGD rates achieved with shuffling, covering non-convex and convex cases.

We improve private training accuracy with learning rate schedules and matrix factorizations.

This paper mixes constant sum and constant product market makers to improve their features.

Polynomial-time algorithm estimates edge density of random graphs with privacy and robustness.

In this paper, we study the sum rate maximization for successive zero-forcing dirty-paper coding (SZFDPC) with per-antenna power constraint (PAPC). Although SZFDPC is a low-complexity alternative to the optimal dirty paper coding (DPC), efficient algorithms to compute its sum rate are still open problems especially und…

Two new Frank-Wolfe algorithms improve convergence for constrained optimization.

A typical approach in estimating the learning rate of a regularized learning scheme is to bound the approximation error by the sum of the sampling error, the hypothesis error and the regularization error. Using a reproducing kernel space that satisfies the linear representer theorem brings the advantage of discarding t…

In a previous paper the authors defined the growth rate of the tunnel number of knots, an invariant that measures that asymptotic behavior of the tunnel number under connected sum. In this paper we calculate the growth rate of the tunnel number of m-small knots in terms of their bridge indices.

We describe a novel optimization method for finite sums (such as empirical risk minimization problems) building on the recently introduced SAGA method. Our method achieves an accelerated convergence rate on strongly convex smooth problems. Our method has only one parameter (a step size), and is radically simpler than o…

A new algorithm improves convergence rates for convex optimization problems.

Understanding a user's motivations provides valuable information beyond the ability to recommend items. Quite often this can be accomplished by perusing both ratings and review texts, since it is the latter where the reasoning for specific preferences is explicitly expressed. Unfortunately matrix factorization approach…

Paper proposes a mean-field gradient descent for zero-sum games, proving convergence to Nash equilibrium.

New method solves root-finding problems with faster convergence.

Paper studies fundamental limits of communication in distributed learning.

In our recent paper, we showed that in exponential family, contrastive divergence (CD) with fixed learning rate will give asymptotically consistent estimates \cite{wu2016convergence}. In this paper, we establish consistency and convergence rate of CD with annealed learning rate $η_t$. Specifically, suppose CD-$m$ gener…

New bounds on homological eigenvalues relate to Weil-Petersson length.

This study improves knowledge distillation for RNN-T models with noisy labels.

We study tensor completion in the agnostic setting. In the classical tensor completion problem, we receive $n$ entries of an unknown rank-$r$ tensor and wish to exactly complete the remaining entries. In agnostic tensor completion, we make no assumption on the rank of the unknown tensor, but attempt to predict unknown …

Unified analysis for shuffling-type gradient methods in optimization.

Paper uses SC to estimate hidden interference for WSRM.

Stochastic optimization algorithms with variance reduction have proven successful for minimizing large finite sums of functions. Unfortunately, these techniques are unable to deal with stochastic perturbations of input data, induced for example by data augmentation. In such cases, the objective is no longer a finite su…

In a previous paper Kobayashi and Rieck defined the growth rate of the tunnel number of a knot $K$, a knot invariant that measures the asymptotic behavior of the tunnel number under iterated connected sum of $K$. We denote the growth rate by $\mbox{gr}_t(K)$. In this paper we construct, for any $ε> 0$, a hyperbolic kno…

New algorithm finds approximate stationary points faster under differential privacy constraints.

Many structured data-fitting applications require the solution of an optimization problem involving a sum over a potentially large number of measurements. Incremental gradient algorithms offer inexpensive iterations by sampling a subset of the terms in the sum. These methods can make great progress initially, but often…

Sampling without replacement speeds up optimization in minimax problems.

We study Frank-Wolfe methods for nonconvex stochastic and finite-sum optimization problems. Frank-Wolfe methods (in the convex case) have gained tremendous recent interest in machine learning and optimization communities due to their projection-free property and their ability to exploit structured constraints. However,…

Optimistic Hedge achieves optimal regret bounds in two-player zero-sum games.

SMG combines shuffling and momentum for non-convex optimization.

Paper establishes a universal growth rate for smooth surrogate losses in classification.

New method finds global minima using function evaluations and kernel approximations.

Recent advances in optimization theory have shown that smooth strongly convex finite sums can be minimized faster than by treating them as a black box "batch" problem. In this work we introduce a new method in this class with a theoretical convergence rate four times faster than existing methods, for sums with sufficie…

Paper proposes a faster SPIDER-EM variant for large-scale nonconvex optimization.

In several recently proposed stochastic optimization methods (e.g. RMSProp, Adam, Adadelta), parameter updates are scaled by the inverse square roots of exponential moving averages of squared past gradients. Maintaining these per-parameter second-moment estimators requires memory equal to the number of parameters. For …

We propose an optimization method for minimizing the finite sums of smooth convex functions. Our method incorporates an accelerated gradient descent (AGD) and a stochastic variance reduction gradient (SVRG) in a mini-batch setting. Unlike SVRG, our method can be directly applied to non-strongly and strongly convex prob…

Mutation improves FTRL convergence in zero-sum games.

We study convergence rates of variational posterior distributions for nonparametric and high-dimensional inference. We formulate general conditions on prior, likelihood, and variational class that characterize the convergence rates. Under similar "prior mass and testing" conditions considered in the literature, the rat…

Min-max formulations have attracted great attention in the ML community due to the rise of deep generative models and adversarial methods, while understanding the dynamics of gradient algorithms for solving such formulations has remained a grand challenge. As a first step, we restrict to bilinear zero-sum games and giv…

In this paper, we study the classical problem of maximization of the sum of the utility of the terminal wealth and the utility of the consumption, in a case where a sudden jump in the risk-free interest rate creates incompleteness. The value function of the dual problem is proved to be solution of a BSDE and the dualit…