arXiv research

Efficient optimization method reduces Full AdaGrad complexity.

ASkotch solves large-scale KRR faster and better than existing methods.

Many structured data-fitting applications require the solution of an optimization problem involving a sum over a potentially large number of measurements. Incremental gradient algorithms offer inexpensive iterations by sampling a subset of the terms in the sum. These methods can make great progress initially, but often…

We study convergence properties of the full truncation Euler scheme for the Cox-Ingersoll-Ross process in the regime where the boundary point zero is inaccessible. Under some conditions on the model parameters (precisely, when the Feller ratio is greater than three), we establish the strong order 1/2 convergence in $L^…

Mutation improves FTRL convergence in zero-sum games.

We study the Heston-Cox-Ingersoll-Ross++ stochastic-local volatility model in the context of foreign exchange markets and propose a Monte Carlo simulation scheme which combines the full truncation Euler scheme for the stochastic volatility component and the stochastic domestic and foreign short interest rates with the …

The paper proves global existence and convergence of Möbius-invariant Willmore flow in 3-sphere.

Paper proposes an algorithm to recover full supervision from weakly labeled data.

We prove the existence of a unique global weak solution to the full bosonic string heat flow from closed Riemannian surfaces to an arbitrary target under smallness conditions on the two-form and the scalar potential. The solution is smooth with the exception of finitely many singular points. Finally, we discuss the con…

Studied SGD convergence under weak conditions.

This work studies the implicit bias of mini-batch SGD in classification.

Growth rates of geodesics on modular orbifolds are studied.

In this paper we define a new convergence called "asymptotically conic convergence" in which a smooth family of Riemannian metrics on a fixed compact manifold degenerate to a metric with isolated conic singularity. Our results are: convergence of the spectrum of the geometric Laplacians and uniform convergence of the c…

Gradient flow of elastic energy converges to elastica.

Study online multiclass classification under bandit feedback, extending previous results.

In this paper, we propose a new Recurrent Neural Network (RNN) architecture. The novelty is simple: We use diagonal recurrent matrices instead of full. This results in better test likelihood and faster convergence compared to regular full RNNs in most of our experiments. We show the benefits of using diagonal recurrent…

Analysis of SGD+M convergence rates in high dimensions with batch size considerations.

New SGDA method speeds up nonconvex minimax optimization.

We show that gradient descent on full-width linear convolutional networks of depth $L$ converges to a linear predictor related to the $\ell_{2/L}$ bridge penalty in the frequency domain. This is in contrast to linearly fully connected networks, where gradient descent converges to the hard margin linear support vector m…

Paper proposes a working set algorithm for non-convex sparse regression with provable convergence.

Adam's bias shifts from full-batch to max-margin of different norms for separable data.

Adaptive regularization methods pre-multiply a descent direction by a preconditioning matrix. Due to the large number of parameters of machine learning problems, full-matrix preconditioning methods are prohibitively expensive. We show how to modify full-matrix adaptive regularization in order to make it practical and e…

Federated learning enables a large amount of edge computing devices to jointly learn a model without data sharing. As a leading algorithm in this setting, Federated Averaging (\texttt{FedAvg}) runs Stochastic Gradient Descent (SGD) in parallel on a small subset of the total devices and averages the sequences only once …

Stochastic gradient descent is the method of choice for large-scale machine learning problems, by virtue of its light complexity per iteration. However, it lags behind its non-stochastic counterparts with respect to the convergence rate, due to high variance introduced by the stochastic updates. The popular Stochastic …

Two new algorithms recover ridge lines from point clouds with convergence guarantees.

We consider the fundamental problem in non-convex optimization of efficiently reaching a stationary point. In contrast to the convex case, in the long history of this basic problem, the only known theoretical results on first-order non-convex optimization remain to be full gradient descent that converges in $O(1/\varep…

The paper studies the convergence of elastic flows of curves into manifolds, proving smooth convergence under certain conditions.

Paper analyzes and proves convergence of a new method for solving complex PDEs.

Two new Frank-Wolfe algorithms improve convergence for constrained optimization.

Adaptive stochastic gradient methods such as AdaGrad have gained popularity in particular for training deep neural networks. The most commonly used and studied variant maintains a diagonal matrix approximation to second order information by accumulating past gradients which are used to tune the step size adaptively. In…

Drop-Muon updates only some layers, speeding up training.

The study provides guarantees for diffusion-based models under log-concave data, offering best-known convergence rates.

We propose a new algorithm for finite sum optimization which we call the curvature-aided incremental aggregated gradient (CIAG) method. Motivated by the problem of training a classifier for a d-dimensional problem, where the number of training data is $m$ and $m \gg d \gg 1$, the CIAG method seeks to accelerate increme…

Given a geodesic space (E, d), we show that full ordinal knowledge on the metric d-i.e. knowledge of the function D d : (w, x, y, z) $\rightarrow$ 1 d(w,x)$\le$d(y,z) , determines uniquely-up to a constant factor-the metric d. For a subspace En of n points of E, converging in Hausdorff distance to E, we construct a met…

The objective is to study an on-line Hidden Markov model (HMM) estimation-based Q-learning algorithm for partially observable Markov decision process (POMDP) on finite state and action sets. When the full state observation is available, Q-learning finds the optimal action-value function given the current action (Q func…

New method speeds up optimization over probability measures.

We prove gradient estimates for hypersurfaces in the hyperbolic space $\mathbb{H}^{n+1},$ expanding by negative powers of a certain class of homogeneous curvature functions. We obtain optimal gradient estimates for hypersurfaces evolving by certain powers $p>1$ of $F^{-1}$ and smooth convergence of the properly rescale…

Full-sampling (e.g., Q-learning) and pure-expectation (e.g., Expected Sarsa) algorithms are efficient and frequently used techniques in reinforcement learning. Q$(σ,λ)$ is the first approach unifies them with eligibility trace through the sampling degree $σ$. However, it is limited to the tabular case, for large-scale …

The article introduces a new convergence concept for Lorentzian spaces and applies it to generalized cones.

Despite remarkable empirical success, the training dynamics of generative adversarial networks (GAN), which involves solving a minimax game using stochastic gradients, is still poorly understood. In this work, we analyze last-iterate convergence of simultaneous gradient descent (simGD) and its variants under the assump…

The paper analyzes convergence properties of NGA and PAMe for $L_1$-norm PCA.

Due to the lack of enough generalization in the state-space, common methods in Reinforcement Learning (RL) suffer from slow learning speed especially in the early learning trials. This paper introduces a model-based method in discrete state-spaces for increasing learning speed in terms of required experience (but not r…

New algorithm shows neural networks can learn without full backpropagation.

Communication overhead is a major bottleneck hampering the scalability of distributed machine learning systems. Recently, there has been a surge of interest in using gradient compression to improve the communication efficiency of distributed neural network training. Using 1-bit quantization, signSGD with majority vote …

In this paper we aim to formally explain the phenomenon of fast convergence of SGD observed in modern machine learning. The key observation is that most modern learning architectures are over-parametrized and are trained to interpolate the data by driving the empirical loss (classification and regression) close to zero…

Adaptive gradient methods like AdaGrad are widely used in optimizing neural networks. Yet, existing convergence guarantees for adaptive gradient methods require either convexity or smoothness, and, in the smooth setting, only guarantee convergence to a stationary point. We propose an adaptive gradient method and show t…

This paper surveys aspects of the convergence and degeneration of Riemannian metrics on a given manifold M - the Cheeger-Gromov theory - and extensions thereof to Ricci curvature in place of full curvature. This theory is then applied to study a collection of different issues in mathematical aapects of General Relativi…

Paper proves minibatch SGD for GP inference converges and improves generalization.