arXiv research

A new L-BFGS method tackles large-scale optimization with fewer evaluations.

A new optimization method handles Euclidean bounds efficiently.

New method achieves superlinear convergence rate with limited memory.

Improved L-BFGS algorithm with faster convergence and practical acceleration strategies.

Global convergence of an online (stochastic) limited memory version of the Broyden-Fletcher- Goldfarb-Shanno (BFGS) quasi-Newton method for solving optimization problems with stochastic objectives that arise in large scale machine learning is established. Lower and upper bounds on the Hessian eigenvalues of the sample …

Proposes a neural network for learning step-size policies for L-BFGS optimization.

Proposes an accelerated optimization algorithm for composite objectives.

A new optimization method improves deep RL performance.

New trust-region algorithms improve ML performance with indefinite Hessians.

Bayesian optimization outperforms other methods in nano-optical shape optimization and parameter reconstruction.

Recently several methods were proposed for sparse optimization which make careful use of second-order information [10, 28, 16, 3] to improve local convergence rates. These methods construct a composite quadratic approximation using Hessian information, optimize this approximation using a first-order method, such as coo…

Quasi-Newton methods improve deep learning optimization.

The question of how to incorporate curvature information in stochastic approximation methods is challenging. The direct application of classical quasi- Newton updating techniques for deterministic optimization leads to noisy curvature estimates that have harmful effects on the robustness of the iteration. In this paper…

In this paper, we further study the forward-backward envelope first introduced in [28] and [30] for problems whose objective is the sum of a proper closed convex function and a twice continuously differentiable possibly nonconvex function with Lipschitz continuous gradient. We derive sufficient conditions on the origin…

Paper accelerates L-BFGS with second-order info and stochastic batches.

Develops efficient quasi-Newton methods for training deep neural networks.

Improved L-BFGS for large-scale ML with progressive batching.

Modified BFGS and LBFGS++ libraries boost performance for non-parallelizable functions.

Paper improves L-BFGS for machine learning with multi-batch and distributed computing.

Asynchronous L-BFGS speeds up non-convex optimization.

We propose a new stochastic L-BFGS algorithm and prove a linear convergence rate for strongly convex and smooth functions. Our algorithm draws heavily from a recent stochastic variant of L-BFGS proposed in Byrd et al. (2014) as well as a recent approach to variance reduction for stochastic gradient descent from Johnson…

RES, a regularized stochastic version of the Broyden-Fletcher-Goldfarb-Shanno (BFGS) quasi-Newton method is proposed to solve convex optimization problems with stochastic objectives. The use of stochastic gradient descent algorithms is widespread, but the number of iterations required to approximate optimal arguments c…

MAYA learns bee foraging decisions with limited memory.

We present a system that enables rapid model experimentation for tera-scale machine learning with trillions of non-zero features, billions of training examples, and millions of parameters. Our contribution to the literature is a new method (SA L-BFGS) for changing batch L-BFGS to perform in near real-time by using stat…

A new quasi-Newton method uses cubic regularization to avoid saddle points in deep learning.

Paper explores challenges in training PINNs and loss landscape effects.

RQMC improves optimization in variational Bayes problems.

A new L-BFGS algorithm for Riemannian optimization converges fast without linesearch.

Recurrent Neural Networks (RNNs) are powerful models that achieve exceptional performance on several pattern recognition problems. However, the training of RNNs is a computationally difficult task owing to the well-known "vanishing/exploding" gradient problem. Algorithms proposed for training RNNs either exploit no (or…

Stochastic second-order methods converge fast under interpolation conditions.

A new method improves convergence in large-scale stochastic optimisation.

Picard-O improves ICA for faster, robust separation of signals.

Faster ICA for real data using Hessian approximations.

The question of how to parallelize the stochastic gradient descent (SGD) method has received much attention in the literature. In this paper, we focus instead on batch methods that use a sizeable fraction of the training set at each iteration to facilitate parallelism, and that employ second-order information. In order…

Episodic memory helps online model selection in complex environments.

KSD Descent uses KSD to sample from a target distribution efficiently.

Efficiently trains large models on limited-memory accelerators.

Proposes an efficient method for GPLVM with arbitrary kernels.

A scalable PyTorch framework for non-crossing quantile regression.

Hessian-free training has become a popular parallel second or- der optimization technique for Deep Neural Network training. This study aims at speeding up Hessian-free training, both by means of decreasing the amount of data used for training, as well as through reduction of the number of Krylov subspace solver iterati…

We extend the well-known BFGS quasi-Newton method and its memory-limited variant LBFGS to the optimization of nonsmooth convex objectives. This is done in a rigorous fashion by generalizing three components of BFGS to subdifferentials: the local quadratic model, the identification of a descent direction, and the Wolfe …

Federated PCA algorithm uses limited memory and privacy-preserving techniques.

Study asks if memory constraints affect optimal convex optimization methods.

In this paper, we propose a second order optimization method to learn models where both the dimensionality of the parameter space and the number of training samples is high. In our method, we construct on each iteration a Krylov subspace formed by the gradient and an approximation to the Hessian matrix, and then use a …

New method solves optimization problems faster than existing methods.

A new method solves sparse regularization problems efficiently and robustly.

The paper studies fully nonlinear equations on Hermitian manifolds, proving existence and interior estimates.

Paper shows intrinsic motivation boosts exploration efficiency in HRL.