arXiv research

Paper proposes an online covariance estimator for sketched Newton methods.

Paper proposes an efficient online Newton method with Nesterov's acceleration for streaming data.

New quasi-Newton method guarantees global superlinear convergence.

The second order method as Newton Step is a suitable technique in Online Learning to guarantee regret bound. The large data is a challenge in Newton method to store second order matrices as hessian. In this paper, we have proposed an modified online Newton step that store first and second order matrices of dimension m …

New adaptive first-order methods improve on quasi-Newton variants.

Improved complexity for smooth nonconvex optimization using quasi-Newton methods.

Paper proposes a quasi-Newton method for nonlinear equations with global convergence guarantees.

This paper introduces online algorithms to estimate robust geometric median in large data streams.

RONM method reduces regret in stochastic convex bandits with decreasing noise.

New method solves convex optimization faster than NAG.

Global convergence of an online (stochastic) limited memory version of the Broyden-Fletcher- Goldfarb-Shanno (BFGS) quasi-Newton method for solving optimization problems with stochastic objectives that arise in large scale machine learning is established. Lower and upper bounds on the Hessian eigenvalues of the sample …

An online learning framework for survival analysis with real-time adaptation.

Paper develops efficient methods for estimating Hessian inverses in stochastic optimization.

Efficient algorithm for zeroth-order bandit convex optimization with bounds on regret.

Online learning with limited information feedback (bandit) tries to solve the problem where an online learner receives partial feedback information from the environment in the course of learning. Under this setting, Flaxman et al.[8] extended Zinkevich's classical Online Gradient Descent (OGD) algorithm [29] by proposi…

Paper analyzes regret bounds for unconstrained online optimization.

A new sequential method estimates Poisson means in streaming data, achieving optimality and efficiency.

We describe a framework for deriving and analyzing online optimization algorithms that incorporate adaptive, data-dependent regularization, also termed preconditioning. Such algorithms have been proven useful in stochastic optimization by reshaping the gradients according to the geometry of the data. Our framework capt…

Incorporating second order curvature information in gradient based methods have shown to improve convergence drastically despite its computational intensity. In this paper, we propose a stochastic (online) quasi-Newton method with Nesterov's accelerated gradient in both its full and limited memory forms for solving lar…

Bayesian online changepoint detection (BOCPD) (Adams & MacKay, 2007) offers a rigorous and viable way to identify changepoints in complex systems. In this work, we introduce a Stein variational online changepoint detection (SVOCD) method to provide a computationally tractable generalization of BOCPD beyond the exponent…

OSGM uses online learning to adapt stepsize for faster convergence.

New approach improves convergence of online learning for ARIMA models.

Improved multiclass logistic regression with lower computational complexity.

Bayesian approach improves AdaLoRA's performance and efficiency.

Nonnegative matrix factorization (NMF) is a popular method for audio spectral unmixing. While NMF is traditionally applied to off-the-shelf time-frequency representations based on the short-time Fourier or Cosine transforms, the ability to learn transforms from raw data attracts increasing attention. However, this adds…

A method for online tensor dictionary learning is proposed. With the assumption of separable dictionaries, tensor contraction is used to diminish a $N$-way model of $\mathcal{O}\left(L^N\right)$ into a simple matrix equation of $\mathcal{O}\left(NL^2\right)$ with a real-time capability. To avoid numerical instability d…

A standard introduction to online learning might place Online Gradient Descent at its center and then proceed to develop generalizations and extensions like Online Mirror Descent and second-order methods. Here we explore the alternative approach of putting Exponential Weights (EW) first. We show that many standard meth…

In this paper, we address the problem of adaptive learning for autoregressive moving average (ARMA) model in the quaternion domain. By transforming the original learning problem into a full information optimization task without explicit noise terms, and then solving the optimization problem using the gradient descent a…

New method solves constrained stochastic optimization problems efficiently.

New method estimates marginal likelihood for deep learning models using training data alone.

A new method for optimization in probability space using Newton's flows.

Newton's method solves variational problems on manifolds.

Any gradient descent optimization requires to choose a learning rate. With deeper and deeper models, tuning that learning rate can easily become tedious and does not necessarily lead to an ideal convergence. We propose a variation of the gradient descent algorithm in the which the learning rate is not fixed. Instead, w…

New method for online inference of constrained optimization problems.

New algorithm achieves optimal regret in non-stochastic control, showing stochasticity is not beneficial.

Regularized nonlinear acceleration (RNA) estimates the minimum of a function by post-processing iterates from an algorithm such as the gradient method. It can be seen as a regularized version of Anderson acceleration, a classical acceleration scheme from numerical analysis. The new scheme provably improves the rate of …

Book covers tools for zeroth-order convex optimisation.

We generalize Newton-type methods for minimizing smooth functions to handle a sum of two convex functions: a smooth function and a nonsmooth function with a simple proximal mapping. We show that the resulting proximal Newton-type methods inherit the desirable convergence behavior of Newton-type methods for minimizing s…

We describe stochastic Newton and stochastic quasi-Newton approaches to efficiently solve large linear least-squares problems where the very large data sets present a significant computational burden (e.g., the size may exceed computer memory or data are collected in real-time). In our proposed framework, stochasticity…

New Q-Newton's method avoids saddle points and converges quadratically.

Unified approach to Bayesian inference with guarantees on covariance matrices.

Newton's method tackles nonlinear mappings into vector bundles with connections and retractions.

The overarching goal of this paper is to derive excess risk bounds for learning from exp-concave loss functions in passive and sequential learning settings. Exp-concave loss functions encompass several fundamental problems in machine learning such as squared loss in linear regression, logistic loss in classification, a…

A new optimization method improves deep learning accuracy without hyper-parameter tuning.

This thesis disentangles Gauss-Newton and variational approximations in Bayesian deep learning.

We present two new remarkably simple stochastic second-order methods for minimizing the average of a very large number of sufficiently smooth and strongly convex functions. The first is a stochastic variant of Newton's method (SN), and the second is a stochastic variant of cubically regularized Newton's method (SCN). W…

SVRN accelerates Newton methods by reducing variance and improving performance.

Deep learning involves a difficult non-convex optimization problem, which is often solved by stochastic gradient (SG) methods. While SG is usually effective, it may not be robust in some situations. Recently, Newton methods have been investigated as an alternative optimization technique, but nearly all existing studies…