arXiv research

Bayesian approach improves AdaLoRA's performance and efficiency.

Improved complexity for smooth nonconvex optimization using quasi-Newton methods.

IVON optimizes large neural networks, matching or outperforming Adam.

New adaptive first-order methods improve on quasi-Newton variants.

Improved IVON boosts LoRA model accuracy and calibration.

Paper proposes an online covariance estimator for sketched Newton methods.

Bayesian online changepoint detection (BOCPD) (Adams & MacKay, 2007) offers a rigorous and viable way to identify changepoints in complex systems. In this work, we introduce a Stein variational online changepoint detection (SVOCD) method to provide a computationally tractable generalization of BOCPD beyond the exponent…

Improved neural network ensembles using Stein Variational Newton updates.

Paper proposes an efficient online Newton method with Nesterov's acceleration for streaming data.

New quasi-Newton method guarantees global superlinear convergence.

Newton's method solves variational problems on manifolds.

The second order method as Newton Step is a suitable technique in Online Learning to guarantee regret bound. The large data is a challenge in Newton method to store second order matrices as hessian. In this paper, we have proposed an modified online Newton step that store first and second order matrices of dimension m …

Paper analyzes regret bounds for unconstrained online optimization.

Efficient algorithm for zeroth-order bandit convex optimization with bounds on regret.

Paper proposes a quasi-Newton method for nonlinear equations with global convergence guarantees.

This paper introduces online algorithms to estimate robust geometric median in large data streams.

Pathfinder uses quasi-Newton optimization for variational inference.

New algorithm improves convergence of Bayesian inference.

This thesis disentangles Gauss-Newton and variational approximations in Bayesian deep learning.

Incorporating second order curvature information in gradient based methods have shown to improve convergence drastically despite its computational intensity. In this paper, we propose a stochastic (online) quasi-Newton method with Nesterov's accelerated gradient in both its full and limited memory forms for solving lar…

We present the Variational Adaptive Newton (VAN) method which is a black-box optimization method especially suitable for explorative-learning tasks such as active learning and reinforcement learning. Similar to Bayesian methods, VAN estimates a distribution that can be used for exploration, but requires computations th…

Online learning with limited information feedback (bandit) tries to solve the problem where an online learner receives partial feedback information from the environment in the course of learning. Under this setting, Flaxman et al.[8] extended Zinkevich's classical Online Gradient Descent (OGD) algorithm [29] by proposi…

Any gradient descent optimization requires to choose a learning rate. With deeper and deeper models, tuning that learning rate can easily become tedious and does not necessarily lead to an ideal convergence. We propose a variation of the gradient descent algorithm in the which the learning rate is not fixed. Instead, w…

RONM method reduces regret in stochastic convex bandits with decreasing noise.

This paper proposes a family of online second order methods for possibly non-convex stochastic optimizations based on the theory of preconditioned stochastic gradient descent (PSGD), which can be regarded as an enhance stochastic Newton method with the ability to handle gradient noise and non-convexity simultaneously. …

An online learning framework for survival analysis with real-time adaptation.

Improved multiclass logistic regression with lower computational complexity.

A standard introduction to online learning might place Online Gradient Descent at its center and then proceed to develop generalizations and extensions like Online Mirror Descent and second-order methods. Here we explore the alternative approach of putting Exponential Weights (EW) first. We show that many standard meth…

Global convergence of an online (stochastic) limited memory version of the Broyden-Fletcher- Goldfarb-Shanno (BFGS) quasi-Newton method for solving optimization problems with stochastic objectives that arise in large scale machine learning is established. Lower and upper bounds on the Hessian eigenvalues of the sample …

New method solves convex optimization faster than NAG.

We introduce a framework for Newton's flows in probability space with information metrics, named information Newton's flows. Here two information metrics are considered, including both the Fisher-Rao metric and the Wasserstein-2 metric. A known fact is that overdamped Langevin dynamics correspond to Wasserstein gradien…

New approach improves convergence of online learning for ARIMA models.

Develops computational methods for simulating rigid body dynamics on SO(3).

Improved continual learning method using variational inference and FiLM layers.

Paper develops efficient methods for estimating Hessian inverses in stochastic optimization.

OSGM uses online learning to adapt stepsize for faster convergence.

Stein variational gradient descent (SVGD) was recently proposed as a general purpose nonparametric variational inference algorithm [Liu & Wang, NIPS 2016]: it minimizes the Kullback-Leibler divergence between the target distribution and its approximation by implementing a form of functional gradient descent on a reprod…

Unified approach to Bayesian inference with guarantees on covariance matrices.

Enhanced VMC methods improve neural wavefunction training.

Bayesian method improves online NARMAX model identification.

Regularized nonlinear acceleration (RNA) estimates the minimum of a function by post-processing iterates from an algorithm such as the gradient method. It can be seen as a regularized version of Anderson acceleration, a classical acceleration scheme from numerical analysis. The new scheme provably improves the rate of …

A new sequential method estimates Poisson means in streaming data, achieving optimality and efficiency.

RQMC improves optimization in variational Bayes problems.

New method estimates marginal likelihood for deep learning models using training data alone.

Improved online convex optimization with delayed feedback using curvature.

A scalable online method for Gaussian processes that improves decision-making in various applications.

New algorithms reduce online learning regret by tracking gradient variation.

We describe a framework for deriving and analyzing online optimization algorithms that incorporate adaptive, data-dependent regularization, also termed preconditioning. Such algorithms have been proven useful in stochastic optimization by reshaping the gradients according to the geometry of the data. Our framework capt…