arXiv research

Second-order optimizers retain residual information after data deletion, affecting machine unlearning.

Data whitening and second order optimization harm generalization by reducing access to dataset information.

Negative step sizes improve second-order methods for neural networks.

AdaSub optimizes with second-order info in low-dims subspace.

Natural gradient descent, which preconditions a gradient descent update with the Fisher information matrix of the underlying statistical model, is a way to capture partial second-order information. Several highly visible works have advocated an approximation known as the empirical Fisher, drawing connections between ap…

Enhances SMC² with Hessian info for more efficient posterior approximation.

Improved SVRG method using BB techniques for faster convergence.

A new method for faster optimization on statistical manifolds.

New algorithm finds approximate stationary points in non-convex optimization.

Improved robustness in optimization methods using second-order information.

First-order stochastic methods are the state-of-the-art in large-scale machine learning optimization owing to efficient per-iteration complexity. Second-order methods, while able to provide faster convergence, have been much less explored due to the high cost of computing the second-order information. In this paper we …

The paper improves ODE solvers by integrating diverse information types.

We introduce TrustVI, a fast second-order algorithm for black-box variational inference based on trust-region optimization and the reparameterization trick. At each iteration, TrustVI proposes and assesses a step based on minibatches of draws from the variational distribution. The algorithm provably converges to a stat…

Policy optimization on high-dimensional continuous control tasks exhibits its difficulty caused by the large variance of the policy gradient estimators. We present the action subspace dependent gradient (ASDG) estimator which incorporates the Rao-Blackwell theorem (RB) and Control Variates (CV) into a unified framework…

A new method improves ICA performance by approximating MDI.

Paper introduces FoMoH for optimization without backpropagation.

Second-order methods improve differential privacy in convex optimization.

Stein variational gradient descent (SVGD) was recently proposed as a general purpose nonparametric variational inference algorithm [Liu & Wang, NIPS 2016]: it minimizes the Kullback-Leibler divergence between the target distribution and its approximation by implementing a form of functional gradient descent on a reprod…

Paper proposes a new method for efficient second-order neural network training.

Improved computational complexity in statistical models using second-order information.

SONIA optimizes machine learning problems with a novel algorithm.

A new method for optimizing deep neural networks using TKFAC.

This paper proposes a framework of L-BFGS based on the (approximate) second-order information with stochastic batches, as a novel approach to the finite-sum minimization problems. Different from the classical L-BFGS where stochastic batches lead to instability, we use a smooth estimate for the evaluations of the gradie…

Finite-sum optimization problems are ubiquitous in machine learning, and are commonly solved using first-order methods which rely on gradient computations. Recently, there has been growing interest in \emph{second-order} methods, which rely on both gradients and Hessians. In principle, second-order methods can require …

Optimistic method adapted for faster convex-concave min-max problems.

COMRADE is a communication-efficient, Byzantine-resilient second-order optimization algorithm.

We study the convergence of a class of gradient-based Model-Agnostic Meta-Learning (MAML) methods and characterize their overall complexity as well as their best achievable accuracy in terms of gradient norm for nonconvex loss functions. We start with the MAML method and its first-order approximation (FO-MAML) and high…

Two classes of methods have been proposed for escaping from saddle points with one using the second-order information carried by the Hessian and the other adding the noise into the first-order information. The existing analysis for algorithms using noise in the first-order information is quite involved and hides the es…

Zeroth-order optimization is an important research topic in machine learning. In recent years, it has become a key tool in black-box adversarial attack to neural network based image classifiers. However, existing zeroth-order optimization algorithms rarely extract second-order information of the model function. In this…

New method provides tighter robustness guarantees for adversarial attacks.

We propose a novel general algorithm LHAC that efficiently uses second-order information to train a class of large-scale l1-regularized problems. Our method executes cheap iterations while achieving fast local convergence rate by exploiting the special structure of a low-rank matrix, constructed via quasi-Newton approx…

A new algorithm for solving constrained convex optimization problems efficiently.

The paper classifies second-order superintegrable systems with torsion and semi-degeneracy.

Online learning with limited information feedback (bandit) tries to solve the problem where an online learner receives partial feedback information from the environment in the course of learning. Under this setting, Flaxman et al.[8] extended Zinkevich's classical Online Gradient Descent (OGD) algorithm [29] by proposi…

Optimizes curves on Riemannian manifolds to minimize curvature.

We study 3-manifolds in $\mathbb{R}^5$ with corank $1$ singularities. At the singular point we define the curvature locus using the first and second fundamental forms, which contains all the local second order geometrical information about the manifold.

New quantum states capture more information, enabling advanced processing tasks.

Study proposes BFEL framework for privacy-preserving FL in personalized healthcare.

Researchers find a class/cross-class structure in deep learning spectra.

First-order methods such as stochastic gradient descent (SGD) are currently the standard algorithm for training deep neural networks. Second-order methods, despite their better convergence rate, are rarely used in practice due to the prohibitive computational cost in calculating the second-order information. In this pa…

This paper proposes an alternative to the classical price-adjustment mechanism (called "tâtonnement" after Walras) that is second-order in time. The proposed mechanism, an analogue to the damped harmonic oscillator, provides a dynamic equilibration process that depends only on local information. We show how such a proc…

Model improves information transfer from visual streams.

ISAAC Newton uses input-based curvature for efficient training.

Incorporating second order curvature information in gradient based methods have shown to improve convergence drastically despite its computational intensity. In this paper, we propose a stochastic (online) quasi-Newton method with Nesterov's accelerated gradient in both its full and limited memory forms for solving lar…

Kernel online convex optimization (KOCO) is a framework combining the expressiveness of non-parametric kernel models with the regret guarantees of online learning. First-order KOCO methods such as functional gradient descent require only $\mathcal{O}(t)$ time and space per iteration, and, when the only information on t…

Extends GCNs to directed graphs for better performance.

We consider the problem of efficiently computing the maximum likelihood estimator in Generalized Linear Models (GLMs) when the number of observations is much larger than the number of coefficients ($n \gg p \gg 1$). In this regime, optimization algorithms can immensely benefit from approximate second order information.…

New method improves online covariance estimation for SGD.