This manuscript proposes a probabilistic framework for algorithms that iteratively solve unconstrained linear problems with positive definite for . The goal is to replace the point estimates returned by existing methods with a Gaussian posterior belief over the elements of the inverse of , which can …
On-device research index
arXiv research
A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
168,742 papers · 148 categories
Trend · papers per month
48 results for “quasi-Newton solvers”
Advances smooth over-parameterization for solving non-smooth optimization problems.
problem Non-smooth optimization with structural constraints in imaging and machine learning.
method Smooth over-parameterization of non-smooth problems, using gradient descent and mirror descent.
result Gradient descent on the reformulated smooth problem converges efficiently without parameter tuning.
A Novel Fast Exact Subproblem Solver for Stochastic Quasi-Newton Cubic Regularized Optimizationmath.OC
New method solves optimization problems faster than existing methods.
problem Large-scale nonconvex optimization problems.
method ARC method using LQN matrices with exact CR subproblem solution.
result Exact solutions to CR subproblem found in matrix-free manner.
cuRegOT accelerates GPU-based entropic OT solving.
problem Slow convergence and high computational cost of optimal transport on GPUs.
method High-performance GPU solver with algorithmic and architectural optimizations.
result Significant speedups over state-of-the-art solvers.
We extend the well-known BFGS quasi-Newton method and its memory-limited variant LBFGS to the optimization of nonsmooth convex objectives. This is done in a rigorous fashion by generalizing three components of BFGS to subdifferentials: the local quadratic model, the identification of a descent direction, and the Wolfe …
Proposes a Quasi-Newton trust region method for policy optimization in reinforcement learning.
problem Lack of stepsize selection criterion and slow convergence in gradient descent for policy optimization.
method Uses a trust region method with Quasi-Newton approximation for the Hessian.
result Demonstrates improved performance and efficiency in continuous control tasks.
A new method solves sparse regularization problems efficiently and robustly.
problem Sparse regularization in machine learning problems.
method Iteratively reweighted least square (IRLS) with bilevel resolution and BFGS solver.
result Achieves top performances on various sparsity and regularization problems.
New quasi-Newton method guarantees global superlinear convergence.
problem Global convergence and superlinear convergence of quasi-Newton methods.
method Hybrid proximal extragradient method with online learning for Hessian approximation.
result First globally convergent quasi-Newton method with explicit superlinear convergence rate.
Four decades after their invention, quasi-Newton methods are still state of the art in unconstrained numerical optimization. Although not usually interpreted thus, these are learning algorithms that fit a local quadratic approximation to the objective function. We show that many, including the most popular, quasi-Newto…
A new quasi-Newton method uses cubic regularization to avoid saddle points in deep learning.
problem Avoiding saddle points and poor local minima in deep learning models.
method Limited-memory symmetric rank-one quasi-Newton approach with adaptive regularized cubics.
result The method effectively avoids saddle points and converges to better local minima.
We propose a novel general algorithm LHAC that efficiently uses second-order information to train a class of large-scale l1-regularized problems. Our method executes cheap iterations while achieving fast local convergence rate by exploiting the special structure of a low-rank matrix, constructed via quasi-Newton approx…
In [19], a general, inexact, efficient proximal quasi-Newton algorithm for composite optimization problems has been proposed and a sublinear global convergence rate has been established. In this paper, we analyze the convergence properties of this method, both in the exact and inexact setting, in the case when the obje…
Recently algorithms incorporating second order curvature information have become popular in training neural networks. The Nesterov's Accelerated Quasi-Newton (NAQ) method has shown to effectively accelerate the BFGS quasi-Newton method by incorporating the momentum term and Nesterov's accelerated gradient vector. A sto…
Paper proposes a new method to efficiently incorporate curvature information in stochastic optimization.
problem Minimizing nonconvex functions with limited curvature information.
method Structured stochastic quasi-Newton method using partial Hessian information.
result Global convergence to stationary point and local superlinear convergence rate established.
Recent studies incorporate Nesterov's accelerated gradient method for the acceleration of gradient based training. The Nesterov's Accelerated Quasi-Newton (NAQ) method has shown to drastically improve the convergence speed compared to the conventional quasi-Newton method. This paper implements NAQ for non-convex optimi…
Paper develops a robust PP distributed quasi-Newton estimation for Byzantine machines.
problem Byzantine machines in distributed computing under Privacy Protection constraints.
method Robust PP distributed quasi-Newton estimation method that transmits only five vectors.
result Reduces privacy budgeting and transmission cost compared to gradient descent and Newton iteration.
We consider stochastic zero-order optimization problems, which arise in settings from simulation optimization to reinforcement learning. We propose an adaptive sampling quasi-Newton method where we estimate the gradients of a stochastic function using finite differences within a common random number framework. We emplo…
We describe stochastic Newton and stochastic quasi-Newton approaches to efficiently solve large linear least-squares problems where the very large data sets present a significant computational burden (e.g., the size may exceed computer memory or data are collected in real-time). In our proposed framework, stochasticity…
New method improves zeroth-order stochastic optimization with adaptive sampling.
problem Optimization problems without gradient information.
method Adaptive sampling quasi-Newton method using finite differences.
result Significant improvement in performance with adaptive sample sizes.
In this paper we present a novel quasi-Newton algorithm for use in stochastic optimisation. Quasi-Newton methods have had an enormous impact on deterministic optimisation problems because they afford rapid convergence and computationally attractive algorithms. In essence, this is achieved by learning the second-order (…
Paper proposes a quasi-Newton method for nonlinear equations with global convergence guarantees.
problem Solving smooth and monotone nonlinear equations efficiently and globally.
method Hybrid proximal extragradient framework combined with online learning for Jacobian approximation.
result First global convergence results showing quasi-Newton method's advantage over extragradient method.
New adaptive first-order methods improve on quasi-Newton variants.
problem Designing efficient gradient methods for practical applications.
method Online scaled gradient methods (OSGM) with new adaptive methods OSGM-Best.
result OSGM-Best matches quasi-Newton variants but requires less memory and cheaper iterations.
Improved complexity for smooth nonconvex optimization using quasi-Newton methods.
problem Finding ε-first-order stationary points of smooth functions with gradient information only.
method Two-level online learning approach involving quasi-Newton methods.
result Gradient complexity improved to O(d^(1/4)ε^(-13/8)) for d = O(ε^(-1/2)).
New method achieves superlinear convergence rate with limited memory.
problem Achieving superlinear convergence rate in quasi-Newton methods with limited memory.
method Limited-memory Greedy BFGS (LG-BFGS) method with displacement aggregation and basis vector selection.
result Explicit non-asymptotic superlinear convergence rate demonstrated.
SGD vs quasi-Newton optimization in neural networks: different landscapes, different generalizability.
problem Understanding neural network optimization and generalizability.
method Comparison of stochastic gradient descent (SGD) and quasi-Newton optimization methods using computational tools.
result SGD solutions are separated by lower barriers than quasi-Newton solutions, but quasi-Newton solutions are deeper and more isolated.
Bayesian coresets improved with random sampling and quasi-Newton optimization.
problem Efficiently approximate Bayesian posterior distributions for computationally expensive inference.
method Randomly select a subset of data points, then optimize weights using quasi-Newton method.
result First algorithm with high-probability KL divergence bound on coreset quality.
Novel algorithm accelerates PnP methods for image deblurring and super-resolution.
problem Efficiently solving inverse problems and imaging with provable convergence guarantees.
method Incorporates quasi-Newton steps into provable PnP framework based on proximal denoisers.
result 2--8x faster convergence compared to other provable PnP methods with similar quality.
Deep learning algorithms often require solving a highly non-linear and nonconvex unconstrained optimization problem. Methods for solving optimization problems in large-scale machine learning, such as deep learning and deep reinforcement learning (RL), are generally restricted to the class of first-order algorithms, lik…
The question of how to incorporate curvature information in stochastic approximation methods is challenging. The direct application of classical quasi- Newton updating techniques for deterministic optimization leads to noisy curvature estimates that have harmful effects on the robustness of the iteration. In this paper…
Apollo improves nonconvex stochastic optimization efficiency.
problem Nonconvex stochastic optimization challenges.
method Adaptive parameter-wise diagonal quasi-Newton method approximating Hessian.
result Significant improvements in convergence speed and generalization over SGD and Adam.
During recent years there has been an increased interest in stochastic adaptations of limited memory quasi-Newton methods, which compared to pure gradient-based routines can improve the convergence by incorporating second order information. In this work we propose a direct least-squares approach conceptually similar to…
Pathfinder uses quasi-Newton optimization for variational inference.
problem Approximating complex posterior distributions efficiently.
method Pathfinder combines quasi-Newton optimization with variational methods to approximate log densities.
result Pathfinder produces draws with lower KL divergence than ADVI and comparable to HMC, requiring fewer evaluations.
In this paper, a novel stochastic extra-step quasi-Newton method is developed to solve a class of nonsmooth nonconvex composite optimization problems. We assume that the gradient of the smooth part of the objective function can only be approximated by stochastic oracles. The proposed method combines general stochastic …
New method solves convex optimization faster than NAG.
problem Unconstrained smooth convex optimization problems.
method Accelerated quasi-Newton proximal extragradient (A-QPNE) method.
result Achieves a faster convergence rate of .
Machine learning (ML) problems are often posed as highly nonlinear and nonconvex unconstrained optimization problems. Methods for solving ML problems based on stochastic gradient descent are easily scaled for very large problems but may involve fine-tuning many hyper-parameters. Quasi-Newton approaches based on the lim…
We consider the class of optimization problems arising from computationally intensive L1-regularized M-estimators, where the function or gradient values are very expensive to compute. A particular instance of interest is the L1-regularized MLE for learning Conditional Random Fields (CRFs), which are a popular class of …
Develops efficient quasi-Newton methods for training deep neural networks.
problem Training deep neural networks with large-scale Hessian matrices.
method Approximates Hessian as block-diagonal Kronecker product of smaller matrices, applies damping.
result Outperforms or matches state-of-the-art methods in autoencoder models.
New method speeds up change-point detection in data sequences.
problem Efficiently detecting change-points in long data sequences.
method Sequential Gradient Descent and Quasi-Newton's Method.
result New method can be orders of magnitude faster than existing methods.
Novel approach simplifies VI problems with faster performance.
problem Black-box VI optimization problems.
method Sample Average Approximation (SAA) combined with quasi-Newton methods and line search.
result Achieves faster performance than existing methods.
A new optimization method handles Euclidean bounds efficiently.
problem Optimization problems with Euclidean bounds.
method Riemannian limited-memory BFGS method combining quasi-Newton and Riemannian adaptations.
result Outperforms existing methods by several orders of magnitude.
Incorporating second order curvature information in gradient based methods have shown to improve convergence drastically despite its computational intensity. In this paper, we propose a stochastic (online) quasi-Newton method with Nesterov's accelerated gradient in both its full and limited memory forms for solving lar…
Particle Metropolis-Hastings enables Bayesian parameter inference in general nonlinear state space models (SSMs). However, in many implementations a random walk proposal is used and this can result in poor mixing if not tuned correctly using tedious pilot runs. Therefore, we consider a new proposal inspired by quasi-Ne…
Riemannian stochastic quasi-Newton algorithm with variance reduction and its convergence analysiscs.LG
Stochastic variance reduction algorithms have recently become popular for minimizing the average of a large, but finite number of loss functions. The present paper proposes a Riemannian stochastic quasi-Newton algorithm with variance reduction (R-SQN-VR). The key challenges of averaging, adding, and subtracting multipl…
Global convergence of an online (stochastic) limited memory version of the Broyden-Fletcher- Goldfarb-Shanno (BFGS) quasi-Newton method for solving optimization problems with stochastic objectives that arise in large scale machine learning is established. Lower and upper bounds on the Hessian eigenvalues of the sample …
Predictive coding networks are shown to be stable, robust, and converge faster than backpropagation.
problem Stability, robustness, and convergence of predictive coding networks.
method Dynamical systems theory and Lyapunov stability analysis.
result Predictive coding networks are Lyapunov stable and converge faster than backpropagation.
A Distributed Quasi-Newton Algorithm for Primal and Dual Regularized Empirical Risk Minimizationcs.LG
A new distributed optimization method for ERM problems.
problem Efficiently solving ERM problems with nonsmooth regularization in a distributed setting.
method Second-order distributed optimization using successive quadratic approximations and Hessian approximation.
result Global linear convergence for a broad range of non-strongly convex problems.
Data privacy and security becomes a major concern in building machine learning models from different data providers. Federated learning shows promise by leaving data at providers locally and exchanging encrypted information. This paper studies the vertical federated learning structure for logistic regression where the …
Pseudo-marginal Metropolis-Hastings (pmMH) is a versatile algorithm for sampling from target distributions which are not easy to evaluate point-wise. However, pmMH requires good proposal distributions to sample efficiently from the target, which can be problematic to construct in practice. This is especially a problem …