A new optimisation method efficiently scales Hessian-vector products for neural networks.
problem Challenges in applying second-order quasi-Newton methods due to large Hessian and non-convexity.
method Proposes an optimisation algorithm that asymptotically uses the exact inverse Hessian with modified eigenvalues.
result Demonstrates scalability and comparable performance to other optimisation methods in neural networks.
Paper develops a gradient-like proposal for discrete distributions without requiring natural differentiability.
problem Lack of natural differentiability in proposal distributions for discrete distributions.
method Locally-balanced proposal combined with Newton's series expansion for efficient exploration.
result Method guarantees convergence rate and outperforms alternatives in various experiments.
Paper describes links of mixed polynomials with specific properties.
problem Understanding the links of mixed polynomials with nice Newton boundaries.
method Analyzes links constructed from sequences of links associated with compact 1-faces of the Newton boundary.
result Links of singularities of inner non-degenerate mixed polynomials can be described using a specific procedure.
The article proves integral formulas for foliated sub-Riemannian manifolds.
problem Integral formulas for foliated sub-Riemannian manifolds.
method Proved a series of integral formulae involving mean curvatures, Newton transformations, and curvature tensor.
result Generalized known integral formulas for codimension-one foliations.
In this text, we establish the risk model based on AR(1) series and propose the basic model which has a dependent structure under intensity of claim number. Considering some properties of the risk model, we take advantage of newton iteration method to figure out the adjustment coefficient and estimate the exponential u…
We present a novel statistical inference framework for convex empirical risk minimization, using approximate stochastic Newton steps. The proposed algorithm is based on the notion of finite differences and allows the approximation of a Hessian-vector product from first-order information. In theory, our method efficient…
A new method for optimization in probability space using Newton's flows.
problem Optimization in probability space with information metrics.
method Information Newton's flows, including Fisher-Rao and Wasserstein-2 metrics, with Newton's Langevin dynamics and variational methods.
result Effective numerical implementation and convergence results for the proposed method.
We describe stochastic Newton and stochastic quasi-Newton approaches to efficiently solve large linear least-squares problems where the very large data sets present a significant computational burden (e.g., the size may exceed computer memory or data are collected in real-time). In our proposed framework, stochasticity…
Newton's method solves variational problems on manifolds.
problem Solving variational equations on manifolds.
method Newton's method with affine covariant damping strategy.
result Numerical results for variational problems demonstrated.
Muon with Newton-Schulz converges to the same stationary point as SVD-polar, up to a constant factor.
problem Improving the convergence rate of Muon optimizer.
method Using Newton-Schulz steps for momentum orthogonalization, proving convergence rate and constant factor.
result Muon with Newton-Schulz converges to the same stationary point as SVD-polar, up to a constant factor.
Simple stochastic Newton and cubic Newton methods with fast convergence.
problem Minimizing large numbers of smooth and strongly convex functions.
method Stochastic Newton and cubic Newton methods with simple local linear-quadratic rates.
result Local linear-quadratic convergence results with fast adaptation to problem's curvature.
RNN operators solve Newton's equations with large timesteps for molecular dynamics.
problem Solving Newton's equations of motion with large timesteps for molecular dynamics simulations.
method Recurrent Neural Networks (RNN) operators to solve Newton's equations using past trajectory data.
result Significant speedup in molecular dynamics simulations with timesteps up to 4000 times larger.
New algorithm improves convergence of gradient boosting trees.
problem Global convergence of Newton boosting in tabular machine learning.
method Introduces Gradient Regularized Newton Descent for GBDTs, proving linear convergence for smooth, strongly convex losses and O(k21) rate for general convex losses. result Achieves globally convergent second-order GBDT algorithm with rate matching first-order boosting.
We generalize Newton-type methods for minimizing smooth functions to handle a sum of two convex functions: a smooth function and a nonsmooth function with a simple proximal mapping. We show that the resulting proximal Newton-type methods inherit the desirable convergence behavior of Newton-type methods for minimizing s…
Boosting algorithms are frequently used in applied data science and in research. To date, the distinction between boosting with either gradient descent or second-order Newton updates is often not made in both applied and methodological research, and it is thus implicitly assumed that the difference is irrelevant. The g…
Newton's method tackles nonlinear mappings into vector bundles with connections and retractions.
problem Finding zeros of mappings from a manifold into a vector bundle.
method Local convergence using differentiability concepts, Banach space Riemannian distance, and affine covariant damping strategy.
result Illustrated application to generalized non-symmetric eigenvalue problems.
Characterizes local tropicalizations of splice type surface singularities.
problem Understanding splice type surface singularities from a tropical geometry perspective.
method Characterization of local tropicalizations as cones over splice diagrams, using tropical methods.
result Characterizes local tropicalizations of splice type surface singularities as cones over associated splice diagrams.
Study uses Newton polytopes to distinguish Lagrangian fillings of Legendrian submanifolds.
problem Distinguishing Lagrangian fillings of Legendrian submanifolds.
method Utilizes Newton polytopes associated with augmented values of Reeb chords.
result Newton polytopes can distinguish infinitely many distinct Lagrangian fillings.
Unified approach to Bayesian inference with guarantees on covariance matrices.
problem Approximate Bayesian inference with PSD guarantees.
method Bayes-Newton methods extending Newton's method for optimisation.
result Novel algorithms with PSD covariance matrices.
Paper proposes an online covariance estimator for sketched Newton methods.
problem Estimating the limiting covariance matrix of sketched Newton methods.
method Proposes a fully online covariance matrix estimator from Newton iterates.
result Establishes the consistency and convergence rate of the proposed estimator.
New Q-Newton's method avoids saddle points and converges quadratically.
problem Optimizing functions with saddle points and ensuring convergence guarantees.
method Modified New Q-Newton's method with Backtracking line search.
result Theorem for Morse functions: quadratic convergence to local minima.
The article derives integral formulas for foliated sub-Riemannian manifolds.
problem Integrating geometric concepts in Riemannian manifolds with foliations.
method Deriving integral formulas involving shape operators and curvature tensor.
result Generalizes results for foliated Riemannian manifolds and includes arbitrary functions.
A new optimization method improves deep learning accuracy without hyper-parameter tuning.
problem Computational demands and convergence behavior in deep learning training.
method Stochastic quasi-Gauss-Newton (SQGN) optimization method combining stochastic quasi-Newton, Gauss-Newton, and variance reduction.
result SQGN provides excellent accuracy without hyper-parameter experimentation, improving convergence and computational performance.
Proposes a Quasi-Newton trust region method for policy optimization in reinforcement learning.
problem Lack of stepsize selection criterion and slow convergence in gradient descent for policy optimization.
method Uses a trust region method with Quasi-Newton approximation for the Hessian.
result Demonstrates improved performance and efficiency in continuous control tasks.
This thesis disentangles Gauss-Newton and variational approximations in Bayesian deep learning.
problem Understanding the interplay between the Gauss-Newton method and variational approximations in Bayesian deep learning.
method Analysis of the Gauss-Newton method and Laplace/Gaussian variational approximations for neural networks.
result The combination of the Gauss-Newton method with approximate inference can be cast as inference in a linear or Gaussian process model.
SVRN accelerates Newton methods by reducing variance and improving performance.
problem Improving the efficiency of Newton methods for large-scale optimization problems.
method Stochastic Variance-Reduced Newton (SVRN) algorithm that accelerates Subsampled Newton and Iterative Hessian Sketch algorithms.
result SVRN accelerates Newton methods by reducing the number of passes over the data, achieving a significant improvement in performance.
In this article we present a natural generalization of Newton's Second Law valid in field theory, i.e., when the parameterized curves are replaced by parameterized submanifolds of higher dimension. For it we introduce what we have called the geodesic k-vector field, analogous to the ordinary geodesic field and which …
The second order method as Newton Step is a suitable technique in Online Learning to guarantee regret bound. The large data is a challenge in Newton method to store second order matrices as hessian. In this paper, we have proposed an modified online Newton step that store first and second order matrices of dimension m …
Deep learning involves a difficult non-convex optimization problem, which is often solved by stochastic gradient (SG) methods. While SG is usually effective, it may not be robust in some situations. Recently, Newton methods have been investigated as an alternative optimization technique, but nearly all existing studies…
Study of measured laminations on surfaces using Newton polytopes and Poisson brackets.
problem Understanding the space of measured laminations on surfaces from a valuative perspective.
method Introducing Newton polytopes for character variety functions, defining tangent spaces, and identifying symplectic structures.
result Trace functions have unit coefficients at the extremal points of their Newton polytopes.
New approach improves convergence of online learning for ARIMA models.
problem Pitfalls in computational complexity and convergence of Online Newton Step optimization method.
method Proposes a new approach combining different Online Gradient Descent learners for iterative training of ARIMA models.
result The proposed approach outperforms existing methods resulting in lower prediction error.
Approximate Newton methods are a standard optimization tool which aim to maintain the benefits of Newton's method, such as a fast rate of convergence, whilst alleviating its drawbacks, such as computationally expensive calculation or estimation of the inverse Hessian. In this work we investigate approximate Newton meth…
Newton-LESS sparsifies Gaussian sketching for faster optimization.
problem Computing the Hessian matrix in optimization is computationally expensive.
method Uses a sparsified version of a dense Gaussian sketching matrix.
result Achieves nearly the same convergence rate as dense Gaussian embeddings without the computational cost.
New quasi-Newton method guarantees global superlinear convergence.
problem Global convergence and superlinear convergence of quasi-Newton methods.
method Hybrid proximal extragradient method with online learning for Hessian approximation.
result First globally convergent quasi-Newton method with explicit superlinear convergence rate.
Paper develops a robust PP distributed quasi-Newton estimation for Byzantine machines.
problem Byzantine machines in distributed computing under Privacy Protection constraints.
method Robust PP distributed quasi-Newton estimation method that transmits only five vectors.
result Reduces privacy budgeting and transmission cost compared to gradient descent and Newton iteration.
Four decades after their invention, quasi-Newton methods are still state of the art in unconstrained numerical optimization. Although not usually interpreted thus, these are learning algorithms that fit a local quadratic approximation to the objective function. We show that many, including the most popular, quasi-Newto…
A new quasi-Newton method uses cubic regularization to avoid saddle points in deep learning.
problem Avoiding saddle points and poor local minima in deep learning models.
method Limited-memory symmetric rank-one quasi-Newton approach with adaptive regularized cubics.
result The method effectively avoids saddle points and converges to better local minima.
Extends Newton's minimal resistance problem to Riemannian surfaces.
problem Minimal resistance on Riemannian surfaces.
method Derive resistance functional, analyze constrained minimization.
result Smooth extremals are loxodromes, global minimizers characterized.
A new multivariate stochastic volatility estimation procedure for financial time series is proposed. A Wishart autoregressive process is considered for the volatility precision covariance matrix, for the estimation of which a two step procedure is adopted. The first step is the conditional inference on the autoregressi…
A new Bayesian filtering method speeds up stochastic Newton optimization.
problem Minimizing log-convex functions using stochastic methods.
method Contextualizes the problem as Bayesian inference, applying Bayesian filtering to update estimates.
result Establishes conditions for diminishing effect of older observations, akin to momentum.
The Gauss-Newton method is analyzed for neural networks using Riemannian optimization techniques.
problem Training neural networks with smooth activations and convergence rates.
method Riemannian optimization perspective, analyzing the Gauss-Newton method in both underparameterized and overparameterized regimes.
result Geometric convergence rates independent of conditioning and eigenvalues, demonstrating accelerated convergence.
Extends Newton's minimal resistance problem to Lorentz-Minkowski space.
problem Minimal resistance in Lorentz-Minkowski space.
method Derived functional energy, determined Euler-Lagrange equation, analyzed maximum principle, found separable and radial solutions.
result Obtained solutions with conical singularities at the origin and analyzed the Single Shock Condition.
ISAAC Newton uses input-based curvature for efficient training.
problem Efficient training in small-batch stochastic regimes.
method ISAAC Newton conditions gradients using selected second-order information based on input.
result Effective training even in small-batch stochastic regimes, competitive to first-order and second-order methods.
Geometric framework for Newton's equations on diffeomorphism groups.
problem Modeling fluid dynamics and related systems on geometric spaces.
method Geodesic approach and infinite-dimensional information geometry.
result Unified framework for various fluid dynamics equations.
Improved root-finding method for smooth functions.
problem Finding roots of smooth functions efficiently and reliably.
method A new improvement over Newton's method for doubly differentiable functions.
result Faster and more reliable convergence compared to Newton's method.
A new method for machine learning updates reduces complexity and improves robustness.
problem Stochastic gradient updates are inefficient and sensitive to feature scaling.
method Incremental Gauss-Newton Descent (IGND) reduces the need for matrix operations and improves robustness.
result IGND improves robustness to sensitivity scaling and can be competitive with common stochastic optimizers.
Paper proposes an efficient online Newton method with Nesterov's acceleration for streaming data.
problem Efficient inference of online Newton methods with robustness to noise and ill-conditioning.
method Online Newton method with Hessian averaging and Nesterov's accelerated sketch-and-project solver.
result Global almost-sure convergence and asymptotic normality of the last iterate with non-asymptotic convergence guarantees.
A new method for faster optimization on statistical manifolds.
problem Slow convergence of first-order methods in manifold optimization.
method Dual Riemannian Newton method on manifolds with dual connections.
result Local quadratic convergence of the dual Riemannian Newton method.