Boosting algorithms are frequently used in applied data science and in research. To date, the distinction between boosting with either gradient descent or second-order Newton updates is often not made in both applied and methodological research, and it is thus implicitly assumed that the difference is irrelevant. The g…
On-device research index
arXiv research
A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
168,657 papers · 148 categories
Trend · papers per month
48 results for “Newton Boosting”
New algorithm improves convergence of gradient boosting trees.
problem Global convergence of Newton boosting in tabular machine learning.
method Introduces Gradient Regularized Newton Descent for GBDTs, proving linear convergence for smooth, strongly convex losses and rate for general convex losses.
result Achieves globally convergent second-order GBDT algorithm with rate matching first-order boosting.
SnapBoost uses random base hypothesis classes to improve gradient boosting performance.
problem Improving gradient boosting performance.
method Heterogeneous Newton Boosting Machine (HNBM) with variable base hypothesis classes.
result SnapBoost achieves better generalization loss than competing frameworks.
As an adaptive, interpretable, robust, and accurate meta-algorithm for arbitrary differentiable loss functions, gradient tree boosting is one of the most popular machine learning techniques, though the computational expensiveness severely limits its usage. Stochastic gradient boosting could be adopted to accelerates gr…
Improved IVON boosts LoRA model accuracy and calibration.
problem Improving the accuracy and calibration of large language models.
method Replaced AdamW with IVON for finetuning Llama-2.
result IVON improves Llama-2 accuracy by 2.8% and calibration error by 4.6%.
We describe stochastic Newton and stochastic quasi-Newton approaches to efficiently solve large linear least-squares problems where the very large data sets present a significant computational burden (e.g., the size may exceed computer memory or data are collected in real-time). In our proposed framework, stochasticity…
We introduce a framework for Newton's flows in probability space with information metrics, named information Newton's flows. Here two information metrics are considered, including both the Fisher-Rao metric and the Wasserstein-2 metric. A known fact is that overdamped Langevin dynamics correspond to Wasserstein gradien…
Newton's method solves variational problems on manifolds.
problem Solving variational equations on manifolds.
method Newton's method with affine covariant damping strategy.
result Numerical results for variational problems demonstrated.
Muon with Newton-Schulz converges to the same stationary point as SVD-polar, up to a constant factor.
problem Improving the convergence rate of Muon optimizer.
method Using Newton-Schulz steps for momentum orthogonalization, proving convergence rate and constant factor.
result Muon with Newton-Schulz converges to the same stationary point as SVD-polar, up to a constant factor.
Solving Newton's Equations of Motion with Large Timesteps using Recurrent Neural Networks based Operatorsphysics.comp-ph
RNN operators solve Newton's equations with large timesteps for molecular dynamics.
problem Solving Newton's equations of motion with large timesteps for molecular dynamics simulations.
method Recurrent Neural Networks (RNN) operators to solve Newton's equations using past trajectory data.
result Significant speedup in molecular dynamics simulations with timesteps up to 4000 times larger.
We generalize Newton-type methods for minimizing smooth functions to handle a sum of two convex functions: a smooth function and a nonsmooth function with a simple proximal mapping. We show that the resulting proximal Newton-type methods inherit the desirable convergence behavior of Newton-type methods for minimizing s…
Newton's method tackles nonlinear mappings into vector bundles with connections and retractions.
problem Finding zeros of mappings from a manifold into a vector bundle.
method Local convergence using differentiability concepts, Banach space Riemannian distance, and affine covariant damping strategy.
result Illustrated application to generalized non-symmetric eigenvalue problems.
Bayesian approach generalizes ADMM for federated learning.
problem Improving federated learning efficiency and accuracy.
method Integrates Bayesian duality with ADMM for optimization.
result New extensions of ADMM for various distributions.
Study uses Newton polytopes to distinguish Lagrangian fillings of Legendrian submanifolds.
problem Distinguishing Lagrangian fillings of Legendrian submanifolds.
method Utilizes Newton polytopes associated with augmented values of Reeb chords.
result Newton polytopes can distinguish infinitely many distinct Lagrangian fillings.
Unified approach to Bayesian inference with guarantees on covariance matrices.
problem Approximate Bayesian inference with PSD guarantees.
method Bayes-Newton methods extending Newton's method for optimisation.
result Novel algorithms with PSD covariance matrices.
Paper proposes an online covariance estimator for sketched Newton methods.
problem Estimating the limiting covariance matrix of sketched Newton methods.
method Proposes a fully online covariance matrix estimator from Newton iterates.
result Establishes the consistency and convergence rate of the proposed estimator.
New Q-Newton's method avoids saddle points and converges quadratically.
problem Optimizing functions with saddle points and ensuring convergence guarantees.
method Modified New Q-Newton's method with Backtracking line search.
result Theorem for Morse functions: quadratic convergence to local minima.
A new optimization method improves deep learning accuracy without hyper-parameter tuning.
problem Computational demands and convergence behavior in deep learning training.
method Stochastic quasi-Gauss-Newton (SQGN) optimization method combining stochastic quasi-Newton, Gauss-Newton, and variance reduction.
result SQGN provides excellent accuracy without hyper-parameter experimentation, improving convergence and computational performance.
This thesis disentangles Gauss-Newton and variational approximations in Bayesian deep learning.
problem Understanding the interplay between the Gauss-Newton method and variational approximations in Bayesian deep learning.
method Analysis of the Gauss-Newton method and Laplace/Gaussian variational approximations for neural networks.
result The combination of the Gauss-Newton method with approximate inference can be cast as inference in a linear or Gaussian process model.
SVRN accelerates Newton methods by reducing variance and improving performance.
problem Improving the efficiency of Newton methods for large-scale optimization problems.
method Stochastic Variance-Reduced Newton (SVRN) algorithm that accelerates Subsampled Newton and Iterative Hessian Sketch algorithms.
result SVRN accelerates Newton methods by reducing the number of passes over the data, achieving a significant improvement in performance.
In this article we present a natural generalization of Newton's Second Law valid in field theory, i.e., when the parameterized curves are replaced by parameterized submanifolds of higher dimension. For it we introduce what we have called the geodesic -vector field, analogous to the ordinary geodesic field and which …
The second order method as Newton Step is a suitable technique in Online Learning to guarantee regret bound. The large data is a challenge in Newton method to store second order matrices as hessian. In this paper, we have proposed an modified online Newton step that store first and second order matrices of dimension m …
Deep learning involves a difficult non-convex optimization problem, which is often solved by stochastic gradient (SG) methods. While SG is usually effective, it may not be robust in some situations. Recently, Newton methods have been investigated as an alternative optimization technique, but nearly all existing studies…
Study of measured laminations on surfaces using Newton polytopes and Poisson brackets.
problem Understanding the space of measured laminations on surfaces from a valuative perspective.
method Introducing Newton polytopes for character variety functions, defining tangent spaces, and identifying symplectic structures.
result Trace functions have unit coefficients at the extremal points of their Newton polytopes.
Approximate Newton methods are a standard optimization tool which aim to maintain the benefits of Newton's method, such as a fast rate of convergence, whilst alleviating its drawbacks, such as computationally expensive calculation or estimation of the inverse Hessian. In this work we investigate approximate Newton meth…
We present two new remarkably simple stochastic second-order methods for minimizing the average of a very large number of sufficiently smooth and strongly convex functions. The first is a stochastic variant of Newton's method (SN), and the second is a stochastic variant of cubically regularized Newton's method (SCN). W…
Newton-LESS sparsifies Gaussian sketching for faster optimization.
problem Computing the Hessian matrix in optimization is computationally expensive.
method Uses a sparsified version of a dense Gaussian sketching matrix.
result Achieves nearly the same convergence rate as dense Gaussian embeddings without the computational cost.
New quasi-Newton method guarantees global superlinear convergence.
problem Global convergence and superlinear convergence of quasi-Newton methods.
method Hybrid proximal extragradient method with online learning for Hessian approximation.
result First globally convergent quasi-Newton method with explicit superlinear convergence rate.
Paper develops a robust PP distributed quasi-Newton estimation for Byzantine machines.
problem Byzantine machines in distributed computing under Privacy Protection constraints.
method Robust PP distributed quasi-Newton estimation method that transmits only five vectors.
result Reduces privacy budgeting and transmission cost compared to gradient descent and Newton iteration.
Four decades after their invention, quasi-Newton methods are still state of the art in unconstrained numerical optimization. Although not usually interpreted thus, these are learning algorithms that fit a local quadratic approximation to the objective function. We show that many, including the most popular, quasi-Newto…
A new quasi-Newton method uses cubic regularization to avoid saddle points in deep learning.
problem Avoiding saddle points and poor local minima in deep learning models.
method Limited-memory symmetric rank-one quasi-Newton approach with adaptive regularized cubics.
result The method effectively avoids saddle points and converges to better local minima.
Extends Newton's minimal resistance problem to Riemannian surfaces.
problem Minimal resistance on Riemannian surfaces.
method Derive resistance functional, analyze constrained minimization.
result Smooth extremals are loxodromes, global minimizers characterized.
A new Bayesian filtering method speeds up stochastic Newton optimization.
problem Minimizing log-convex functions using stochastic methods.
method Contextualizes the problem as Bayesian inference, applying Bayesian filtering to update estimates.
result Establishes conditions for diminishing effect of older observations, akin to momentum.
A Riemannian Optimization Perspective of the Gauss-Newton Method for Feedforward Neural Networksmath.OC
The Gauss-Newton method is analyzed for neural networks using Riemannian optimization techniques.
problem Training neural networks with smooth activations and convergence rates.
method Riemannian optimization perspective, analyzing the Gauss-Newton method in both underparameterized and overparameterized regimes.
result Geometric convergence rates independent of conditioning and eigenvalues, demonstrating accelerated convergence.
Extends Newton's minimal resistance problem to Lorentz-Minkowski space.
problem Minimal resistance in Lorentz-Minkowski space.
method Derived functional energy, determined Euler-Lagrange equation, analyzed maximum principle, found separable and radial solutions.
result Obtained solutions with conical singularities at the origin and analyzed the Single Shock Condition.
ISAAC Newton uses input-based curvature for efficient training.
problem Efficient training in small-batch stochastic regimes.
method ISAAC Newton conditions gradients using selected second-order information based on input.
result Effective training even in small-batch stochastic regimes, competitive to first-order and second-order methods.
Improved root-finding method for smooth functions.
problem Finding roots of smooth functions efficiently and reliably.
method A new improvement over Newton's method for doubly differentiable functions.
result Faster and more reliable convergence compared to Newton's method.
A new method for machine learning updates reduces complexity and improves robustness.
problem Stochastic gradient updates are inefficient and sensitive to feature scaling.
method Incremental Gauss-Newton Descent (IGND) reduces the need for matrix operations and improves robustness.
result IGND improves robustness to sensitivity scaling and can be competitive with common stochastic optimizers.
Paper proposes an efficient online Newton method with Nesterov's acceleration for streaming data.
problem Efficient inference of online Newton methods with robustness to noise and ill-conditioning.
method Online Newton method with Hessian averaging and Nesterov's accelerated sketch-and-project solver.
result Global almost-sure convergence and asymptotic normality of the last iterate with non-asymptotic convergence guarantees.
A distributed semismooth Newton based augmented Lagrangian method for distributed optimizationmath.OC
A new method solves distributed optimization problems over networks.
problem Solving optimization problems over networks with local cost functions and limited communication.
method Distributed semismooth Newton based augmented Lagrangian method.
result The method efficiently solves distributed optimization problems over networks.
In [19], a general, inexact, efficient proximal quasi-Newton algorithm for composite optimization problems has been proposed and a sublinear global convergence rate has been established. In this paper, we analyze the convergence properties of this method, both in the exact and inexact setting, in the case when the obje…
A new method for faster optimization on statistical manifolds.
problem Slow convergence of first-order methods in manifold optimization.
method Dual Riemannian Newton method on manifolds with dual connections.
result Local quadratic convergence of the dual Riemannian Newton method.
In this work, we present a globalized stochastic semismooth Newton method for solving stochastic optimization problems involving smooth nonconvex and nonsmooth convex terms in the objective function. We assume that only noisy gradient and Hessian information of the smooth part of the objective function is available via…
Recently algorithms incorporating second order curvature information have become popular in training neural networks. The Nesterov's Accelerated Quasi-Newton (NAQ) method has shown to effectively accelerate the BFGS quasi-Newton method by incorporating the momentum term and Nesterov's accelerated gradient vector. A sto…
Paper proposes a new method to efficiently incorporate curvature information in stochastic optimization.
problem Minimizing nonconvex functions with limited curvature information.
method Structured stochastic quasi-Newton method using partial Hessian information.
result Global convergence to stationary point and local superlinear convergence rate established.
For distributed computing environment, we consider the empirical risk minimization problem and propose a distributed and communication-efficient Newton-type optimization method. At every iteration, each worker locally finds an Approximate NewTon (ANT) direction, which is sent to the main driver. The main driver, then, …
A new method avoids saddle points in Newton's method.
problem Avoiding saddle points in optimization problems.
method New Q-Newton's method with specific update rule.
result The method guarantees convergence to a critical point that is not a saddle point.
A new hybrid Newton algorithm improves convergence in logistic regression.
problem Solving large-scale binary classification problems efficiently.
method Proposes a hybrid stochastic Newton algorithm with two weighted components in the Hessian matrix estimation.
result Proves almost sure convergence to the true parameter of logistic regression.