arXiv research

Negative step sizes improve second-order methods for neural networks.

Exact second-order optimization for deep learning reduces computational cost and improves performance.

SOLBP extends efficient inference to uncertain Bayesian networks.

Second-order methods improve differential privacy in convex optimization.

First-order stochastic methods are the state-of-the-art in large-scale machine learning optimization owing to efficient per-iteration complexity. Second-order methods, while able to provide faster convergence, have been much less explored due to the high cost of computing the second-order information. In this paper we …

New method explains predictive uncertainty by focusing on second-order effects.

AdamQLR optimizes Adam with K-FAC heuristics, achieving comparable performance to tuned benchmarks.

The second order method as Newton Step is a suitable technique in Online Learning to guarantee regret bound. The large data is a challenge in Newton method to store second order matrices as hessian. In this paper, we have proposed an modified online Newton step that store first and second order matrices of dimension m …

New methods using natural gradient for structured optimization.

A new method for faster optimization of noisy functions.

A new method for faster optimization in high dimensions.

Second-order optimizers retain residual information after data deletion, affecting machine unlearning.

Develops unbiased averaging methods for second order optimization in distributed systems.

New method speeds up deep learning optimization.

Paper proposes a new method for efficient second-order neural network training.

AdaSub optimizes with second-order info in low-dims subspace.

Finite-sum optimization problems are ubiquitous in machine learning, and are commonly solved using first-order methods which rely on gradient computations. Recently, there has been growing interest in \emph{second-order} methods, which rely on both gradients and Hessians. In principle, second-order methods can require …

A first-order model for a stock market assigns to each stock a return parameter and a variance parameter that depend only on the rank of the stock. A second-order model assigns these parameters based on both the rank and the name of the stock. First- and second-order models exhibit stability properties that make them a…

New algorithm finds approximate stationary points in non-convex optimization.

Paper introduces STSL, a second-order Tweedie sampler for efficient posterior sampling in inverse problems.

Paper proposes a second-order method for faster SVI convergence.

Efficient method classifies locally stationary time series based on second-order characteristics.

Optimistic method adapted for faster convex-concave min-max problems.

ADAHESSIAN optimizes machine learning models with adaptive second-order methods.

We apply the Cartan equivalence method to the study of real analytic second order ODEs under the local real analytic diffeomorphism of $\C^2$ which are area-preserving. This enables us to give a characterization of the second order ODEs which are equivalent to $y^{\prime\prime}=0$ under such transformations. Moreover w…

Proposes a new stock prediction method that accounts for market dynamics.

First-order methods such as stochastic gradient descent (SGD) are currently the standard algorithm for training deep neural networks. Second-order methods, despite their better convergence rate, are rarely used in practice due to the prohibitive computational cost in calculating the second-order information. In this pa…

Clarifies method of phase synchronization for decoupling linear differential equations.

Paper develops a TR-SSQP method for noisy optimization with heavy-tailed noise.

We use E. Cartan's method to solve the problem of equivalence of the second order ordinary differential equations with respect to the pseudogroup of point transformations.

New findings show second-order scoring rules can't accurately represent epistemic uncertainty.

Paper proposes a method to find approximate SOSP for nonconvex conic optimization problems.

Paper proposes an online covariance estimator for sketched Newton methods.

Enhances SMC² with Hessian info for more efficient posterior approximation.

Two new algorithms improve federated optimization under second-order similarity.

New algorithm reduces regret in stochastic bandit convex optimization.

Method studies equivalence of second order ODEs under specific transformations.

Paper introduces FoMoH for optimization without backpropagation.

A new method for faster optimization on statistical manifolds.

New pruning method captures global correlations for efficient neural network inference.

Clustering is fundamental for gaining insights from complex networks, and spectral clustering (SC) is a popular approach. Conventional SC focuses on second-order structures (e.g., edges connecting two nodes) without direct consideration of higher-order structures (e.g., triangles and cliques). This has motivated SC ext…

In this paper, we provide an overview of first-order and second-order variants of the gradient descent method that are commonly used in machine learning. We propose a general framework in which 6 of these variants can be interpreted as different instances of the same approach. They are the vanilla gradient descent, the…

A new optimization method improves deep learning accuracy without hyper-parameter tuning.

Trust region and cubic regularization methods have demonstrated good performance in small scale non-convex optimization, showing the ability to escape from saddle points. Each iteration of these methods involves computation of gradient, Hessian and function value in order to obtain the search direction and adjust the r…

New method solves stochastic optimization problems with random models.

Second-order methods fail to fully quantify epistemic uncertainty, leading to biased predictions.

TSCD is an algorithm for causal discovery using second-order statistics.

Single-site Markov Chain Monte Carlo (MCMC) is a variant of MCMC in which a single coordinate in the state space is modified in each step. Structured relational models are a good candidate for this style of inference. In the single-site context, second order methods become feasible because the typical cubic costs assoc…