Second-order optimizers retain residual information after data deletion, affecting machine unlearning.
problem Residual information in second-order optimizers after data deletion.
method Comparison of first-order and second-order learners, eigendecomposition analysis.
result Second-order optimizers retain residual information, not detectable by first-order analysis.
Data whitening and second order optimization harm generalization by reducing access to dataset information.
problem Harmful effects of data whitening and second order optimization on generalization in machine learning.
method Analysis of fully connected models and experimental verification.
result Data whitening and second order optimization reduce or prevent generalization by limiting access to dataset information.
Negative step sizes improve second-order methods for neural networks.
problem Second-order methods discard negative curvature, limiting their effectiveness.
method Introduce negative step sizes in second-order methods combined with Wolfe line search.
result Negative step sizes lead to global convergence and improved performance.
AdaSub optimizes with second-order info in low-dims subspace.
problem Efficiently use second-order optimization methods with low computational cost.
method Adaptive subspace selection for second-order optimization.
result AdaSub outperforms other stochastic optimizers in time and iterations.
Enhances SMC² with Hessian info for more efficient posterior approximation.
problem Improving accuracy and efficiency in Bayesian inference.
method Integrates second-order information (Hessian) into SMC²'s proposal distribution.
result Second-order proposals lead to more accurate posterior approximations and better step-size selection.
A new method for faster optimization on statistical manifolds.
problem Slow convergence of first-order methods in manifold optimization.
method Dual Riemannian Newton method on manifolds with dual connections.
result Local quadratic convergence of the dual Riemannian Newton method.
Natural gradient descent, which preconditions a gradient descent update with the Fisher information matrix of the underlying statistical model, is a way to capture partial second-order information. Several highly visible works have advocated an approximation known as the empirical Fisher, drawing connections between ap…
The paper improves ODE solvers by integrating diverse information types.
problem Improving accuracy and physical meaningfulness of ODE solutions.
method Leveraging probabilistic solvers to include second-order information and physical conservation laws.
result Solutions become more accurate and physically meaningful with additional information.
Improved SVRG method using BB techniques for faster convergence.
problem Improving the convergence speed of stochastic variance reduction methods.
method Incorporates Barzilai-Borwein (BB) techniques as second-order information into SVRG.
result Proves linear convergence of the proposed method and its variants.
New algorithm finds approximate stationary points in non-convex optimization.
problem Finding approximate stationary points in non-convex stochastic optimization.
method Design of an algorithm using O(ε−3) stochastic gradient and Hessian-vector products. result Optimal rate of O(ε−3) for finding ε-approximate stationary points, matching lower bounds. We introduce TrustVI, a fast second-order algorithm for black-box variational inference based on trust-region optimization and the reparameterization trick. At each iteration, TrustVI proposes and assesses a step based on minibatches of draws from the variational distribution. The algorithm provably converges to a stat…
First-order stochastic methods are the state-of-the-art in large-scale machine learning optimization owing to efficient per-iteration complexity. Second-order methods, while able to provide faster convergence, have been much less explored due to the high cost of computing the second-order information. In this paper we …
Two classes of methods have been proposed for escaping from saddle points with one using the second-order information carried by the Hessian and the other adding the noise into the first-order information. The existing analysis for algorithms using noise in the first-order information is quite involved and hides the es…
Improved robustness in optimization methods using second-order information.
problem Scalability and sensitivity to mini-batch size in optimization methods.
method Mini-Batch Stochastic Variance-Reduced Newton (extttMb−SVRN) algorithm incorporating partial second-order information. result Achieves a fast linear convergence rate independent of mini-batch size for large data sizes.
Zeroth-order optimization is an important research topic in machine learning. In recent years, it has become a key tool in black-box adversarial attack to neural network based image classifiers. However, existing zeroth-order optimization algorithms rarely extract second-order information of the model function. In this…
Second-order methods improve differential privacy in convex optimization.
problem Improving differential privacy in convex optimization.
method Developed a private variant of the regularized cubic Newton method for strongly convex loss functions.
result Achieves quadratic convergence and optimal excess loss for strongly convex loss functions.
Improved computational complexity in statistical models using second-order information.
problem Polynomial convergence of gradient descent in singular statistical models.
method Normalized Gradient Descent (NormGD) algorithm with second-order information.
result NormGD reaches final statistical radius in logarithmic iterations of n. New quantum states capture more information, enabling advanced processing tasks.
problem Quantum information processing challenges with limited statistical information.
method Introducing Random-Coefficient Pure States (RCPS) and exploiting their higher-order statistics.
result RCPS provide richer information than density operators, enabling new quantum tasks.
Policy optimization on high-dimensional continuous control tasks exhibits its difficulty caused by the large variance of the policy gradient estimators. We present the action subspace dependent gradient (ASDG) estimator which incorporates the Rao-Blackwell theorem (RB) and Control Variates (CV) into a unified framework…
Paper proposes a new method for efficient second-order neural network training.
problem Infeasibility of Hessian calculation and noisy second-order information in deep learning.
method Adopting complex-step directional derivative (CSFD) for accurate Hessian computation and designing an effective Newton Krylov procedure.
result Our method outperforms existing methods and often converges one-order faster.
Paper introduces FoMoH for optimization without backpropagation.
problem Optimizing machine learning models without backpropagation.
method Second-order hyperplane search, forward-mode stochastic gradient method, hyper-dual numbers, FoMoH.
result Developed a novel optimization algorithm that avoids backpropagation.
A new method improves ICA performance by approximating MDI.
problem Improving F astICA's performance with nonlinear functions.
method Second-order approximation of MDI for joint maximization.
result Efficiency validated through experiments compared to other ICA algorithms.
We study 3-manifolds in R5 with corank 1 singularities. At the singular point we define the curvature locus using the first and second fundamental forms, which contains all the local second order geometrical information about the manifold.
Finite-sum optimization problems are ubiquitous in machine learning, and are commonly solved using first-order methods which rely on gradient computations. Recently, there has been growing interest in \emph{second-order} methods, which rely on both gradients and Hessians. In principle, second-order methods can require …
Stein variational gradient descent (SVGD) was recently proposed as a general purpose nonparametric variational inference algorithm [Liu & Wang, NIPS 2016]: it minimizes the Kullback-Leibler divergence between the target distribution and its approximation by implementing a form of functional gradient descent on a reprod…
SONIA optimizes machine learning problems with a novel algorithm.
problem Empirical risk minimization in machine learning.
method Symmetric Blockwise Truncated Optimization (SONIA) algorithm combining second-order and steepest descent steps.
result SONIA converges to stationary points in both convex and nonconvex cases.
A new method for optimizing deep neural networks using TKFAC.
problem Optimizing deep neural networks with second-order methods.
method Proposes Trace-restricted Kronecker-factored Approximate Curvature (TKFAC) for Fisher information matrix approximation.
result TKFAC improves performance on deep network architectures compared to state-of-the-art algorithms.
This work improves online SGD's sample complexity for multi-index models by considering higher-order terms.
problem Suboptimal sample complexity for learning multi-index models using online SGD.
method Focus on both second- and higher-order terms to improve sample complexity.
result Online SGD achieves ildeO(dPL−1) samples for multi-index models. We study the convergence of a class of gradient-based Model-Agnostic Meta-Learning (MAML) methods and characterize their overall complexity as well as their best achievable accuracy in terms of gradient norm for nonconvex loss functions. We start with the MAML method and its first-order approximation (FO-MAML) and high…
This paper proposes a framework of L-BFGS based on the (approximate) second-order information with stochastic batches, as a novel approach to the finite-sum minimization problems. Different from the classical L-BFGS where stochastic batches lead to instability, we use a smooth estimate for the evaluations of the gradie…
A new algorithm for solving constrained convex optimization problems efficiently.
problem Constrained convex optimization problems requiring high accuracy solutions.
method Second-Order Conditional Gradient Sliding (SOCGS) algorithm, using projection-free methods to solve quadratic subproblems inexactly.
result Converges quadratically in primal gap after a finite number of linearly convergent iterations.
In this paper, we try to uncover the second-order essence of several first-order optimization methods. For Nesterov Accelerated Gradient, we rigorously prove that the algorithm makes use of the difference between past and current gradients, thus approximates the Hessian and accelerates the training. For adaptive method…
Optimistic method adapted for faster convex-concave min-max problems.
problem Solving convex-concave min-max optimization problems efficiently.
method Adaptive, line search-free second-order methods combining optimistic updates and second-order information.
result Achieves optimal convergence rate without line search or backtracking.
DEO uses gradient information to escape saddle points in neural networks.
problem Training deep neural networks struggles with flat regions and saddle points.
method Dimer-Enhanced Optimization (DEO) uses gradient information to estimate curvature and escape saddle points.
result DEO improves training efficiency and performance compared to standard first-order methods.
Online learning with limited information feedback (bandit) tries to solve the problem where an online learner receives partial feedback information from the environment in the course of learning. Under this setting, Flaxman et al.[8] extended Zinkevich's classical Online Gradient Descent (OGD) algorithm [29] by proposi…
COMRADE is a communication-efficient, Byzantine-resilient second-order optimization algorithm.
problem Byzantine failures in distributed optimization.
method COMRADE is a communication-efficient, second-order optimization algorithm that uses a simple norm-based thresholding rule to filter out Byzantine workers.
result COMRADE achieves linear-quadratic convergence and is robust against Byzantine workers.
First-order methods such as stochastic gradient descent (SGD) are currently the standard algorithm for training deep neural networks. Second-order methods, despite their better convergence rate, are rarely used in practice due to the prohibitive computational cost in calculating the second-order information. In this pa…
Model improves information transfer from visual streams.
problem Challenges in unsupervised learning from continuous visual data.
method Inspired by physics, maximizes mutual information through temporal process.
result Focus of attention enhances information transfer from input stream.
Study proposes BFEL framework for privacy-preserving FL in personalized healthcare.
problem Privacy and security concerns in traditional cloud-centric ML, especially in wearable devices.
method Develops a blockchain-enhanced federated edge learning (BFEL) framework based on FedCurv, incorporating fisher information matrix and public key encryption.
result Significant reduction in communication cost and high efficiency for federated training on non-iid and heterogeneous data.
Study shows how macroeconomic news affects intraday price and order flow dynamics.
problem Understanding how macroeconomic news impacts intraday price and order flow dynamics.
method Structural VAR model identified through heteroskedasticity, estimated at one-second frequency for each 15-minute interval.
result Macroeconomic news announcements reshape price-flow dynamics, with significant impacts on price and flow impacts at the one-second horizon.
ISAAC Newton uses input-based curvature for efficient training.
problem Efficient training in small-batch stochastic regimes.
method ISAAC Newton conditions gradients using selected second-order information based on input.
result Effective training even in small-batch stochastic regimes, competitive to first-order and second-order methods.
The paper classifies second-order superintegrable systems with torsion and semi-degeneracy.
problem Classifying second-order superintegrable systems with torsion and semi-degeneracy.
method Information-geometric structure and geometric conditions for non-degeneracy.
result A (n+1)-parameter potential is non-degenerate if a certain trace-free tensor field vanishes. Paper models limit order book with informed traders and market makers.
problem Modeling the limit order book with heterogeneous market participants.
method Agent-based model with four types of participants: informed traders, noise traders, informed market makers, and noise market makers. Based on Glosten-Milgrom and Huang-Rosenbaum-Saliba approaches.
result Derived the static limit order book characteristics and compared them with existing models.
New method provides tighter robustness guarantees for adversarial attacks.
problem Ensuring robustness against adversarial attacks in machine learning models.
method Developed a Second-order Smoothing (SoS) robustness certificate using Gaussian random smoothing.
result SoS certificates are tighter and provide improved robustness on high-dimensional datasets.
We propose a novel general algorithm LHAC that efficiently uses second-order information to train a class of large-scale l1-regularized problems. Our method executes cheap iterations while achieving fast local convergence rate by exploiting the special structure of a low-rank matrix, constructed via quasi-Newton approx…
Incorporating second order curvature information in gradient based methods have shown to improve convergence drastically despite its computational intensity. In this paper, we propose a stochastic (online) quasi-Newton method with Nesterov's accelerated gradient in both its full and limited memory forms for solving lar…
In this paper, we propose and analyze zeroth-order stochastic approximation algorithms for nonconvex and convex optimization, with a focus on addressing constrained optimization, high-dimensional setting and saddle-point avoiding. To handle constrained optimization, we first propose generalizations of the conditional g…
Optimizes curves on Riemannian manifolds to minimize curvature.
problem Minimizing curvature on curves with fixed length and endpoints on Riemannian manifolds.
method Solves a second order ODE system derived from the optimization problem.
result Solutions to the optimization problem satisfy a second order ODE system.