Two new inverse-free ELM algorithms for incremental and decremental learning are proposed.
problem Efficiently updating and removing multiple hidden nodes in ELM.
method Improved inverse-free recursive algorithms for Tikhonov regularization.
result Inverse-free algorithms for ELM with multiple hidden nodes and redundant nodes.
The inverse-free extreme learning machine (ELM) algorithm proposed in [4] was based on an inverse-free algorithm to compute the regularized pseudo-inverse, which was deduced from an inverse-free recursive algorithm to update the inverse of a Hermitian matrix. Before that recursive algorithm was applied in [4], its impr…
SINGD improves KFAC for memory-efficiency and stability in low-precision training.
problem Memory inefficiency and numerical instability of KFAC in low-precision training.
method Formulated inverse-free KFAC update and imposed structures in Kronecker factors.
result SINGD is memory-efficient and numerically robust, often outperforming AdamW in half precision.
Inversion-free natural gradient method for Riemannian manifolds.
problem Hindered by the need for Euclidean space, Fisher information matrix inversion, and computational cost.
method Intrinsic, inversion-free natural gradient method on Riemannian manifolds, using moving approximation of inverse FIM.
result Almost-sure convergence rates and sub-quadratic storage complexity for large-scale applications.
New method trains sparse Gaussian processes without matrix inversion.
problem Costly training of Gaussian processes at scale.
method Inverse-free approach using matmul-only natural-gradient updates.
result Significantly improved stability and convergence in training.
Paper develops IFTRR to solve sparse generalized eigenvalue problems efficiently.
problem Finding the leading eigenvector with at most k nonzero entries in sparse generalized eigenvalue problems.
method Inverse-free truncated Rayleigh-Ritz method (IFTRR) with a new truncation strategy.
result IFTRR efficiently finds the support set of the leading eigenvector for large scale problems.
In this paper, we present a scalable distributed implementation of the Sampled Limited-memory Symmetric Rank-1 (S-LSR1) algorithm. First, we show that a naive distributed implementation of S-LSR1 requires multiple rounds of expensive communications at every iteration and thus is inefficient. We then propose DS-LSR1, a …
Simplified optimization for structured matrices in deep learning.
problem Computational challenges in Riemannian submanifold optimization for structured symmetric positive-definite matrices.
method Proposed a generalized Riemannian normal coordinates that dynamically orthonormalizes the metric and converts the problem into an unconstrained Euclidean space problem.
result Simplified existing approaches for structured covariances and developed matrix-inverse-free 2nd-order optimizers for deep learning with low precision.
Gaussian random fields are a powerful tool for modeling environmental processes. For high dimensional samples, classical approaches for estimating the covariance parameters require highly challenging and massive computations, such as the evaluation of the Cholesky factorization or solving linear systems. Recently, Anit…
Improved CNN training speed with K-FAC on large datasets.
problem Challenges in training large-scale neural networks with distributed processors.
method Scalable K-FAC design, layer-wise distribution, inverse-free second-order gradient evaluation, dynamic K-FAC update decoupling.
result Distributed K-FAC implementation converges to 75.9% MLPerf baseline in 18-25% less time than SGD.
Solving a bilevel optimization problem is at the core of several machine learning problems such as hyperparameter tuning, data denoising, meta- and few-shot learning, and training-data poisoning. Different from simultaneous or multi-objective optimization, the steepest descent direction for minimizing the upper-level c…
Proposes neural networks for solving complex free boundary problems.
problem Solving free boundary and Stefan problems with complex interfaces.
method Physics-informed neural networks for approximating solutions and boundaries.
result Successfully approximates solutions and moving boundaries in various Stefan problems.
New algorithm tackles stochastic bilevel optimization under relaxed smoothness conditions.
problem Optimal algorithms for stochastic bilevel optimization under relaxed smoothness conditions.
method Introduces a novel fully single-loop and Hessian-inversion-free algorithmic framework for stochastic bilevel optimization.
result Demonstrates state-of-the-art oracle complexity results for multi-objective robust bilevel optimization.