A new method for optimizing deep neural networks using TKFAC.
problem Optimizing deep neural networks with second-order methods.
method Proposes Trace-restricted Kronecker-factored Approximate Curvature (TKFAC) for Fisher information matrix approximation.
result TKFAC improves performance on deep network architectures compared to state-of-the-art algorithms.
We introduce the Kronecker factored online Laplace approximation for overcoming catastrophic forgetting in neural networks. The method is grounded in a Bayesian online learning framework, where we recursively approximate the posterior after every task with a Gaussian, leading to a quadratic penalty on changes to the we…
Second-order optimization methods such as natural gradient descent have the potential to speed up training of neural networks by correcting for the curvature of the loss function. Unfortunately, the exact natural gradient is impractical to compute for large models, and most approximations either require an expensive it…
A new iterative K-FAC algorithm reduces training time and memory usage.
problem Training deep learning models efficiently.
method Uses conjugate gradient to approximate Fisher information matrix without generating the matrix or factors.
result Time and memory complexity of iterative CG-FAC is less than standard K-FAC.
New methods use Kronecker-factored approximations for faster deep learning optimization.
problem Optimizing deep learning models with rich curvature information.
method Approximate Hessian using Kronecker products for efficient quasi-Newton methods.
result New methods outperform first-order methods and perform comparably to second-order methods.
Improved continual learning for neural networks with BN layers using K-FAC extension.
problem Continual learning challenges in neural networks with BN layers.
method Extended K-FAC method to account for inter-example relations, weight merging, and reparameterization for BN layers; proposed weight merging and reparameterization for BN layers; proposed method to select hyperparameters without source task data.
result Better performance in continual learning tasks with BN layers compared to baselines.
Optimization algorithms that leverage gradient covariance information, such as variants of natural gradient descent (Amari, 1998), offer the prospect of yielding more effective descent directions. For models with many parameters, the covariance matrix they are based on becomes gigantic, making them inapplicable in thei…
Develops efficient quasi-Newton methods for training deep neural networks.
problem Training deep neural networks with large-scale Hessian matrices.
method Approximates Hessian as block-diagonal Kronecker product of smaller matrices, applies damping.
result Outperforms or matches state-of-the-art methods in autoencoder models.
This paper investigates Shampoo's heuristics and decouples preconditioner updates.
problem Improving Shampoo's heuristics for training neural networks.
method Decomposing preconditioner updates, correcting eigenvalues, and adapting eigenbasis computation frequency.
result Principled techniques to remove Shampoo's heuristics and improve training algorithms.
K-FAC speeds up training of modern neural networks with linear weight-sharing.
problem Efficiently training modern neural networks with linear weight-sharing layers.
method Kronecker-Factored Approximate Curvature (K-FAC) applied to linear weight-sharing layers.
result K-FAC-reduce is generally faster than K-FAC-expand for deep linear networks.
We propose an efficient method for approximating natural gradient descent in neural networks which we call Kronecker-Factored Approximate Curvature (K-FAC). K-FAC is based on an efficiently invertible approximation of a neural network's Fisher information matrix which is neither diagonal nor low-rank, and in some cases…
Enhances Deep Hedging with K-FAC for financial data.
problem High computational burden in training neural networks for financial applications.
method Integrates Kronecker-Factored Approximate Curvature (K-FAC) optimization with LSTM networks.
result Significant improvements in convergence and hedging efficacy, reducing transaction costs and P&L variance.
Reducing the test time resource requirements of a neural network while preserving test accuracy is crucial for running inference on resource-constrained devices. To achieve this goal, we introduce a novel network reparameterization based on the Kronecker-factored eigenbasis (KFE), and then apply Hessian-based structure…
Method learns invariances in deep nets without human validation.
problem Manual selection of data augmentation parameters is cumbersome.
method Differentiable Laplace approximation for Bayesian model selection.
result Method successfully recovers invariances and improves generalization.
SINGD improves KFAC for memory-efficiency and stability in low-precision training.
problem Memory inefficiency and numerical instability of KFAC in low-precision training.
method Formulated inverse-free KFAC update and imposed structures in Kronecker factors.
result SINGD is memory-efficient and numerically robust, often outperforming AdamW in half precision.
Despite all the impressive advances of recurrent neural networks, sequential data is still in need of better modelling. Truncated backpropagation through time (TBPTT), the learning algorithm most widely used in practice, suffers from the truncation bias, which drastically limits its ability to learn long-term dependenc…
A key challenge for gradient based optimization methods in model-free reinforcement learning is to develop an approach that is sample efficient and has low variance. In this work, we apply Kronecker-factored curvature estimation technique (KFAC) to a recently proposed gradient estimator for control variate optimization…
Most neural networks are trained using first-order optimization methods, which are sensitive to the parameterization of the model. Natural gradient descent is invariant to smooth reparameterizations because it is defined in a coordinate-free way, but tractable approximations are typically defined in terms of coordinate…
ViViT efficiently computes curvature for deep networks without approximations.
problem Efficiently computing curvature for deep networks without approximations.
method Leverages the GGN's low-rank structure without further approximations.
result ViViT allows for efficient computation of eigenvalues, eigenvectors, and directional derivatives.
This paper speeds up K-FAC for deep learning by focusing on only a few eigen-modes.
problem Time-consuming computation of Kronecker factors in K-FAC for large layers.
method Theoretical analysis and randomized numerical linear algebra to approximate eigen-spectrum decay.
result Reduces time complexity from cubic to quadratic in layer width, improving efficiency.
Better Hessian approximations improve influence function attributions in deep learning.
problem Influence functions are difficult to compute due to ill-conditioned Hessians, leading to poor data attribution performance.
method Investigated the impact of Hessian approximation quality on influence-function attributions in a controlled setting.
result Better Hessian approximations consistently yield better influence score quality.
A new method for learning Bayesian neural networks using layerwise inference.
problem Learning Bayesian neural networks efficiently and accurately.
method Bayesian layerwise inference, treating neural networks as stacked Bayesian linear models, with pseudo-targets defined by backpropagated gradients.
result The method converges quickly and performs well on various benchmarks.
A new optimization method reduces memory and compute requirements for deep learning.
problem Memory and compute constraints in second-order stochastic optimizers for deep learning.
method Proposes KrAD, a novel factorization to approximate inverse Fisher matrix without inversion, leading to KrADagrad.
result Improves performance over Shampoo for 32-bit precision and comparable/generalization on real datasets.
New methods improve Fisher Matrix approximations for neural networks at low cost.
problem High cost of solving Fisher Information Matrix (FIM) in neural networks.
method Direct minimization via Kronecker product singular value decomposition.
result Improved approximations to FIM provide more accurate and faster optimization.
This paper tackles catastrophic forgetting in neural networks by providing a unified framework for regularization-based continual learning.
problem Catastrophic forgetting in neural networks trained sequentially on multiple tasks.
method Formulates regularization-based continual learning as a second-order Taylor approximation of the loss function, leading to a unified framework.
result Theoretical results indicate the importance of accurate approximation of the Hessian matrix for optimization and generalization.
A new method for efficient uncertainty estimation in deep learning models.
problem High computational costs in uncertainty estimation for deep neural networks.
method Variational sparse Gaussian Process approximation of the Linearized Laplace Approximation.
result Sub-linear training time and improved performance compared to existing methods.
Large-scale distributed training of deep neural networks suffer from the generalization gap caused by the increase in the effective mini-batch size. Previous approaches try to solve this problem by varying the learning rate and batch size over epochs and layers, or some ad hoc modification of the batch normalization. W…
Improved CNN training speed with K-FAC on large datasets.
problem Challenges in training large-scale neural networks with distributed processors.
method Scalable K-FAC design, layer-wise distribution, inverse-free second-order gradient evaluation, dynamic K-FAC update decoupling.
result Distributed K-FAC implementation converges to 75.9% MLPerf baseline in 18-25% less time than SGD.
Scalable Gaussian Process Operator tackles high-dimensional PDEs.
problem Scaling Gaussian Process Operators to high-dimensional, data-intensive regimes.
method Nearest-neighbor-based local kernel approximations, sparse kernel approximation, structured Kronecker factorizations, operator-aware kernel structures, task-informed mean functions.
result Consistently achieves high accuracy across varying discretization scales.
LQF linearizes deep models for better interpretability.
problem Lack of interpretability in deep neural networks.
method Simple modifications to architecture, loss function, and optimization.
result Comparable performance to non-linear fine-tuning, with interpretability.
Second-order optimization speeds up deep hedging for complex options.
problem Hedging exotic options with market frictions in realistic markets.
method Second-order optimization scheme leveraging pathwise differentiability and Kronecker-factoring.
result Our method optimizes the policy in 1/4 the steps of standard optimization.
This paper studies iteration convergence of Kronecker graphical lasso (KGLasso) algorithms for estimating the covariance of an i.i.d. Gaussian random sample under a sparse Kronecker-product covariance model and MSE convergence rates. The KGlasso model, originally called the transposable regularized covariance model by …
In this technical report, we consider an approach that combines the PPO objective and K-FAC natural gradient optimization, for which we call PPOKFAC. We perform a range of empirical analysis on various aspects of the algorithm, such as sample complexity, training speed, and sensitivity to batch size and training epochs…
Influence functions help study large language model generalization, revealing surprising decay patterns.
problem Understanding and mitigating risks in large language models (LLMs).
method Eigenvalue-corrected Kronecker-Factored Approximation (EK-FAC) to scale influence functions to LLMs.
result Influences decay to near-zero when key phrases order is flipped, revealing a surprising limitation.
In stochastic optimization, using large batch sizes during training can leverage parallel resources to produce faster wall-clock training times per training epoch. However, for both training loss and testing error, recent results analyzing large batch Stochastic Gradient Descent (SGD) have found sharp diminishing retur…
Variational Bayesian neural networks combine the flexibility of deep learning with Bayesian uncertainty estimation. However, inference procedures for flexible variational posteriors are computationally expensive. A recently proposed method, noisy natural gradient, is a surprisingly simple method to fit expressive poste…
Self-supervised reward prediction improves RL in sparse reward settings.
problem Data efficiency and sparse reward signals in reinforcement learning.
method Learning a state representation for reward prediction and using it to shape rewards.
result Self-supervised reward prediction enhances RL algorithms in single-goal environments.
Feed-forward neural networks can be understood as a combination of an intermediate representation and a linear hypothesis. While most previous works aim to diversify the representations, we explore the complementary direction by performing an adaptive and data-dependent regularization motivated by the empirical Bayes m…
Sketchy reduces memory and compute requirements for adaptive regularization in deep learning.
problem Prohibitive memory and running time for adaptive regularization methods in deep learning.
method Low-rank sketching approach using Frequent Directions (FD) to reduce memory and compute requirements.
result Efficient interpolation between resource requirements and degradation in regret guarantees with rank k. New bounds for large language models using token properties.
problem Vacuous generalization bounds for large language models.
method Martingale properties and Monarch matrices.
result Non-vacuous generalization bounds for LLMs up to 70B parameters.
The study provides conditions for approximating Riemannian manifolds with polyhedral metrics.
problem Approximating Riemannian manifolds with polyhedral metrics.
method Conditions on curvature tensors for Lipschitz and local polyhedral approximations.
result Conditions are sufficient for local polyhedral approximations, conjectured to be sufficient for global approximations.
Geometric Gaussian approximations capture any distribution.
problem Approximating complex probability distributions.
method Geometric Gaussian approximations through diffeomorphisms or exponential maps.
result Geometric Gaussian approximations are universal, capturing any distribution.
Method approximates Riemannian barycenter on manifolds.
problem Computing the exact Riemannian barycenter is computationally expensive.
method Uses under- and over-approximations of Riemannian distance to compute an approximate barycenter.
result Approximation method is more efficient than exact methods and steepest descent.
We consider in this paper the optimal approximations of convex univariate functions with feed-forward Relu neural networks. We are interested in the following question: what is the minimal approximation error given the number of approximating linear pieces? We establish the necessary and sufficient conditions and uniqu…
Efficiently reduces tensor ranks using mean-field approximation.
problem Low-rank approximation of non-negative tensors.
method Mean-field approximation of tensor rank reduction.
result Our algorithm achieves faster and competitive tensor rank reduction.
Study approximates unknown function levels with queries.
problem Approximating unknown function levels through sequential queries.
method Introduce Bisect and Approximate algorithms to reduce to local function approximation.
result Rate-optimal sample complexity guarantees for H{ö}lder functions.
We study sparse approximate solutions to convex optimization problems. It is known that in many engineering applications researchers are interested in an approximate solution of an optimization problem as a linear combination of elements from a given system of elements. There is an increasing interest in building such …
Softmax attention approximates complex functions and subsumes many known universal approximators.
problem Universal approximation of continuous sequence-to-sequence functions.
method Interpolation-based analysis of attention's internal mechanism, showing its ability to approximate ReLU functions.
result Softmax attention is a universal approximator for continuous sequence-to-sequence functions.