A new iterative K-FAC algorithm reduces training time and memory usage.
problem Training deep learning models efficiently.
method Uses conjugate gradient to approximate Fisher information matrix without generating the matrix or factors.
result Time and memory complexity of iterative CG-FAC is less than standard K-FAC.
Improved CNN training speed with K-FAC on large datasets.
problem Challenges in training large-scale neural networks with distributed processors.
method Scalable K-FAC design, layer-wise distribution, inverse-free second-order gradient evaluation, dynamic K-FAC update decoupling.
result Distributed K-FAC implementation converges to 75.9% MLPerf baseline in 18-25% less time than SGD.
K-FAC doesn't improve large batch training efficiency.
problem Inefficiency of K-FAC in large batch size training.
method Empirical analysis of K-FAC and SGD on ResNet and AlexNet.
result K-FAC doesn't exhibit improved scalability to large batch sizes.
AdamQLR optimizes Adam with K-FAC heuristics, achieving comparable performance to tuned benchmarks.
problem Improving the performance of Adam optimizers with stabilizing heuristics.
method Combining Adam's update directions with K-FAC's heuristics (damping and learning rate selection).
result Untuned AdamQLR can achieve comparable performance to tuned benchmarks.
K-FAC speeds up training of modern neural networks with linear weight-sharing.
problem Efficiently training modern neural networks with linear weight-sharing layers.
method Kronecker-Factored Approximate Curvature (K-FAC) applied to linear weight-sharing layers.
result K-FAC-reduce is generally faster than K-FAC-expand for deep linear networks.
This review explores automated pain detection from facial expressions using FACS.
problem Automated detection of pain from facial expressions is challenging and underexplored.
method Examines progress in AFER and APD, focusing on FACS-based methods and deep learning.
result Limited studies on APD, highlighting the need for clinical applications.
This paper speeds up K-FAC for deep learning by focusing on only a few eigen-modes.
problem Time-consuming computation of Kronecker factors in K-FAC for large layers.
method Theoretical analysis and randomized numerical linear algebra to approximate eigen-spectrum decay.
result Reduces time complexity from cubic to quadratic in layer width, improving efficiency.
We propose an efficient method for approximating natural gradient descent in neural networks which we call Kronecker-Factored Approximate Curvature (K-FAC). K-FAC is based on an efficiently invertible approximation of a neural network's Fisher information matrix which is neither diagonal nor low-rank, and in some cases…
Improved neural network inference with eigenvalue correction.
problem Inference of flexible variational posteriors is computationally expensive.
method Eigenvalue correction to matrix-variate Gaussian posterior.
result Empirically, the method outperforms existing algorithms.
The paper proposes a new way to approximate Riemannian metrics using discrete wall systems.
problem Approximating Riemannian metrics and proving geometric conjectures.
method Discretization of metrics using walls and triangulations.
result The discrete filling area conjecture is equivalent to Gromov's original conjecture.
Enhances Deep Hedging with K-FAC for financial data.
problem High computational burden in training neural networks for financial applications.
method Integrates Kronecker-Factored Approximate Curvature (K-FAC) optimization with LSTM networks.
result Significant improvements in convergence and hedging efficacy, reducing transaction costs and P&L variance.
Most neural networks are trained using first-order optimization methods, which are sensitive to the parameterization of the model. Natural gradient descent is invariant to smooth reparameterizations because it is defined in a coordinate-free way, but tractable approximations are typically defined in terms of coordinate…
Better Hessian approximations improve influence function attributions in deep learning.
problem Influence functions are difficult to compute due to ill-conditioned Hessians, leading to poor data attribution performance.
method Investigated the impact of Hessian approximation quality on influence-function attributions in a controlled setting.
result Better Hessian approximations consistently yield better influence score quality.
FEAFA dataset annotates facial expressions with high detail.
problem Lack of detailed facial expression annotations in existing datasets.
method Manual annotation of 122 participants' facial expressions.
result FEAFA dataset provides detailed annotations for facial expressions.
Improved continual learning for neural networks with BN layers using K-FAC extension.
problem Continual learning challenges in neural networks with BN layers.
method Extended K-FAC method to account for inter-example relations, weight merging, and reparameterization for BN layers; proposed weight merging and reparameterization for BN layers; proposed method to select hyperparameters without source task data.
result Better performance in continual learning tasks with BN layers compared to baselines.
Variational Bayesian neural nets combine the flexibility of deep learning with Bayesian uncertainty estimation. Unfortunately, there is a tradeoff between cheap but simple variational families (e.g.~fully factorized) or expensive and complicated inference procedures. We show that natural gradient ascent with adaptive w…
Three mechanisms of weight decay found for different optimizers and architectures.
problem Understanding the regularization effect of weight decay in neural networks.
method Empirical investigation of weight decay for SGD, Adam, and K-FAC with various network architectures.
result Identified three distinct mechanisms of weight decay effect.
New algorithm estimates matrix-valued regression parameters efficiently.
problem High-dimensional matrix regression with limited sample size.
method KRO-PRO-FAC algorithm using Kronecker product factorization.
result Algorithm provides accurate parameter estimates without covariance estimation.
Influence functions help study large language model generalization, revealing surprising decay patterns.
problem Understanding and mitigating risks in large language models (LLMs).
method Eigenvalue-corrected Kronecker-Factored Approximation (EK-FAC) to scale influence functions to LLMs.
result Influences decay to near-zero when key phrases order is flipped, revealing a surprising limitation.
Study uses a bivariate model to price crude oil futures.
problem Pricing crude oil futures using latent factors and state-space models.
method Modelled short and long term factors as OU processes, estimated using Kalman Filter and maximised Gaussian likelihood.
result Successfully estimated model parameters and factors from WTI Crude Oil NYMEX futures data.
In this technical report, we consider an approach that combines the PPO objective and K-FAC natural gradient optimization, for which we call PPOKFAC. We perform a range of empirical analysis on various aspects of the algorithm, such as sample complexity, training speed, and sensitivity to batch size and training epochs…
The paper rethinks the use of exponential averaging in machine learning optimization.
problem The inefficiency of using exponential averaging in optimization algorithms.
method The paper connects EA-CM algorithms to Wake of Quadratic regularized models and proposes new algorithms, KLD-WRM.
result The new algorithms outperform existing methods like K-FAC on MNIST.
New comparison theorem for submanifolds with geometric inequalities.
problem Geometric inequalities for submanifolds in ambient spaces.
method Explicit Jacobian determinant formula for normal exponential map.
result Establishes new comparison theorem related to Heintze-Karcher's.
Recurrent Neural Networks (RNNs) are powerful models that achieve exceptional performance on several pattern recognition problems. However, the training of RNNs is a computationally difficult task owing to the well-known "vanishing/exploding" gradient problem. Algorithms proposed for training RNNs either exploit no (or…
Study shows how algorithmic choices affect optimal batch sizes in neural networks.
problem Understanding how batch size impacts neural network training efficiency.
method Experiments and analysis of a simple quadratic model to study algorithmic choices.
result Preconditioned optimizers like Adam and K-FAC allow larger batch sizes before diminishing returns.
Online advertisement is the main source of revenue for Internet business. Advertisers are typically ranked according to a score that takes into account their bids and potential click-through rates(eCTR). Generally, the likelihood that a user clicks on an ad is often modeled by optimizing for the click through rates rat…
The paper proves formulas for capillary surfaces and applies them to inequalities and area estimates.
problem Understanding capillary surfaces and their properties.
method Established monotonicity formulas for capillary surfaces in half-space and unit ball.
result Extended Li-Yau-type inequalities and optimal area estimates for capillary surfaces.
Natural gradient descent speeds up convergence in neural networks, especially with overparameterization.
problem Mitigating the effects of curvature in neural network optimization.
method Analysis of natural gradient descent on nonlinear neural networks with stability conditions.
result Natural gradient descent converges efficiently under specific conditions for overparameterized networks.
Second-order optimization methods such as natural gradient descent have the potential to speed up training of neural networks by correcting for the curvature of the loss function. Unfortunately, the exact natural gradient is impractical to compute for large models, and most approximations either require an expensive it…
FedFisher improves one-shot FL by using Fisher information.
problem Reducing communication rounds in federated learning.
method Bayesian perspective, Fisher information matrices, diagonal Fisher, K-FAC approximation.
result FedFisher achieves vanishingly small error in two-layer neural networks.
This study explains why approximate NGD works well in wide neural networks.
problem Understanding why NGD with approximate Fisher information converges fast in wide neural networks.
method Analyzing asymptotic training dynamics in function space via the neural tangent kernel.
result NGD with approximate Fisher information achieves the same fast convergence as exact NGD under specific conditions.
Patient pain can be detected highly reliably from facial expressions using a set of facial muscle-based action units (AUs) defined by the Facial Action Coding System (FACS). A key characteristic of facial expression of pain is the simultaneous occurrence of pain-related AU combinations, whose automated detection would …
A new geometric concept, the dead direction, bridges singular learning theory and information geometry.
problem The gap between singular learning theory and information geometry.
method Introducing the dead direction, a unit vector along degenerating Fisher metric, and showing its KL order can be recovered.
result The KL order of the dead direction can be recovered as the decay rate of the directional Fisher curvature, providing a handle on singular geometry.