EigenDamage reduces neural network size and FLOPs with structured pruning in the Kronecker-Factored Eigenbasis.
problem Reducing neural network size and FLOPs while maintaining accuracy for resource-constrained devices.
method Kronecker-Factored Eigenbasis reparameterization and Hessian-based structured pruning.
result Empirically validated improvements in model size and FLOPs with negligible accuracy loss.
This paper investigates Shampoo's heuristics and decouples preconditioner updates.
problem Improving Shampoo's heuristics for training neural networks.
method Decomposing preconditioner updates, correcting eigenvalues, and adapting eigenbasis computation frequency.
result Principled techniques to remove Shampoo's heuristics and improve training algorithms.
Optimization algorithms that leverage gradient covariance information, such as variants of natural gradient descent (Amari, 1998), offer the prospect of yielding more effective descent directions. For models with many parameters, the covariance matrix they are based on becomes gigantic, making them inapplicable in thei…
Variational Bayesian neural networks combine the flexibility of deep learning with Bayesian uncertainty estimation. However, inference procedures for flexible variational posteriors are computationally expensive. A recently proposed method, noisy natural gradient, is a surprisingly simple method to fit expressive poste…
A new method for optimizing deep neural networks using TKFAC.
problem Optimizing deep neural networks with second-order methods.
method Proposes Trace-restricted Kronecker-factored Approximate Curvature (TKFAC) for Fisher information matrix approximation.
result TKFAC improves performance on deep network architectures compared to state-of-the-art algorithms.
SINGD improves KFAC for memory-efficiency and stability in low-precision training.
problem Memory inefficiency and numerical instability of KFAC in low-precision training.
method Formulated inverse-free KFAC update and imposed structures in Kronecker factors.
result SINGD is memory-efficient and numerically robust, often outperforming AdamW in half precision.
Bayesian methods estimate regression functions on submanifolds using graph Laplacian eigenbasis.
problem Estimating regression functions on unknown smooth submanifolds.
method Random geometric graph structure, Bayesian priors based on random basis expansion in graph Laplacian eigenbasis.
result Posterior contraction rates are minimax optimal for any positive smoothness index.
Revisits Gaussian process model with spherical harmonics for scalable deep learning.
problem Scaling Gaussian process models to large input dimensions with high frequency learning.
method Introduces new kernels related to deep models, variational learning of spherical harmonic phases, and sparseness in eigenbasis.
result Enables scaling to larger input dimensions and learning of high frequency variations.
A new iterative K-FAC algorithm reduces training time and memory usage.
problem Training deep learning models efficiently.
method Uses conjugate gradient to approximate Fisher information matrix without generating the matrix or factors.
result Time and memory complexity of iterative CG-FAC is less than standard K-FAC.
New methods use Kronecker-factored approximations for faster deep learning optimization.
problem Optimizing deep learning models with rich curvature information.
method Approximate Hessian using Kronecker products for efficient quasi-Newton methods.
result New methods outperform first-order methods and perform comparably to second-order methods.
A key challenge for gradient based optimization methods in model-free reinforcement learning is to develop an approach that is sample efficient and has low variance. In this work, we apply Kronecker-factored curvature estimation technique (KFAC) to a recently proposed gradient estimator for control variate optimization…
We introduce the Kronecker factored online Laplace approximation for overcoming catastrophic forgetting in neural networks. The method is grounded in a Bayesian online learning framework, where we recursively approximate the posterior after every task with a Gaussian, leading to a quadratic penalty on changes to the we…
Second-order optimization methods such as natural gradient descent have the potential to speed up training of neural networks by correcting for the curvature of the loss function. Unfortunately, the exact natural gradient is impractical to compute for large models, and most approximations either require an expensive it…
Paper proposes a statistical test for transfer learning in linear regression.
problem Theoretical framework for parameter transfer in linear regression.
method Developed a statistical test to predict transfer quality.
result The test can predict if a fine-tuned model has lower prediction risk.
Improved continual learning for neural networks with BN layers using K-FAC extension.
problem Continual learning challenges in neural networks with BN layers.
method Extended K-FAC method to account for inter-example relations, weight merging, and reparameterization for BN layers; proposed weight merging and reparameterization for BN layers; proposed method to select hyperparameters without source task data.
result Better performance in continual learning tasks with BN layers compared to baselines.
K-FAC speeds up training of modern neural networks with linear weight-sharing.
problem Efficiently training modern neural networks with linear weight-sharing layers.
method Kronecker-Factored Approximate Curvature (K-FAC) applied to linear weight-sharing layers.
result K-FAC-reduce is generally faster than K-FAC-expand for deep linear networks.
Enhances Deep Hedging with K-FAC for financial data.
problem High computational burden in training neural networks for financial applications.
method Integrates Kronecker-Factored Approximate Curvature (K-FAC) optimization with LSTM networks.
result Significant improvements in convergence and hedging efficacy, reducing transaction costs and P&L variance.
Develops efficient quasi-Newton methods for training deep neural networks.
problem Training deep neural networks with large-scale Hessian matrices.
method Approximates Hessian as block-diagonal Kronecker product of smaller matrices, applies damping.
result Outperforms or matches state-of-the-art methods in autoencoder models.
This paper speeds up K-FAC for deep learning by focusing on only a few eigen-modes.
problem Time-consuming computation of Kronecker factors in K-FAC for large layers.
method Theoretical analysis and randomized numerical linear algebra to approximate eigen-spectrum decay.
result Reduces time complexity from cubic to quadratic in layer width, improving efficiency.
A new method for learning Bayesian neural networks using layerwise inference.
problem Learning Bayesian neural networks efficiently and accurately.
method Bayesian layerwise inference, treating neural networks as stacked Bayesian linear models, with pseudo-targets defined by backpropagated gradients.
result The method converges quickly and performs well on various benchmarks.
Method learns invariances in deep nets without human validation.
problem Manual selection of data augmentation parameters is cumbersome.
method Differentiable Laplace approximation for Bayesian model selection.
result Method successfully recovers invariances and improves generalization.
We propose an efficient method for approximating natural gradient descent in neural networks which we call Kronecker-Factored Approximate Curvature (K-FAC). K-FAC is based on an efficiently invertible approximation of a neural network's Fisher information matrix which is neither diagonal nor low-rank, and in some cases…
LQF linearizes deep models for better interpretability.
problem Lack of interpretability in deep neural networks.
method Simple modifications to architecture, loss function, and optimization.
result Comparable performance to non-linear fine-tuning, with interpretability.
The aim of this paper is to study a possible "boundary phenomenon" for Spinc Dirac operators in a special case. If you parametrise Spinc Dirac operators by a family of connections on a Spinc 4-manifold with boundary, this boundary inherits also a family of Spinc Dirac operators which has a spectral section (in the sens…
Despite all the impressive advances of recurrent neural networks, sequential data is still in need of better modelling. Truncated backpropagation through time (TBPTT), the learning algorithm most widely used in practice, suffers from the truncation bias, which drastically limits its ability to learn long-term dependenc…
SCaLE tackles dynamic regret in noisy bandit feedback with switching costs.
problem Unbounded metric movement costs in bandit online convex optimization.
method SCaLE algorithm for high-dimensional dynamic quadratic hitting costs and ℓ2-norm switching costs, with spectral regret analysis. result First algorithm achieving sub-linear dynamic regret without hitting cost knowledge.
This paper studies iteration convergence of Kronecker graphical lasso (KGLasso) algorithms for estimating the covariance of an i.i.d. Gaussian random sample under a sparse Kronecker-product covariance model and MSE convergence rates. The KGlasso model, originally called the transposable regularized covariance model by …
In this technical report, we consider an approach that combines the PPO objective and K-FAC natural gradient optimization, for which we call PPOKFAC. We perform a range of empirical analysis on various aspects of the algorithm, such as sample complexity, training speed, and sensitivity to batch size and training epochs…
New spectral clustering method using LASSO regularization for robust graph partitioning.
problem Lack of theoretical guarantees for spectral clustering on general graph models.
method 1-spectral clustering on a new random model with LASSO regularization.
result Effective and robust to small noise perturbations, validated by simulations and real data.
Most neural networks are trained using first-order optimization methods, which are sensitive to the parameterization of the model. Natural gradient descent is invariant to smooth reparameterizations because it is defined in a coordinate-free way, but tractable approximations are typically defined in terms of coordinate…
Simplified kernel ridge regression with a conservation law.
problem Understanding the test risk and generalization of kernel ridge regression.
method Identification of a conservation law that limits KRR's learning ability, leading to simplified expressions for test risk.
result Transparency in test risk expressions through the conserved quantity in the kernel eigenbasis.
Large-scale distributed training of deep neural networks suffer from the generalization gap caused by the increase in the effective mini-batch size. Previous approaches try to solve this problem by varying the learning rate and batch size over epochs and layers, or some ad hoc modification of the batch normalization. W…
The paper studies non-integer curvature flows and proves convergence to spheres under specific conditions.
problem Analyzing the convergence of non-integer curvature flows on rotationally symmetric surfaces.
method Spectral theory of singular Sturm-Liouville operators to construct an eigenbasis and prove convergence.
result The flow converges to a round sphere if the focal points coincide at the poles, otherwise to a non-round Hopf sphere.
Novel Hilbert space Gaussian process improves sequential design accuracy and efficiency.
problem Efficiently implementing Gaussian process acquisition functions for expensive simulations.
method Proposed a truncated eigenbasis representation for closed-form evaluation of IMSE acquisition function.
result Significantly lower prediction error and reduced computation time compared to benchmarks.
We extend Kyle's model to include stochastic liquidity and multiple assets.
problem Modeling informed trading with stochastic liquidity and multiple assets.
method Developed a variational formulation and derived a matrix-valued martingale depth process.
result A linear-Gaussian equilibrium with stochastic matrix-valued price impact.
Covariance shrinkage via stochastic interpolation
problem High-dimensional covariance estimation
method Recasting shrinkage as empirical risk minimization
result Reduces statistical risk through scheduling, flow maps, and early stopping
Second-order optimization speeds up deep hedging for complex options.
problem Hedging exotic options with market frictions in realistic markets.
method Second-order optimization scheme leveraging pathwise differentiability and Kronecker-factoring.
result Our method optimizes the policy in 1/4 the steps of standard optimization.
ViViT efficiently computes curvature for deep networks without approximations.
problem Efficiently computing curvature for deep networks without approximations.
method Leverages the GGN's low-rank structure without further approximations.
result ViViT allows for efficient computation of eigenvalues, eigenvectors, and directional derivatives.
A key question in modern statistics is how to make fast and reliable inferences for complex, high-dimensional data. While there has been much interest in sparse techniques, current methods do not generalize well to data with nonlinear structure. In this work, we present an orthogonal series estimator for predictors tha…
A new optimization method reduces memory and compute requirements for deep learning.
problem Memory and compute constraints in second-order stochastic optimizers for deep learning.
method Proposes KrAD, a novel factorization to approximate inverse Fisher matrix without inversion, leading to KrADagrad.
result Improves performance over Shampoo for 32-bit precision and comparable/generalization on real datasets.
Self-supervised reward prediction improves RL in sparse reward settings.
problem Data efficiency and sparse reward signals in reinforcement learning.
method Learning a state representation for reward prediction and using it to shape rewards.
result Self-supervised reward prediction enhances RL algorithms in single-goal environments.
Improved CNN training speed with K-FAC on large datasets.
problem Challenges in training large-scale neural networks with distributed processors.
method Scalable K-FAC design, layer-wise distribution, inverse-free second-order gradient evaluation, dynamic K-FAC update decoupling.
result Distributed K-FAC implementation converges to 75.9% MLPerf baseline in 18-25% less time than SGD.
We develop a framework for analyzing extreme values in correlated financial data.
problem Quantifying and mitigating risk in complex financial systems.
method Developed a practical framework for handling finite, multivariate, and correlated time series in finance.
result We successfully analyze high-frequency stock returns using univariate extreme value tools.
Adaptive regularization improves neural network performance on small datasets.
problem Improving neural network performance on limited data.
method Adaptive regularization using a matrix-variate normal prior with a Kronecker product structure.
result The method leads to networks with smaller stable ranks and spectral norms, suggesting better generalization.
New methods improve Fisher Matrix approximations for neural networks at low cost.
problem High cost of solving Fisher Information Matrix (FIM) in neural networks.
method Direct minimization via Kronecker product singular value decomposition.
result Improved approximations to FIM provide more accurate and faster optimization.
A new method for spectral positional encodings in directed graphs using Hermitian block Krylov subspaces.
problem Challenges in spectral positional encodings for directed graphs, including computational complexity and gauge invariance issues.
method Learnable spectral positional encodings of the form hθ(Aq)R, computed in a Hermitian block Krylov subspace from sparse matrix-vector products. result The method is gauge-invariant and converges to the exact eigendecomposition oracle as the depth grows.
This paper tackles catastrophic forgetting in neural networks by providing a unified framework for regularization-based continual learning.
problem Catastrophic forgetting in neural networks trained sequentially on multiple tasks.
method Formulates regularization-based continual learning as a second-order Taylor approximation of the loss function, leading to a unified framework.
result Theoretical results indicate the importance of accurate approximation of the Hessian matrix for optimization and generalization.
Sketchy reduces memory and compute requirements for adaptive regularization in deep learning.
problem Prohibitive memory and running time for adaptive regularization methods in deep learning.
method Low-rank sketching approach using Frequent Directions (FD) to reduce memory and compute requirements.
result Efficient interpolation between resource requirements and degradation in regret guarantees with rank k.