Muon optimizer simplifies matrix optimization with spectral orthogonalization.
problem Matrix optimization challenges, especially with large condition numbers.
method Simplified Muon optimizer using spectral orthogonalization of gradients.
result Simplified Muon converges linearly with independent scalar sequences, outperforming gradient descent and Adam.
The paper analyzes and improves a deep learning optimization technique using matrix gradient orthogonality.
problem Improving deep learning training through more effective optimization methods.
method Develops a stochastic non-Euclidean trust-region gradient method for deep learning optimization.
result Proves state-of-the-art convergence results for the proposed algorithm in various scenarios.
Recurrent neural networks (RNNs) have been successfully used on a wide range of sequential data problems. A well known difficulty in using RNNs is the \textit{vanishing or exploding gradient} problem. Recently, there have been several different RNN architectures that try to mitigate this issue by maintaining an orthogo…
In this paper, we propose a scalable algorithm for spectral embedding. The latter is a standard tool for graph clustering. However, its computational bottleneck is the eigendecomposition of the graph Laplacian matrix, which prevents its application to large-scale graphs. Our contribution consists of reformulating spect…
Recurrent Neural Networks (RNNs) are designed to handle sequential data but suffer from vanishing or exploding gradients. Recent work on Unitary Recurrent Neural Networks (uRNNs) have been used to address this issue and in some cases, exceed the capabilities of Long Short-Term Memory networks (LSTMs). We propose a simp…
New differential geometry perspective on orthogonal RNNs.
problem Mitigating exploding and vanishing gradients in RNNs.
method Using tools from differential geometry, parameterizing vector fields via directional derivatives of scalar functions.
result Our approach achieves comparable or better results on benchmark tasks.
The paper shows how gradient flow on over-parametrized tensor decomposition behaves like deflation.
problem Understanding the training dynamics of gradient flow on tensor decomposition.
method Empirical observation and mathematical proof of gradient flow dynamics for orthogonally decomposable tensors.
result Gradient flow dynamics for orthogonally decomposable tensors follows a tensor deflation process, recovering all tensor components.
The paper tackles CF in CL by analyzing NTK overlap matrix and proposing OGD.
problem Catastrophic Forgetting in continual learning.
method Analysis of NTK overlap matrix, OGD with PCA.
result Proposes OGD to mitigate CF, supported by experiments.
Several variants of recurrent neural networks (RNNs) with orthogonal or unitary recurrent matrices have recently been developed to mitigate the vanishing/exploding gradient problem and to model long-term dependencies of sequences. However, with the eigenvalues of the recurrent matrix on the unit circle, the recurrent s…
Efficiently optimizes orthogonal and Stiefel matrices on parallel units.
problem Optimization over orthogonal groups on parallel units.
method CWY and T-CWY transforms for parametrization and optimization.
result CWY and T-CWY methods lead to convergence on parallel units.
Recently mean field theory has been successfully used to analyze properties of wide, random neural networks. It gave rise to a prescriptive theory for initializing feed-forward neural networks with orthogonal weights, which ensures that both the forward propagated activations and the backpropagated gradients are near $…
Gradient descent with large steps leads to chaotic parameter space and unpredictable outcomes.
problem Understanding the behavior of gradient descent with large step sizes in matrix factorization.
method Analyzing the fractal structure of the parameter space and deriving critical step sizes for convergence.
result Gradient descent with large steps exhibits chaotic behavior and sensitivity to initialization, creating a fractal boundary between converging and diverging minimizers.
SpecNet2 improves spectral embedding without orthogonalization, achieving better performance and efficiency.
problem Improving spectral embedding methods for better performance and efficiency.
method Optimizes an equivalent objective of the eigen-problem without orthogonalization, allowing separate row and column sampling.
result Local and global convergence of the new objective using batch-based gradient descent is proven, and improved performance and efficiency are demonstrated on simulated and image datasets.
LEGO estimates tangent spaces more robustly than LPCA in noisy data.
problem Estimating tangent spaces in high-noise settings.
method Spectral method using graph Laplacian eigenvectors and gradient orthogonization.
result LEGO yields more robust tangent space estimates than LPCA.
New paradigm for Neural ODEs stabilizes training and improves model performance.
problem Gradient vanishing-explosion problem in training deep neural networks.
method ODEtoODE: Nested system of flows with orthogonal group constraints.
result Strong convergence results and improved downstream models in reinforcement learning and supervised learning.
NS-RGS improves orthogonal group synchronization with faster convergence.
problem Orthogonal group synchronization from pairwise measurements.
method Newton-Schulz iteration for Riemannian gradient optimization.
result NS-RGS achieves linear convergence and near-optimal accuracy.
Study differential properties of matrix square roots in specific cases.
problem Understanding matrix square roots in semi-simple, symmetric, and orthogonal cases.
method Analysis of differential and metric structures of real square roots of matrices under specific conditions.
result Differential properties of matrix square roots in semi-simple, symmetric, and orthogonal cases.
MuonEq improves training of matrix-valued parameters by rebalancing momentum before orthogonalization.
problem Training matrix-valued parameters with orthogonalized-update optimizers like Muon.
method MuonEq introduces three lightweight pre-orthogonalization equilibration schemes: two-sided row/column normalization (RC), row normalization (R), and column normalization (C).
result Row/column normalization acts as a zeroth-order surrogate for whitening and improves the geometry seen by orthogonalization.
SketchOGD improves memory efficiency for continual learning.
problem Catastrophic forgetting in continual learning.
method Matrix sketching to compress model gradients.
result SketchOGD outperforms existing memory-efficient OGD variants.
Random Matrix Theory explains loss surface Hessians in neural networks.
problem Understanding the loss surfaces of neural networks.
method Investigation of local spectral statistics of neural network Hessians.
result Excellent agreement with Gaussian Orthogonal Ensemble statistics.
New convergence guarantees for learning with unknown nuisance parameters.
problem Learning problems with unknown nuisance parameters.
method Stochastic gradient optimization with Neyman orthogonality and approximately orthogonalized updates.
result Stochastic gradient algorithms can converge under conditions of nuisance parameters.
Corrected whitening restores orthogonality in high-dimensional spherical Gaussian mixtures.
problem In high-dimensional data, standard whitening fails to preserve orthogonality of mixture means.
method Derived exact limits for whitened means dot products using random matrix theory, constructed a corrected whitening matrix.
result Corrected whitening allows for improved estimation of spherical Gaussian mixtures in the large-dimensional regime.
The Hessian of neural networks can be decomposed into a sum of two matrices: (i) the positive semidefinite generalized Gauss-Newton matrix G, and (ii) the matrix H containing negative eigenvalues. We observe that for wider networks, minimizing the loss with the gradient descent optimization maneuvers through surfaces o…
Algorithm finds isotropy subgroups of orthogonal similarity on symmetric matrices.
problem Computing isotropy subgroups of orthogonal similarity on symmetric matrices.
method Algorithmic procedure solving a Toeplitz matrix equation.
result Structure of isotropy subgroups described.
Computes isotropy subgroups of orthogonal matrices acting on Hermitian matrices.
problem Computing isotropy subgroups of orthogonal matrices acting on Hermitian matrices.
method Algorithm for solving a matrix equation to compute isotropy subgroups.
result Computed isotropy subgroups of orthogonal matrices acting on Hermitian matrices.
Training recurrent neural networks (RNNs) is a hard problem due to degeneracies in the optimization landscape, a problem also known as vanishing/exploding gradients. Short of designing new RNN architectures, previous methods for dealing with this problem usually boil down to orthogonalization of the recurrent dynamics,…
In this paper, we prove that any complete shrinking gradient Kähler-Ricci solitons with positive orthogonal bisectional curvature must be compact. We also obtain a classification of the complete shrinking gradient Kähler-Ricci solitons with nonnegative orthogonal bisectional curvature.
This work proves the asymptotic freeness of layerwise Jacobians in MLPs with Haar orthogonal matrices.
problem Proving the asymptotic freeness of layerwise Jacobians in multilayer perceptrons (MLPs).
method Replacing each layer's parameter matrix with itself multiplied by a Haar orthogonal matrix, and using the invariance of the MLP.
result Proves the asymptotic freeness of layerwise Jacobians in MLPs with Haar orthogonal matrices.
Study optimizes estimation of orthogonal and rotation matrices from noisy data.
problem Estimating orthogonal and rotation matrices from noisy data.
method Iterative polar decomposition algorithm initialized by spectral methods.
result Algorithm achieves optimal error rate of $(1+o(1)) rac{σ^2 d(d-1)}{2np}$.
We develop matrix models for Grassmann, flag, and Stiefel manifolds.
problem Creating efficient models for Grassmann, flag, and Stiefel manifolds.
method Orthogonally-equivariant matrix submanifold models derived for each manifold.
result Exhaustive list of orthogonally-equivariant submanifold models for the lowest dimensions.
We present an intriguing discovery related to Random Fourier Features: in Gaussian kernel approximation, replacing the random Gaussian matrix by a properly scaled random orthogonal matrix significantly decreases kernel approximation error. We call this technique Orthogonal Random Features (ORF), and provide theoretical…
A new NMF model for co-clustering and data approximation.
problem Finding a low rank approximation for nonnegative data.
method Generalizes separability assumption for NMF, proposing Co-Separable NMF (CoS-NMF).
result CoS-NMF outperforms state-of-the-art methods in co-clustering and data approximation.
We orthogonalize the NSS model to condition and diagnose its ill-conditioned parameters.
problem The ill-conditioning of the NSS model's design matrix.
method Exact orthogonal reparametrization via QR decomposition.
result Orthogonalization isolates the conditioning structure and maintains fit uncertainty.
This paper considers the recovery of a rank r positive semidefinite matrix XXT∈Rn×n from m scalar measurements of the form yi:=aiTXXTai (i.e., quadratic measurements of X). Such problems arise in a variety of applications, including covariance sketching of high-dimensional data…
Equivalent tests for SGD batch size selection found.
problem Finding equivalent tests for adaptive batch size selection in SGD.
method Norm and inner product/orthogonality tests equivalence demonstration.
result Norm and inner product/orthogonality tests are equivalent under specific conditions.
Curious structure of special orthogonal, unitary, and symplectic groups as products of Grassmannians discovered.
problem Understanding the structure of special orthogonal, unitary, and symplectic groups.
method Expressing these groups as products of Grassmannians realized as involution matrices.
result Special orthogonal, special unitary, and symplectic groups can be expressed as products of their corresponding Grassmannians.
Deep networks without non-linearities are equivalent to shallow ones.
problem Training deep orthogonal linear networks with no non-linearity.
method Riemannian gradient descent and gradient descent on factorization.
result Training deep overparametrized networks is equivalent to shallow ones.
VRSGT algorithm reduces orthogonality constraints in decentralized optimization.
problem Decentralized optimization with orthogonality constraints.
method VRSGT algorithm with variance reduction and orthogonal techniques.
result VRSGT achieves convergence rate of O(1 / k) for orthogonality constraints.
NSA-Flow optimizes matrix representations for interpretability in complex data.
problem Balancing interpretability and model flexibility in high-dimensional data.
method Non-negative Stiefel Approximating Flow (NSA-Flow) unifies sparse matrix factorization and orthogonalization.
result NSA-Flow yields sparse, stable, and interpretable representations.
A new algorithm POGO optimizes thousands of orthogonal matrices efficiently.
problem Optimizing thousands of orthogonal constraints at scale is computationally expensive.
method Revisits Landing algorithm, uses modern adaptive optimizers, reduces hyperparameters.
result POGO optimizes thousands of orthogonal matrices in minutes, outperforming alternatives.
New algorithm speeds up group equivariant neural networks computations.
problem Challenging computations in group equivariant neural networks.
method Diagrammatic framework based on category theory for matrix multiplication.
result Exponential improvement in time complexity for matrix multiplication.
Pion optimizes LLMs by preserving weight matrix singular values.
problem Training large language models (LLMs) with standard optimizers leads to unstable weight matrices.
method Pion uses orthogonal transformations to update weight matrices, preserving their singular values.
result Pion offers a stable alternative to standard optimizers for LLM pretraining and finetuning.
New method relaxes PCA orthogonality constraints using explained variance of correlated components.
problem Difficulty in using PCA for sparse design due to orthogonality constraints and non-differentiable penalty.
method Introduce expvar(Y) to measure variance explained by correlated components, relax orthogonality constraints.
result Two expvar(Y) definitions suitable for block PCA formulations without orthogonality constraints.
Gradient descent outperforms ridge regression under certain covariance matrix decay conditions.
problem Comparing the performance of gradient descent and ridge regression in linear models.
method Investigated gradient descent and ridge regression for linear regression with random isotropic ground truth.
result Gradient descent outperforms ridge regression under specific covariance matrix decay conditions.
OSA overcomes instability in skipless Transformers.
problem Instability in skipless Transformers using Softmax Self-Attention.
method OSA parametrizes attention matrix to be orthogonal via skew-symmetric matrix exponential.
result OSA allows for training non-causal Transformers without skip connections and normalisation layers.
Vanishing and exploding gradients are two of the main obstacles in training deep neural networks, especially in capturing long range dependencies in recurrent neural networks~(RNNs). In this paper, we present an efficient parametrization of the transition matrix of an RNN that allows us to stabilize the gradients that …
Gradient descent biases towards stable rank networks for nearly-orthogonal data.
problem Understanding implicit bias in non-smooth neural networks trained by gradient descent.
method Analysis of two-layer ReLU and leaky ReLU networks trained by gradient descent on nearly-orthogonal data.
result Gradient descent biases towards networks with stable rank and uniform margin for nearly-orthogonal data.
A recent theoretical analysis shows the equivalence between non-negative matrix factorization (NMF) and spectral clustering based approach to subspace clustering. As NMF and many of its variants are essentially linear, we introduce a nonlinear NMF with explicit orthogonality and derive general kernel-based orthogonal m…