arXiv research

One reflection suffices for orthogonal weights, reducing GPU usage.

Introduces new curvature concept for Kähler manifolds.

In the multiple linear regression setting, we propose a general framework, termed weighted orthogonal components regression (WOCR), which encompasses many known methods as special cases, including ridge regression and principal components regression. WOCR makes use of the monotonicity inherent in orthogonal components …

Pion optimizes LLMs by preserving weight matrix singular values.

Orthogonal initialization does not speed up training in ultra-wide neural networks.

Many scientific questions require estimating the effects of continuous treatments. Outcome modeling and weighted regression based on the generalized propensity score are the most commonly used methods to evaluate continuous effects. However, these techniques may be sensitive to model misspecification, extreme weights o…

Deep networks with orthogonal weights show stable fluctuations, improving generalization and training speed.

MuonEq improves training of matrix-valued parameters by rebalancing momentum before orthogonalization.

Wasserstein-GANs have been introduced to address the deficiencies of generative adversarial networks (GANs) regarding the problems of vanishing gradients and mode collapse during the training, leading to improved convergence behaviour and improved image quality. However, Wasserstein-GANs require the discriminator to be…

A new classifier uses weighted orthogonal regression for robust classification with limited data.

A well-conditioned Jacobian spectrum has a vital role in preventing exploding or vanishing gradients and speeding up learning of deep neural networks. Free probability theory helps us to understand and handle the Jacobian spectrum. We rigorously show almost sure asymptotic freeness of layer-wise Jacobians of deep neura…

This work proves the asymptotic freeness of layerwise Jacobians in MLPs with Haar orthogonal matrices.

Recently mean field theory has been successfully used to analyze properties of wide, random neural networks. It gave rise to a prescriptive theory for initializing feed-forward neural networks with orthogonal weights, which ensures that both the forward propagated activations and the backpropagated gradients are near $…

A new method improves continual learning by replaying pseudo data and using orthogonal weight modification.

Deep networks without non-linearities are equivalent to shallow ones.

Study connects curvature to graph theory and reveals differences.

OPT framework improves neural network generalization by learning an orthogonal transformation.

In this short note we show the following result: Let $(M^{2n+1},g)$ ($n \geq 2$) be a compact Sasaki manifold with positive transverse orthogonal bisectional curvature. Then $π_1(M)$ is finite, and the universal cover of $(M^{2n+1},g)$ is isomorphic to a weighted Sasaki sphere. We also get some results in the case of n…

New LT-O-learners improve HLTE estimation with low overlap.

Recurrent Neural Networks (RNNs) are designed to handle sequential data but suffer from vanishing or exploding gradients. Recent work on Unitary Recurrent Neural Networks (uRNNs) have been used to address this issue and in some cases, exceed the capabilities of Long Short-Term Memory networks (LSTMs). We propose a simp…

LOFT separates subspace rotation and transformation for orthogonal fine-tuning.

New method enforces orthogonality in convolutional layers for improved robustness.

We introduce a novel approach to perform first-order optimization with orthogonal and unitary constraints. This approach is based on a parametrization stemming from Lie group theory through the exponential map. The parametrization transforms the constrained optimization problem into an unconstrained one over a Euclidea…

In this paper, we introduce the algorithms of Orthogonal Deep Neural Networks (OrthDNNs) to connect with recent interest of spectrally regularized deep learning methods. OrthDNNs are theoretically motivated by generalization analysis of modern DNNs, with the aim to find solution properties of network weights that guara…

New algorithm speeds up group equivariant neural networks computations.

An algorithm for efficient computation of equivariant neural network layers.

The selection of initial parameter values for gradient-based optimization of deep neural networks is one of the most impactful hyperparameter choices in deep learning systems, affecting both convergence times and model performance. Yet despite significant empirical and theoretical analysis, relatively little has been p…

The paper proves smoothness of almost-minimizers' boundaries near the free boundary.

The paper proves a distribution claim for neural network Jacobians.

Effective features can improve the performance of a model, which can thus help us understand the characteristics and underlying structure of complex data. Previous feature selection methods usually cannot keep more local structure information. To address the defects previously mentioned, we propose a novel supervised o…

New survival learners estimate heterogeneous treatment effects from time-to-event data.

Paper refines Alesker-Bernig-Schuster theorem, proving Hodge-Riemann relations for Euclidean balls.

Recurrent neural networks (RNNs) have been successfully used on a wide range of sequential data problems. A well known difficulty in using RNNs is the \textit{vanishing or exploding gradient} problem. Recently, there have been several different RNN architectures that try to mitigate this issue by maintaining an orthogo…

GAIT-prop derives a biologically plausible learning rule from backpropagation.

This paper seeks to answer the question: as the (near-) orthogonality of weights is found to be a favorable property for training deep convolutional neural networks, how can we enforce it in more effective and easy-to-use ways? We develop novel orthogonality regularizations on training deep CNNs, utilizing various adva…

OPAA estimates probability densities using functional analysis.

The paper computes characteristic classes for Lie group representations.

We consider the problem of estimating a low-rank matrix from a noisy observed matrix. Previous work has shown that the optimal method depends crucially on the choice of loss function. In this paper, we use a family of weighted loss functions, which arise naturally for problems such as submatrix denoising, denoising wit…

This paper proposes a new evaluation metric and boosting method for weight separability in neural network design. In contrast to general visual recognition methods designed to encourage both intra-class compactness and inter-class separability of latent features, we focus on estimating linear independence of column vec…

The paper analyzes and improves a deep learning optimization technique using matrix gradient orthogonality.

Gradient-enhanced GSA uses Poincaré chaos expansions for accurate sensitivity analysis.

Recently, Neural networks have seen a huge surge in its adoption due to their ability to provide high accuracy on various tasks. On the other hand, the existence of adversarial examples have raised suspicions regarding the generalization capabilities of neural networks. In this work, we focus on the weight matrix learn…

Different neural networks trained on the same dataset often learn similar input-output mappings with very different weights. Is there some correspondence between these neural network solutions? For linear networks, it has been shown that different instances of the same network architecture encode the same representatio…

Regularization techniques are widely used to improve the generality, robustness, and efficiency of deep convolutional neural networks (DCNNs). In this paper, we propose a novel approach of regulating DCNN convolutional kernels by a structured filter bank. Comparing with the existing regularization methods, such as $\el…

In this paper, we study the dynamics of gradient descent in learning neural networks for classification problems. Unlike in existing works, we consider the linearly non-separable case where the training data of different classes lie in orthogonal subspaces. We show that when the network has sufficient (but not exceedin…

OrthoGrad improves neural calibration by constraining gradient updates orthogonally.

It is well known that the initialization of weights in deep neural networks can have a dramatic impact on learning speed. For example, ensuring the mean squared singular value of a network's input-output Jacobian is $O(1)$ is essential for avoiding the exponential vanishing or explosion of gradients. The stronger condi…

DFRot improves LLMs by reducing outlier and massive activation effects.