arXiv research

New method enforces orthogonality in convolutional layers for improved robustness.

Derives new orthogonal coordinates for evolving surfaces and curves.

Orthogonium offers unified, efficient layers for robust deep learning.

An algorithm for efficient computation of equivariant neural network layers.

This work proves the asymptotic freeness of layerwise Jacobians in MLPs with Haar orthogonal matrices.

Characterizes group-equivariant neural networks for three groups.

Batch normalization makes deep neural networks' representations increasingly orthogonal.

Two-layer networks trained on low-dimensional subspaces are vulnerable to adversarial examples.

Deep networks without non-linearities are equivalent to shallow ones.

Study shows RFRR's effectiveness with nearly orthogonal data in overparameterized settings.

A new method for disentangled representations without supervision.

LOT improves adversarial robustness by training 1-Lipschitz convolution layers.

Batch normalization has become ubiquitous in many state-of-the-art nets. It accelerates training and yields good performance results. However, there are various other alternatives to normalization, e.g. orthonormalization. The objective of this paper is to explore the possible alternatives to channel normalization with…

This paper explores how Transformers predict next tokens in autoregressive tasks.

New algorithm speeds up group equivariant neural networks computations.

Soft-Radial Projection solves gradient saturation in constrained deep learning.

A well-conditioned Jacobian spectrum has a vital role in preventing exploding or vanishing gradients and speeding up learning of deep neural networks. Free probability theory helps us to understand and handle the Jacobian spectrum. We rigorously show almost sure asymptotic freeness of layer-wise Jacobians of deep neura…

SVD training reduces DNN rank and computation load without SVD per step.

ManifoldFlow relaxes fixed-spectrum Stiefel layers to learn a positive spectrum.

Gradient descent biases towards stable rank networks for nearly-orthogonal data.

LOTOS improves ensemble robustness by promoting orthogonal transformations.

Transformers learn to recall with non-orthogonal embeddings in realistic settings.

In recent years, state-of-the-art methods in computer vision have utilized increasingly deep convolutional neural network architectures (CNNs), with some of the most successful models employing hundreds or even thousands of layers. A variety of pathologies such as vanishing/exploding gradients make training such deep n…

SpecNet2 improves spectral embedding without orthogonalization, achieving better performance and efficiency.

OSA overcomes instability in skipless Transformers.

SOFARI improves inference on multi-task learning latent factors.

We present a riemannian structure on the disk that has a remarkably rich structure. Geodesics are hypocycloids and the (negative of the) laplacian has integer spectrum with multiplicity the Dirichlet divisor function. Eigenfunctions of the laplacian are orthogonal polynomials naturally suited to the analysis of acousti…

Develops a new framework for temporal anchoring in deep embedding spaces.

Paper finds conditions for benign overfitting in neural networks.

This study explains gradient flow dynamics in neural networks for small initialisation.

Deep linear networks exhibit collapsing features and classifiers across datasets.

Conformal Autoencoders infer intrinsic dimensionality and impose invariance.

This work presents a novel approach to train invertible linear layers by adding rank-one perturbations.

EigenGAN discovers interpretable dimensions in GAN layers for semantic control.

Many modern big data applications feature large scale in both numbers of responses and predictors. Better statistical efficiency and scientific insights can be enabled by understanding the large-scale response-predictor association network structures via layers of sparse latent factors ranked by importance. Yet sparsit…

Gradient descent dynamics in quadratic regression models are analyzed, revealing five phases: monotonic, catapult, periodic, chaotic, and divergent.

Despite the phenomenal success of deep learning in recent years, there remains a gap in understanding the fundamental mechanics of neural nets. More research is focussed on handcrafting complex and larger networks, and the design decisions are often ad-hoc and based on intuition. Some recent research has aimed to demys…

A new method improves continual learning by replaying pseudo data and using orthogonal weight modification.

GAIT-prop derives a biologically plausible learning rule from backpropagation.

Gradient descent fails to learn simple neural networks efficiently.

This work introduces a tensor-based method to perform supervised classification on spatiotemporal data processed in an echo state network. Typically when performing supervised classification tasks on data processed in an echo state network, the entire collection of hidden layer node states from the training dataset is …

Gradient descent can efficiently learn a target function with diverse and near-orthogonal features.

LipKernel adds robustness to CNNs by enforcing Lipschitz bounds.

Lipschitz constraints under L2 norm on deep neural networks are useful for provable adversarial robustness bounds, stable training, and Wasserstein distance estimation. While heuristic approaches such as the gradient penalty have seen much practical success, it is challenging to achieve similar practical performance wh…

Neural networks learn incrementally from orthogonal data, interpolating with minimal complexity.

Reservoir Computing (RC) refers to a Recurrent Neural Networks (RNNs) framework, frequently used for sequence learning and time series prediction. The RC system consists of a random fixed-weight RNN (the input-hidden reservoir layer) and a classifier (the hidden-output readout layer). Here we focus on the sequence lear…

Invariant and equivariant networks have been successfully used for learning images, sets, point clouds, and graphs. A basic challenge in developing such networks is finding the maximal collection of invariant and equivariant linear layers. Although this question is answered for the first three examples (for popular tra…

OLinear forecasts time series more efficiently by transforming data orthogonally.