arXiv research

New approach ties loss curvature to model performance in deep learning.

JacNet learns Jacobians to enforce structure on derivatives for invertibility and Lipschitz functions.

Maxout networks study gradients and propose initialization strategies.

It is well known that the initialization of weights in deep neural networks can have a dramatic impact on learning speed. For example, ensuring the mean squared singular value of a network's input-output Jacobian is $O(1)$ is essential for avoiding the exponential vanishing or explosion of gradients. The stronger condi…

Recent work has shown that tight concentration of the entire spectrum of singular values of a deep network's input-output Jacobian around one at initialization can speed up learning by orders of magnitude. Therefore, to guide important design choices, it is important to build a full theoretical understanding of the spe…

This work introduces the concept of tangent space regularization for neural-network models of dynamical systems. The tangent space to the dynamics function of many physical systems of interest in control applications exhibits useful properties, e.g., smoothness, motivating regularization of the model Jacobian along sys…

Exact spectral norm regularization improves neural network generalization.

In practice it is often found that large over-parameterized neural networks generalize better than their smaller counterparts, an observation that appears to conflict with classical notions of function complexity, which typically favor smaller models. In this work, we investigate this tension between complexity and gen…

The paper proves a distribution claim for neural network Jacobians.

Sigmoid autoencoders can implement associative memory with certain conditions.

We introduce MinimalRNN, a new recurrent neural network architecture that achieves comparable performance as the popular gated RNNs with a simplified structure. It employs minimal updates within RNN, which not only leads to efficient learning and testing but more importantly better interpretability and trainability. We…

We give a rigorous analysis of the statistical behavior of gradients in a randomly initialized fully connected network N with ReLU activations. Our results show that the empirical variance of the squares of the entries in the input-output Jacobian of N is exponential in a simple architecture-dependent constant beta, gi…

Optimizes wide low-rank neural networks for reduced parameters and cost.

Weight decay is one of the standard tricks in the neural network toolbox, but the reasons for its regularization effect are poorly understood, and recent results have cast doubt on the traditional interpretation in terms of $L_2$ regularization. Literal weight decay has been shown to outperform $L_2$ regularization for…

Deep learning algorithms have increasingly been shown to lack robustness to simple adversarial examples (AdvX). An equally troubling observation is that these adversarial examples transfer between different architectures trained on different datasets. We investigate the transferability of adversarial examples between m…

The Jacobian Conjecture is proven for all Jacobian maps.

This study connects Jacobian regularization to adversarial robustness and improves generalization.

The paper improves alignment methods for deep neural networks using geometric and spectral analysis.

GeMA learns latent manifolds to benchmark complex systems.

Study shows connections between Jacobian torsors and Fermat curves.

We revisit the initialization of deep residual networks (ResNets) by introducing a novel analytical tool in free probability to the community of deep learning. This tool deals with non-Hermitian random matrices, rather than their conventional Hermitian counterparts in the literature. As a consequence, this new tool ena…

This work proves the asymptotic freeness of layerwise Jacobians in MLPs with Haar orthogonal matrices.

We demonstrate that in residual neural networks (ResNets) dynamical isometry is achievable irrespectively of the activation function used. We do that by deriving, with the help of Free Probability and Random Matrix Theories, a universal formula for the spectral density of the input-output Jacobian at initialization, in…

Efficiently regularizes deep learning models using Jacobian nuclear norm.

Abstract: Unknown status of Jacobian Conjecture, proof has a gap.

Safe neural networks for input-output specifications.

A well-conditioned Jacobian spectrum has a vital role in preventing exploding or vanishing gradients and speeding up learning of deep neural networks. Free probability theory helps us to understand and handle the Jacobian spectrum. We rigorously show almost sure asymptotic freeness of layer-wise Jacobians of deep neura…

Recently mean field theory has been successfully used to analyze properties of wide, random neural networks. It gave rise to a prescriptive theory for initializing feed-forward neural networks with orthogonal weights, which ensures that both the forward propagated activations and the backpropagated gradients are near $…

In recent years, state-of-the-art methods in computer vision have utilized increasingly deep convolutional neural network architectures (CNNs), with some of the most successful models employing hundreds or even thousands of layers. A variety of pathologies such as vanishing/exploding gradients make training such deep n…

The Jacobian conjecture is simplified using polynomial mappings.

Paper tackles Hessian/Jacobian-free stochastic bilevel optimization with ${O}(ε^{-1.5})$ complexity.

Recent work (Pennington et al, 2017) suggests that controlling the entire distribution of Jacobian singular values is an important design consideration in deep learning. Motivated by this, we study the distribution of singular values of the Jacobian of the generator in Generative Adversarial Networks (GANs). We find th…

The paper discusses fractional Sobolev immersions of flat domains into 3D space.

We derive an analytic formula for the dual Jacobian matrix of a generalised hyperbolic tetrahedron. Two cases are considered: a mildly truncated and a prism truncated tetrahedron. The Jacobian for the latter arises as an analytic continuation of the former, that falls in line with a similar behaviour of the correspondi…

Economic systems, traditionally analyzed as almost independent national systems, are increasingly connected on a global scale. Only recently becoming available, the World Input-Output Database (WIOD) is one of the first efforts to construct the multi-regional input-output (MRIO) tables at the global level. By viewing t…

New algorithms estimate Jacobian matrices for large-scale machine learning.

The paper extends infinite-width analysis to neural network Jacobians, revealing convergence to Gaussian processes and linear ODEs.

GrokAlign aligns Jacobians to accelerate grokking in deep networks.

We provide a characterization for complex analytic curves among two-dimensional minimal graphs in $\mathbb{R}^{4}$ via the Jacobian

We extend the well-known result that any $f \in W^{1,n}(Ω,\mathbb{R}^n)$, $Ω\subset \mathbb{R}^n$ with strictly positive Jacobian is actually continuous: it is also true for fractional Sobolev spaces $W^{s,\frac{n}{s}}(Ω)$ for any $s \geq \frac{n}{n+1}$, where the sign condition on the Jacobian is understood in a distr…

New method reduces deep learning training costs by approximating vector-jacobian products.

There has been a recent shift in sequence-to-sequence modeling from recurrent network architectures to convolutional network architectures due to computational advantages in training and operation while still achieving competitive performance. For systems having limited long-term temporal dependencies, the approximatio…

Unitary recurrent neural networks (URNNs) have been proposed as a method to overcome the vanishing and exploding gradient problem in modeling data with long-term dependencies. A basic question is how restrictive is the unitary constraint on the possible input-output mappings of such a network? This work shows that for …

Normalizing flows optimize Jacobian determinant for unique likelihood objective.

The Jacobian of Douady-Earle extension equals 1 only for isometries.

The aim here is to continue the investigation in \cite{AB} of Jacobians of a Klein surface and also to correct an error in \cite{AB}.

To a compact Riemann surface of genus g can be assigned a principally polarized abelian variety (PPAV) of dimension g, the Jacobian of the Riemann surface. The Schottky problem is to discern the Jacobians among the PPAVs. Buser and Sarnak showed, that the square of the first successive minimum, the squared norm of the …

We show that the Goldman flows preserve the holomorphic structure on the moduli space of homomorphisms of the fundamental group of a Riemann surface into U(1), in other words the Jacobian.