Deep neural networks' Jacobian spectrum becomes well-conditioned with orthogonal weights.
problem Understanding and handling the Jacobian spectrum of deep neural networks.
method Applying free probability theory to show almost sure asymptotic freeness of Jacobians in the wide limit.
result Layer-wise Jacobians of deep neural networks with orthogonal weights are almost surely asymptotically free.
Recent work (Pennington et al, 2017) suggests that controlling the entire distribution of Jacobian singular values is an important design consideration in deep learning. Motivated by this, we study the distribution of singular values of the Jacobian of the generator in Generative Adversarial Networks (GANs). We find th…
The study finds infinitely many Shimura subvarieties in Jacobian loci for curves of genus 2, 3, and 4.
problem Understanding Shimura subvarieties in Jacobian loci for curves of positive genus.
method Analyzing Galois covers of curves and their Shimura subvarieties under specific numerical conditions.
result The Jacobian locus contains infinitely many Shimura subvarieties of positive dimension for g≤4. This paper investigates how policy conditioning affects reinforcement learning stability.
problem Improving stability and generalization of reinforcement learning agents.
method The authors study Jacobian conditioning behavior during policy optimization and propose a conditioning regularization algorithm.
result The proposed conditioning regularization algorithm enhances reinforcement learning agent generalization.
Generative adversarial networks (GANs) are notoriously difficult to train and the reasons underlying their (non-)convergence behaviors are still not completely understood. By first considering a simple yet representative GAN example, we mathematically analyze its local convergence behavior in a non-asymptotic way. Furt…
We extend the well-known result that any f∈W1,n(Ω,Rn), Ω⊂Rn with strictly positive Jacobian is actually continuous: it is also true for fractional Sobolev spaces Ws,sn(Ω) for any s≥n+1n, where the sign condition on the Jacobian is understood in a distr…
The paper studies global invertibility of maps on Finsler manifolds.
problem Global invertibility of locally Lipschitz maps on Finsler manifolds.
method Introduces pseudo-Jacobian and studies its relations with local metric properties of the map.
result Conditions for a map to be globally invertible and covering.
We propose a fast, simple and robust algorithm for computing shortest paths and distances on Riemannian manifolds learned from data. This amounts to solving a system of ordinary differential equations (ODEs) subject to boundary conditions. Here standard solvers perform poorly because they require well-behaved Jacobians…
The Jacobian Conjecture is proven for all Jacobian maps.
problem Proving the Jacobian Conjecture for all Jacobian maps.
method Using the Weyl algebra and holonomic modules, the paper shows that the Jacobian module is 1-generated and has finite length.
result The Jacobian Conjecture is true for all Jacobian maps.
This study connects Jacobian regularization to adversarial robustness and improves generalization.
problem Adversarial attacks make deep neural networks vulnerable.
method Developed a connection between Jacobian regularization and adversarial training, and established robust generalization gaps.
result Jacobian norms are related to both standard and robust generalization.
A new method for faster bandwidth selection in Gaussian kernel ridge regression.
problem Efficiently selecting the bandwidth in Gaussian kernel ridge regression.
method Formulated an approximate Jacobian expression for bandwidth selection, proposing a closed-form heuristic.
result Our method is as accurate as cross-validation and marginal likelihood maximization but up to six orders of magnitude faster.
Study shows connections between Jacobian torsors and Fermat curves.
problem Understanding torsors of Jacobian of universal Fermat curves.
method Analyzes torsors of Jacobian of universal family of degree-m Fermat curves. result Every torsor is a connected component of the Picard scheme.
We compute the local Lipschitz constant of ReLU networks precisely.
problem Estimating the local Lipschitz constant of ReLU networks is hard.
method We use a novel approach involving the generalized Jacobian and backpropagation.
result We provide an algorithm to compute the exact Lipschitz constant of ReLU networks.
Self-shrinkers are important geometric objects in the study of mean curvature flows, while the Bernstein Theorem is one of the most profound results in minimal surface theory. We prove a Bernstein type result for graphical self-shrinker surfaces with codimension two in R4. Namely, under certain natural cond…
This work proves the asymptotic freeness of layerwise Jacobians in MLPs with Haar orthogonal matrices.
problem Proving the asymptotic freeness of layerwise Jacobians in multilayer perceptrons (MLPs).
method Replacing each layer's parameter matrix with itself multiplied by a Haar orthogonal matrix, and using the invariance of the MLP.
result Proves the asymptotic freeness of layerwise Jacobians in MLPs with Haar orthogonal matrices.
Efficiently regularizes deep learning models using Jacobian nuclear norm.
problem Regularizing deep learning models to prevent overfitting and improve generalization.
method Proposes a denoising-style approximation to penalize the Jacobian nuclear norm without computing the Jacobian matrix.
result Demonstrates that penalizing the average squared Frobenius norm of Jg and Jh is equivalent to penalizing the Jacobian nuclear norm for function compositions. Jacobian regularization boosts neural network robustness without degrading generalization.
problem Ensuring robustness of machine learning models against input perturbations.
method Developed a computationally efficient Jacobian regularization technique.
result Significant improvements in robustness measured against random and adversarial perturbations.
Abstract: Unknown status of Jacobian Conjecture, proof has a gap.
problem Status of Jacobian Conjecture
method Analysis of proof of theorem 2.1
result Proof of theorem 2.1 contains a gap
AutoInit automatically finds good neural network initialization.
problem Finding optimal neural network initialization is crucial but time-consuming.
method Uses Jacobian tuning to automatically adjust network hyperparameters.
result The method finds good initialization for various network architectures.
StAD predicts divergence of diffusion and flow models without Jacobian computation.
problem Computing likelihood from diffusion and flow models is computationally expensive.
method Introduces StAD, a distillation method to predict divergence using Langevin-Stein operator.
result StAD predicts divergence with competitive variance and speed compared to existing methods.
The Jacobian conjecture is simplified using polynomial mappings.
problem Simplifying the Jacobian conjecture over the real field.
method Using polynomial mappings to restrict transitions on manifolds.
result An equivalent statement of the Jacobian conjecture.
Paper tackles Hessian/Jacobian-free stochastic bilevel optimization with O(ε−1.5) complexity.
problem Nonconvex-strongly-convex bilevel optimization problem.
method FdeHBO optimizer with finite-difference Hessian/Jacobian-vector approximation and momentum.
result FdeHBO achieves O(ε−1.5) iterations for ε-accurate stationary point. NODEs can approximate a wide range of diffeomorphisms with strong guarantees.
problem The approximation power of NODEs under certain conditions.
method Leveraging a structure theorem of the diffeomorphism group.
result NODEs can approximate a large class of diffeomorphisms with a stronger guarantee.
This paper shows how to approximate any log-concave distribution using well-conditioned affine coupling flows.
problem Understanding the representational power of affine coupling flows for log-concave distributions.
method Leveraging connections between affine coupling architectures, Langevin dynamics, and Hénon maps to prove log-concave approximation.
result Any log-concave distribution can be approximated using well-conditioned affine-coupling flows.
Sigmoid autoencoders can implement associative memory with certain conditions.
problem Implementing associative memory in neural networks.
method Theoretical analysis of overparameterized sigmoid autoencoders using the NTK and iterative maps.
result Overparameterized sigmoid autoencoders can have attractors in the NTK limit, leading to associative memory.
The paper discusses fractional Sobolev immersions of flat domains into 3D space.
problem Developing C1 regularity and isometric immersions of flat domains with fractional Sobolev regularity. method Analysis of weak Codazzi-Mainardi equations, study of $W^{2, rac2s}$ planar deformations, and properties of the distributional Jacobian determinant.
result Generalization of isometric immersions with local fractional Sobolev regularity.
We derive an analytic formula for the dual Jacobian matrix of a generalised hyperbolic tetrahedron. Two cases are considered: a mildly truncated and a prism truncated tetrahedron. The Jacobian for the latter arises as an analytic continuation of the former, that falls in line with a similar behaviour of the correspondi…
Study shows a universal local obstruction to the Samuelson condition for tangent Lagrangian 2-webs.
problem Obstruction to the Samuelson condition for tangent Lagrangian 2-webs.
method Local analysis of tangent lines and their intersection maps.
result A universal local phenomenon produces a nonzero mixed derivative, obstructing the Samuelson condition.
New algorithms estimate Jacobian matrices for large-scale machine learning.
problem Efficiently computing search directions for large nonlinear least squares.
method Exploit low-rank structure in Hessian to estimate Jacobian matrices.
result Two algorithms perform well compared to state-of-the-art methods.
The paper extends infinite-width analysis to neural network Jacobians, revealing convergence to Gaussian processes and linear ODEs.
problem Understanding the training dynamics of neural networks in the infinite-width limit.
method Extending infinite-width analysis to Jacobians, characterizing convergence to Gaussian processes and linear ODEs.
result The evolution of MLPs under robust training in the infinite-width limit is described by a linear ODE.
OSA overcomes instability in skipless Transformers.
problem Instability in skipless Transformers using Softmax Self-Attention.
method OSA parametrizes attention matrix to be orthogonal via skew-symmetric matrix exponential.
result OSA allows for training non-causal Transformers without skip connections and normalisation layers.
GrokAlign aligns Jacobians to accelerate grokking in deep networks.
problem Accelerating the training dynamics of deep networks to avoid delayed generalisation and robustness.
method Aligning the Jacobians of a deep network with the training data to ensure grokking under a low-rank assumption.
result GrokAlign regularizes Jacobians to induce grokking sooner than conventional methods.
We provide a characterization for complex analytic curves among two-dimensional minimal graphs in R4 via the Jacobian
New method reduces deep learning training costs by approximating vector-jacobian products.
problem Efficiently training deep neural networks with reduced computational and memory costs.
method Randomized, unbiased approximations of vector-jacobian products during backpropagation.
result Validated potential for reducing deep learning training costs through unbiased estimates.
It is well known that the initialization of weights in deep neural networks can have a dramatic impact on learning speed. For example, ensuring the mean squared singular value of a network's input-output Jacobian is O(1) is essential for avoiding the exponential vanishing or explosion of gradients. The stronger condi…
Normalizing flows optimize Jacobian determinant for unique likelihood objective.
problem Optimizing normalizing flows for unique likelihood.
method Showed Jacobian determinant is unique for given distributions, leading to a unique global optimum. Used eigenvalues of auto-correlation matrix for explicit likelihood expression.
result Explicit expression of likelihood for flows, independent of neural network parameterization, with theoretical optimal value.
The Jacobian of Douady-Earle extension equals 1 only for isometries.
problem Investigating the Jacobian of Douady-Earle extension maps.
method Analyzing the Jacobian of the Douady-Earle extension map and constructing sequences of hyperbolic surfaces.
result The Jacobian of the Douady-Earle extension map is 1 only when the map is an isometry, and it can grow arbitrarily large for certain sequences of surfaces.
The aim here is to continue the investigation in \cite{AB} of Jacobians of a Klein surface and also to correct an error in \cite{AB}.
To a compact Riemann surface of genus g can be assigned a principally polarized abelian variety (PPAV) of dimension g, the Jacobian of the Riemann surface. The Schottky problem is to discern the Jacobians among the PPAVs. Buser and Sarnak showed, that the square of the first successive minimum, the squared norm of the …
We show that the Goldman flows preserve the holomorphic structure on the moduli space of homomorphisms of the fundamental group of a Riemann surface into U(1), in other words the Jacobian.
Paper tackles catastrophic forgetting in sequential learning.
problem Catastrophic forgetting in sequential learning.
method Regularizes training with sketches of Jacobian matrix of past data.
result Proves overcoming catastrophic forgetting for linear and wide neural networks.
Recovering hidden influence networks from cascade data using Jacobian-based machine learning.
problem Recovering influence networks behind dynamic cascades.
method CascadeNet, a Jacobian-based machine learning framework.
result CascadeNet achieves high accuracy in network recovery.
Geometrically represents path integral reduction Jacobian for interacting systems.
problem Quantizing a model mechanical system with dependent coordinates.
method Geometric representation using scalar curvature and Christoffel symbols in a nonholonomic basis.
result Found a geometric representation for the path integral reduction Jacobian.
This work relaxes energy constraints in self-attention layers for a more general analysis.
problem Understanding inherent biases and dynamics in self-attention layers without energy functions.
method Dynamical systems analysis and Jacobian matrix examination.
result Normalized dynamics are close to a critical state, indicating high inference performance.
We compute some value of the harmonic volume for the Fermat sextic. Using this computation, we prove that some special algebraic cycle in the Jacobian variety of the Fermat sextic is not algebraically equivalent to zero.
JacNet learns Jacobians to enforce structure on derivatives for invertibility and Lipschitz functions.
problem Enforcing structure on derivatives of neural network mappings.
method Proposes using a neural network to directly learn the Jacobian of the input-output function, allowing control over derivative structure.
result Demonstrates learning invertible approximations to simple and 1-Lipschitz functions.
The Jacobian matrix (or the gradient for single-output networks) is directly related to many important properties of neural networks, such as the function landscape, stationary points, (local) Lipschitz constants and robustness to adversarial attacks. In this paper, we propose a recursive algorithm, RecurJac, to comput…
We show that there are separated nets in the Euclidean plane which are not biLipschitz equivalent to the integer lattice. The argument is based on the construction of a continuous function which is not the Jacobian of a biLipschitz map.