A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
This work tackles exploding inverses in INNs, revealing and mitigating their numerical non-invertibility.
The paper stabilizes invertible neural networks by using Gaussian mixture models.
This paper extends de Rham theory of smooth manifolds to exploded manifolds. Included are versions of Stokes' theorem, De Rham cohomology, Poincare duality, and integration along the fiber. The resulting cohomology theory is used to define Gromov Witten invariants of exploded manifolds in a separate paper.
It is well known that the problem of vanishing/exploding gradients is a challenge when training deep networks. In this paper, we describe another phenomenon, called vanishing nodes, that also increases the difficulty of training deep neural networks. As the depth of a neural network increases, the network's hidden node…
Notes for a short lecture series, covering exploded manifolds, the moduli stack of curves in exploded manifolds, and a tropical gluing formula for Gromov-Witten invariants: a gluing formula providing a degeneration formula for Gromov-Witten invariants in normal-crossing degenerations. I gave the original lecture series…
New neural network approach mitigates vanishing/exploding gradients.
Normalization layers are widely used in deep neural networks to stabilize training. In this paper, we consider the training of convolutional neural networks with gradient descent on a single training example. This optimization problem arises in recent approaches for solving inverse problems such as the deep image prior…
Method uses normalizing flows to efficiently sample from complex target densities.
Unified framework for lifted training and inversion of neural networks.
A new model reconciles rough volatility and jumps.
We present a gluing formula for Gromov-Witten invariants in the case of a triple product. This gluing formula is a simple case of a much more general gluing formula proved and stated using exploded manifolds. We present this simple case because it is relatively easy to explain without any knowledge of exploded manifold…
The exploding and vanishing gradient problem has been the major conceptual principle behind most architecture and training improvements in recurrent neural networks (RNNs) during the last decade. In this paper, we argue that this principle, while powerful, might need some refinement to explain recent developments. We r…
TSGO optimizes gradients in tensor networks to avoid vanishing/exploding issues.
Derives a new formula for optimal stopping problems with exploding derivatives.
Extends LIBOR market model to reduce exploding scenarios.
Initialization of parameters in deep neural networks has been shown to have a big impact on the performance of the networks (Mishkin & Matas, 2015). The initialization scheme devised by He et al, allowed convolution activations to carry a constrained mean which allowed deep networks to be trained effectively (He et al.…
This paper considers multi-dimensional affine processes with continuous sample paths. By analyzing the Riccati system, which is associated with affine processes via the transform formula, we fully characterize the regions of exponents in which exponential moments of a given process do not explode at any time or explode…
We define Gromov--Witten invariants of exploded manifolds. The technical heart of this paper is a construction of a virtual fundamental class of any Kuranishi category (which is a simplified, more general version of an embedded Kuranishi structure.) We also show how to integrate differential…
Blind Descent avoids gradient issues, using a different learning approach.
Maxout networks study gradients and propose initialization strategies.
Volume-preserving neural networks prevent gradient issues.
A new RNN model tackles long-time dependencies with fast, invertible, and memory-efficient hidden states.
ADMMiRNN solves RNN training issues with stable convergence.
In \cite{FOinteger}, Fukaya and Ono outlined a way of counting pseudo-holomorphic curves in a general compact symplectic manifold to obtain integer valued invariants. This paper contains the details of Fukaya and Ono's suggested construction for any compact symplectic manifold and a large class of exploded manifolds.
ENRNN uses eigenvalue normalization for short-term memory in RNNs.
A simple gating mechanism improves deep learning convergence.
Recurrent neural networks (RNNs) are particularly well-suited for modeling long-term dependencies in sequential data, but are notoriously hard to train because the error backpropagated in time either vanishes or explodes at an exponential rate. While a number of works attempt to mitigate this effect through gated recur…
iEFM trains CNF models from unnormalized densities efficiently.
We prove two tropical gluing formulae for Gromov-Witten invariants of exploded manifolds, useful for calculating Gromov-Witten invariants of a symplectic manifold using a normal-crossing degeneration. The first formula generalizes the symplectic-sum formula for Gromov-Witten invariants. The second formula is stronger, …
Improves diffusion models by controlling total variance and signal-to-noise-ratio.
Paper tackles Bayesian image restoration in low-photon Poisson imaging problems.
New RNN model handles long-term dependencies in irregularly-sampled time series.
Recurrent Neural Networks (RNN), Long Short-Term Memory Networks (LSTM), and Memory Networks which contain memory are popularly used to learn patterns in sequential data. Sequential data has long sequences that hold relationships. RNN can handle long sequences but suffers from the vanishing and exploding gradient probl…
A well-conditioned Jacobian spectrum has a vital role in preventing exploding or vanishing gradients and speeding up learning of deep neural networks. Free probability theory helps us to understand and handle the Jacobian spectrum. We rigorously show almost sure asymptotic freeness of layer-wise Jacobians of deep neura…
New method improves DAG learning by using large coefficients for higher-order terms.
Hamiltonian RNN controls hidden states gradient for long-term dependencies.
Conformally invariant functionals on the space of knots are introduced via extrinsic conformal geometry of the knot and integral geometry on the space of spheres. Our functionals are expressed in terms of a complex-valued 2-form which can be considered as the cross-ratio of a pair of infinitesimal segments of the knot.…
Residual Network (ResNet) is the state-of-the-art architecture that realizes successful training of really deep neural network. It is also known that good weight initialization of neural network avoids problem of vanishing/exploding gradients. In this paper, simplified models of ResNets are analyzed. We argue that good…
Binary Neural Networks (BNNs) are difficult to train, and suffer from drop of accuracy. It appears in practice that BNNs fail to train in the absence of Batch Normalization (BatchNorm) layer. We find the main role of BatchNorm is to avoid exploding gradients in the case of BNNs. This finding suggests that the common in…
One of the difficulties of training deep neural networks is caused by improper scaling between layers. Scaling issues introduce exploding / gradient problems, and have typically been addressed by careful scale-preserving initialization. We investigate the value of preserving scale, or isometry, beyond the initial weigh…
Scaling ResNets requires careful consideration of the layer depth and output scaling factors.
Novel BSG method for efficient stochastic optimization.
AuON is a linear-time optimizer that improves upon Muon's performance without approximate orthogonal matrices.
Vanishing and exploding gradients are two of the main obstacles in training deep neural networks, especially in capturing long range dependencies in recurrent neural networks~(RNNs). In this paper, we present an efficient parametrization of the transition matrix of an RNN that allows us to stabilize the gradients that …
RNNs struggle with chaotic dynamics due to exploding gradients, but we found a way to optimize training.
Training recurrent neural networks (RNNs) is a hard problem due to degeneracies in the optimization landscape, a problem also known as vanishing/exploding gradients. Short of designing new RNN architectures, previous methods for dealing with this problem usually boil down to orthogonalization of the recurrent dynamics,…
Quasi-Gaussian HJM models are a popular approach for modeling the dynamics of the yield curve. This is due to their low dimensional Markovian representation, which greatly simplifies their numerical implementation. We present a qualitative study of the solutions of the quasi-Gaussian log-normal HJM model. Using a small…
Batch normalization (batch norm) is often used in an attempt to stabilize and accelerate training in deep neural networks. In many cases it indeed decreases the number of parameter updates required to achieve low training error. However, it also reduces robustness to small adversarial input perturbations and noise by d…