A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
Improves diffusion models by controlling total variance and signal-to-noise-ratio.
iEFM trains CNF models from unnormalized densities efficiently.
A new method reduces variance in training early-stage rankers for large-scale search systems.
This paper extends de Rham theory of smooth manifolds to exploded manifolds. Included are versions of Stokes' theorem, De Rham cohomology, Poincare duality, and integration along the fiber. The resulting cohomology theory is used to define Gromov Witten invariants of exploded manifolds in a separate paper.
A new gradient estimator reduces variance near boundaries for binary latent variables.
A new method estimates marginal likelihood using normalizing flows.
We consider the off-policy estimation problem of estimating the expected reward of a target policy using samples collected by a different behavior policy. Importance sampling (IS) has been a key technique to derive (nearly) unbiased estimators, but is known to suffer from an excessively high variance in long-horizon pr…
Notes for a short lecture series, covering exploded manifolds, the moduli stack of curves in exploded manifolds, and a tropical gluing formula for Gromov-Witten invariants: a gluing formula providing a degeneration formula for Gromov-Witten invariants in normal-crossing degenerations. I gave the original lecture series…
Classical (Itô diffusions) stochastic volatility models are not able to capture the steepness of small-maturity implied volatility smiles. Jumps, in particular exponential Lévy and affine models, which exhibit small-maturity exploding smiles, have historically been proposed to remedy this (see \cite{Tank} for an overvi…
New neural network approach mitigates vanishing/exploding gradients.
We give a rigorous analysis of the statistical behavior of gradients in a randomly initialized fully connected network N with ReLU activations. Our results show that the empirical variance of the squares of the entries in the input-output Jacobian of N is exponential in a simple architecture-dependent constant beta, gi…
We present a gluing formula for Gromov-Witten invariants in the case of a triple product. This gluing formula is a simple case of a much more general gluing formula proved and stated using exploded manifolds. We present this simple case because it is relatively easy to explain without any knowledge of exploded manifold…
We identify and study two common failure modes for early training in deep ReLU nets. For each we give a rigorous proof of when it occurs and how to avoid it, for fully connected and residual architectures. The first failure mode, exploding/vanishing mean activation length, can be avoided by initializing weights from a …
This work tackles exploding inverses in INNs, revealing and mitigating their numerical non-invertibility.
The paper improves DR estimators for off-policy evaluation in contextual bandits.
The exploding and vanishing gradient problem has been the major conceptual principle behind most architecture and training improvements in recurrent neural networks (RNNs) during the last decade. In this paper, we argue that this principle, while powerful, might need some refinement to explain recent developments. We r…
Derives a new formula for optimal stopping problems with exploding derivatives.
In this paper we investigate the asymptotics of forward-start options and the forward implied volatility smile in the Heston model as the maturity approaches zero. We prove that the forward smile for out-of-the-money options explodes and compute a closed-form high-order expansion detailing the rate of the explosion. Fu…
Linear RNNs exhibit a bias towards shorter memory due to initialization variance.
Deep Learning has revolutionized vision via convolutional neural networks (CNNs) and natural language processing via recurrent neural networks (RNNs). However, success stories of Deep Learning with standard feed-forward neural networks (FNNs) are rare. FNNs that perform well are typically shallow and, therefore cannot …
Enhances learning of structured distributions using nonlinear denoising score matching.
Extends LIBOR market model to reduce exploding scenarios.
Improved optimization technique reduces training complexity for non-convex problems.
The study provides a comprehensive error analysis for diffusion models.
This paper considers multi-dimensional affine processes with continuous sample paths. By analyzing the Riccati system, which is associated with affine processes via the transform formula, we fully characterize the regions of exponents in which exponential moments of a given process do not explode at any time or explode…
We define Gromov--Witten invariants of exploded manifolds. The technical heart of this paper is a construction of a virtual fundamental class of any Kuranishi category (which is a simplified, more general version of an embedded Kuranishi structure.) We also show how to integrate differential…
Blind Descent avoids gradient issues, using a different learning approach.
We propose a novel approach to addressing the vanishing (or exploding) gradient problem in deep neural networks. We construct a new architecture for deep neural networks where all layers (except the output layer) of the network are a combination of rotation, permutation, diagonal, and activation sublayers which are all…
Maxout networks study gradients and propose initialization strategies.
Previously, the exploding gradient problem has been explained to be central in deep learning and model-based reinforcement learning, because it causes numerical issues and instability in optimization. Our experiments in model-based reinforcement learning imply that the problem is not just a numerical issue, but it may …
A new RNN model tackles long-time dependencies with fast, invertible, and memory-efficient hidden states.
ADMMiRNN solves RNN training issues with stable convergence.
In \cite{FOinteger}, Fukaya and Ono outlined a way of counting pseudo-holomorphic curves in a general compact symplectic manifold to obtain integer valued invariants. This paper contains the details of Fukaya and Ono's suggested construction for any compact symplectic manifold and a large class of exploded manifolds.
We study products of random matrices in the regime where the number of terms and the size of the matrices simultaneously tend to infinity. Our main theorem is that the logarithm of the norm of such a product applied to any fixed vector is asymptotically Gaussian. The fluctuations we find can be thought of as a…
The paper examines the sampling dynamics of diffusion models using ODEs.
A simple gating mechanism improves deep learning convergence.
It is well known that the problem of vanishing/exploding gradients is a challenge when training deep networks. In this paper, we describe another phenomenon, called vanishing nodes, that also increases the difficulty of training deep neural networks. As the depth of a neural network increases, the network's hidden node…
The gradient-based optimization method for deep machine learning models suffers from gradient vanishing and exploding problems, particularly when the computational graph becomes deep. In this work, we propose the tangent-space gradient optimization (TSGO) for the probabilistic models to keep the gradients from vanishin…
The paper examines when NTK theory applies to real finite-width neural networks.
Recurrent neural networks (RNNs) have gained a great deal of attention in solving sequential learning problems. The learning of long-term dependencies, however, remains challenging due to the problem of a vanishing or exploding hidden states gradient. By exploring further the recently established connections between RN…
Recurrent neural networks (RNNs) are particularly well-suited for modeling long-term dependencies in sequential data, but are notoriously hard to train because the error backpropagated in time either vanishes or explodes at an exponential rate. While a number of works attempt to mitigate this effect through gated recur…
We prove two tropical gluing formulae for Gromov-Witten invariants of exploded manifolds, useful for calculating Gromov-Witten invariants of a symplectic manifold using a normal-crossing degeneration. The first formula generalizes the symplectic-sum formula for Gromov-Witten invariants. The second formula is stronger, …
New RNN model handles long-term dependencies in irregularly-sampled time series.
A well-conditioned Jacobian spectrum has a vital role in preventing exploding or vanishing gradients and speeding up learning of deep neural networks. Free probability theory helps us to understand and handle the Jacobian spectrum. We rigorously show almost sure asymptotic freeness of layer-wise Jacobians of deep neura…
Recurrent Neural Networks (RNN), Long Short-Term Memory Networks (LSTM), and Memory Networks which contain memory are popularly used to learn patterns in sequential data. Sequential data has long sequences that hold relationships. RNN can handle long sequences but suffers from the vanishing and exploding gradient probl…
New method improves DAG learning by using large coefficients for higher-order terms.
Conformally invariant functionals on the space of knots are introduced via extrinsic conformal geometry of the knot and integral geometry on the space of spheres. Our functionals are expressed in terms of a complex-valued 2-form which can be considered as the cross-ratio of a pair of infinitesimal segments of the knot.…
Residual Network (ResNet) is the state-of-the-art architecture that realizes successful training of really deep neural network. It is also known that good weight initialization of neural network avoids problem of vanishing/exploding gradients. In this paper, simplified models of ResNets are analyzed. We argue that good…