A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
One of the difficulties of training deep neural networks is caused by improper scaling between layers. Scaling issues introduce exploding / gradient problems, and have typically been addressed by careful scale-preserving initialization. We investigate the value of preserving scale, or isometry, beyond the initial weigh…
We develop a gluing construction which adds scaled and truncated asymptotically Euclidean solutions of the Einstein constraint equations to compact solutions with potentially non-trivial cosmological constants. The result is a one-parameter family of initial data which has ordinary and scaled "point-particle" limits an…
Initializing the weights and the biases is a key part of the training process of a neural network. Unlike the subsequent optimization phase, however, the initialization phase has gained only limited attention in the literature. In this paper we discuss some consequences of commonly used initialization strategies for va…
We propose a system for calculating a "scaling constant" for layers and weights of neural networks. We relate this scaling constant to two important quantities that relate to the optimizability of neural networks, and argue that a network that is "preconditioned" via scaling, in the sense that all weights have the same…
The paper develops a theory linking pretraining and fine-tuning in neural networks.
problem Understanding how initialization choices impact feature learning and generalization in neural networks.
method Analytical theory of diagonal linear networks, deriving generalization error as a function of initialization parameters and task statistics.
result Different initialization choices place networks into four fine-tuning regimes with varying abilities to support feature learning and generalization.
Scale-free distributions and correlation functions found in financial data are reminiscent of the scale invariance of physical observables in the vicinity of a critical point. Here, we present empirical evidence for a transition phenomenon, accompanied by a symmetry breaking, in the investors' demand for stocks. We stu…
Model shows feature learning can improve neural scaling laws for hard tasks.
problem Understanding and improving neural network scaling laws for various task difficulties.
method Developed a solvable model of neural scaling laws, identified three scaling regimes, and demonstrated feature learning's impact on scaling exponents.
result Feature learning can improve scaling with training time and compute for hard tasks, nearly doubling the exponent.
We analyze the global convergence of gradient descent for deep linear residual networks by proposing a new initialization: zero-asymmetric (ZAS) initialization. It is motivated by avoiding stable manifolds of saddle points. We prove that under the ZAS initialization, for an arbitrary target matrix, gradient descent con…
Public debt is one of the important economic variables that quantitatively describes a nation's economy. Because bankruptcy is a risk faced even by institutions as large as governments (e.g. Iceland), national debt should be strictly controlled with respect to national wealth. Also, the problem of eliminating extreme p…
The selection of initial parameter values for gradient-based optimization of deep neural networks is one of the most impactful hyperparameter choices in deep learning systems, affecting both convergence times and model performance. Yet despite significant empirical and theoretical analysis, relatively little has been p…
In this note, we prove a uniform distance distortion estimate for Ricci flows with uniformly bounded scalar curvature, independent of the lower bound of the initial μ-entropy. Our basic principle tells that once correctly renormalized, the metric-measure quantities obey similar estimates as in the non-collapsing case…
Using the procedure initiated in \cite{Ma2013}, we deform Lax-type equations though a scaling of the time parameter. This gives an equivalent (deformed) equation which is integrable in terms of power series of the scaling parameter. We then describe a regular Frölicher Lie group of symmetries of this deformed equation
The paper establishes principles for initializing and designing GNNs with ReLU activations to avoid oversmoothing and correlation collapse.
problem Oversmoothing and correlation collapse in deep ReLU GNNs.
method The paper derives and validates three principles for initialization and architecture selection in finite width graph neural networks with ReLU activations.
result Correct initialization, residual aggregation operators, and residual connections significantly improve early training dynamics in deep ReLU GNNs.
Gradient Descent (GD) approximators often fail in the solution space with multiple scales of convexities, i.e., in subspace learning and neural network scenarios. To handle that, one solution is to run GD multiple times from different randomized initial states and select the best solution over all experiments. However,…
Proves long-time Ricci flow existence and topological rigidity for pinched integral curvature manifolds.
problem Proving long-time existence and topological rigidity for manifolds with pinched scale-invariant integral curvature.
method Proves long-time existence of Ricci flow for manifolds with bounded curvature and pinched scale-invariant integral curvature, converging to a flat metric.
result Flow converges to a flat metric, implying topological rigidity of the manifold.
Huisken and Sinestrari have recently defined a surgery process for mean curvature flow when the initial data is a two-convex hypersurface. The process depends on a parameter H. Its role is to initiate a surgery when the maximum of the mean curvature of the evolving hypersurface becomes H, and to control the scale at wh…
We consider inverse curvature flows in hyperbolic space with starshaped initial hypersurface, driven by positive powers of a homogeneous curvature function. The solutions exist for all time and, after rescaling, converge to a sphere.
Unified learning-rate scale for CNNs and ResNets, avoiding depth imbalance.
problem Challenges in choosing an appropriate learning rate for deep networks, especially as depth increases.
method Introduces Arithmetic-Mean μP (AM-μP), constraining network-wide average pre-activation second moment to a constant scale, combined with residual-aware He fan-in initialization.
result Demonstrates a −3/2 scaling law for learning rates across depths, enabling zero-shot learning-rate transfer.
Training very deep networks is an important open problem in machine learning. One of many difficulties is that the norm of the back-propagated error gradient can grow or decay exponentially. Here we show that training very deep feed-forward networks (FFNs) is not as difficult as previously thought. Unlike when back-pro…
We present an algorithm for supervised learning using tensor networks, employing a step of preprocessing the data by coarse-graining through a sequence of wavelet transformations. We represent these transformations as a set of tensor network layers identical to those in a multi-scale entanglement renormalization ansatz…