A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
Proves existence of optimal shallow neural networks with ReLU activation.
problem Proving the existence of optimal shallow feedforward networks with ReLU activation.
method Proves existence of global minima in the loss landscape for continuous target functions using shallow feedforward neural networks with ReLU activation.
result Existence of global minima in the loss landscape for shallow feedforward networks with ReLU activation.
We establish, for the first time, connections between feedforward neural networks with ReLU activation and tropical geometry --- we show that the family of such neural networks is equivalent to the family of tropical rational maps. Among other things, we deduce that feedforward ReLU neural networks with one hidden laye…
The paper examines topological features of ReLU networks and their relation to decision boundaries and training loss.
problem Understanding the topological structure of ReLU neural network activation patterns.
method Polytope decomposition of feature space, Fiedler partition of dual graph, homology computation of cellular decomposition.
result The Fiedler partition of the dual graph correlates with decision boundaries in binary classification tasks, and similar patterns in training loss and polyhedral cell-count emerge in regression tasks.
This study connects ReLU neural networks to toric geometry to analyze function realization.
problem Determining which continuous piecewise linear functions can be realized by ReLU neural networks.
method Established a connection between toric geometry and ReLU neural networks, defining key structures like the ReLU fan, toric variety, and Cartier divisor.
result Proved a criterion for functions realizable by unbiased shallow ReLU networks using intersection numbers.
How can local-search methods such as stochastic gradient descent (SGD) avoid bad local minima in training multi-layer neural networks? Why can they fit random labels even given non-convex and non-smooth architectures? Most existing theory only covers networks with one hidden layer, so can we go deeper? In this paper, w…
A wide variety of activation functions have been proposed for neural networks. The Rectified Linear Unit (ReLU) is especially popular today. There are many practical reasons that motivate the use of the ReLU. This paper provides new theoretical characterizations that support the use of the ReLU, its variants such as th…
Consider the multivariate nonparametric regression model. It is shown that estimators based on sparsely connected deep neural networks with ReLU activation function and properly chosen network architecture achieve the minimax rates of convergence (up to logn-factors) under a general composition assumption on the re…
The exclusive or (xor) function is one of the simplest examples that illustrate why nonlinear feedforward networks are superior to linear regression for machine learning applications. We review the xor representation and approximation problems and discuss their solutions in terms of probabilistic logic and associative …
Study on deep neural networks using concentration inequalities and optimal stopping.
problem Understanding the performance and structure of stochastic deep neural networks.
method Introduced concentration inequalities for SDNN outputs and an EC classifier. Determined the optimal number of layers via an optimal stopping procedure.
result Optimal number of layers for SDNNs determined via an optimal stopping procedure.
An interesting approach to analyzing neural networks that has received renewed attention is to examine the equivalent kernel of the neural network. This is based on the fact that a fully connected feedforward network with one hidden layer, a certain weight distribution, an activation function, and an infinite number of…
We consider deep feedforward neural networks with rectified linear units from a signal processing perspective. In this view, such representations mark the transition from using a single (data-driven) linear representation to utilizing a large collection of affine linear representations tailored to particular regions of…
In the recent literature the important role of depth in deep learning has been emphasized. In this paper we argue that sufficient width of a feedforward network is equally important by answering the simple question under which conditions the decision regions of a neural network are connected. It turns out that for a cl…
The weight initialization and the activation function of deep neural networks have a crucial impact on the performance of the training procedure. An inappropriate selection can lead to the loss of information of the input during forward propagation and the exponential vanishing/exploding of gradients during back-propag…
Deep neural networks are often trained in the over-parametrized regime (i.e. with far more parameters than training examples), and understanding why the training converges to solutions that generalize remains an open problem. Several studies have highlighted the fact that the training procedure, i.e. mini-batch Stochas…
Transformers can outperform feedforward and recurrent networks due to dynamic sparsity.
problem Understanding when and why Transformers outperform other neural network architectures.
method Analyzing a sequence-to-sequence data generating model with dynamic sparsity, proving sample complexity differences between feedforward, recurrent, and Transformers.
result Transformers can learn dynamic sparsity models with lower sample complexity than feedforward and recurrent networks.