arXiv research

Inverse depth scaling found in LLMs due to similar layers averaging error.

Recent work in signal propagation theory has shown that dropout limits the depth to which information can propagate through a neural network. In this paper, we investigate the effect of initialisation on training speed and generalisation for ReLU networks within this depth limit. We ask the following research question:…

This work challenges the Neural Tangent Kernel's role in overparameterized neural networks, especially with large width and depth.

Study on neural network initialization with shaped infinite depth-and-width networks.

Dropout and RaM become equivalent in large ResNets as depth and width increase.

New method improves reliability of depth estimation models.

New method for multi-modal depth prediction challenges.

The paper proves barriers to approximating functions with small weights and depth in neural networks.

The paper explores how the depth of neural networks affects their ability to represent data accurately.

Study on functions computed by deep-layered machines finds same distribution in neural networks and Boolean circuits.

Scene recognition with RGB images has been extensively studied and has reached very remarkable recognition levels, thanks to convolutional neural networks (CNN) and large scene datasets. In contrast, current RGB-D scene data is much more limited, so often leverages RGB large datasets, by transferring pretrained RGB CNN…

Introduces Polar Depth for analyzing multivariate heavy-tailed data extremes.

We describe a procedure for creating infinite families of hyperbolic knots having unique minimal genus Seifert surface. A large subset of these knots have the further property that the surface cannot be the sole compact leaf of a depth one foliation of the knot exterior.

We prove the precise scaling, at finite depth and width, for the mean and variance of the neural tangent kernel (NTK) in a randomly initialized ReLU network. The standard deviation is exponential in the ratio of network depth to width. Thus, even in the limit of infinite overparameterization, the NTK is not determinist…

Large neural networks learn low-dimensional representations that balance complexity and regularity.

Adversarial examples in deep ReLU networks with constant depth.

Study on feature learning dynamics in infinite-depth neural networks, focusing on ResNets.

Stable ResNet stabilizes gradients in deep networks.

Bayesian linear networks reveal optimal depth and width trade-offs.

Study on learning rates in neural networks of varying depth.

Recent work by Jacot et al. (2018) has shown that training a neural network using gradient descent in parameter space is related to kernel gradient descent in function space with respect to the Neural Tangent Kernel (NTK). Lee et al. (2019) built on this result by establishing that the output of a neural network traine…

This study presents a novel, deep, fully convolutional architecture which is optimized for the task of EEG-based neonatal seizure detection. Architectures of different depths were designed and tested; varying network depth impacts convolutional receptive fields and the corresponding learned feature complexity. Two deep…

Study of deep linear neural networks with proportional width and depth.

Investigates the relationship between ResNets and Neural ODEs, quantifying their closeness and providing training methods.

This note explores the consequences of nonlinear price impact functions on price dynamics within the chartist-fundamentalist framework. Price impact functions may be nonlinear with respect to trading volume. As indicated by recent empirical studies, a given transaction may cause a large (small) price change if market d…

Deep architecture such as hierarchical semi-Markov models is an important class of models for nested sequential data. Current exact inference schemes either cost cubic time in sequence length, or exponential time in model depth. These costs are prohibitive for large-scale problems with arbitrary length and depth. In th…

We present a model for the joint estimation of disparity and motion. The model is based on learning about the interrelations between images from multiple cameras, multiple frames in a video, or the combination of both. We show that learning depth and motion cues, as well as their combinations, from data is possible wit…

The study analyzes deep linear networks from random initialization, capturing dynamics and hyperparameter effects.

Unified theory of deep neural networks with diverse activations.

Deep networks can perfectly classify two low-dimensional manifolds on a sphere with large depth and width.

Deep networks improve by progressively refining approximations at each layer.

Residual networks with depthwise hyperparameter scaling transfer optimal hyperparameters across width and depth.

In order-driven markets, limit-order book (LOB) resiliency is an important microscopic indicator of market quality when the order book is hit by a liquidity shock and plays an essential role in the design of optimal submission strategies of large orders. However, the evolutionary behavior of LOB resilience around liqui…

We show that the standard stochastic gradient decent (SGD) algorithm is guaranteed to learn, in polynomial time, a function that is competitive with the best function in the conjugate kernel space of the network, as defined in Daniely, Frostig and Singer. The result holds for log-depth networks from a rich family of ar…

New algorithm quantifies uncertainty in regression models for complex data types.

Our study analyzes how neural network initialization affects privacy and utility in overparameterized models.

The study analyzes games and social hierarchies, incorporating luck and depth of competition.

Study reveals how depth of reasoning affects generalization in models.

We present the notion of asymptotically large depth for a metric space which is (a priory) weaker than having subexponential asymptotic dimension growth and (a priory) stronger than property A.

We present a self-supervised approach to training convolutional neural networks for dense depth estimation from monocular endoscopy data without a priori modeling of anatomy or shading. Our method only requires monocular endoscopic videos and a multi-view stereo method, e.g., structure from motion, to supervise learnin…

Graphs avoid oversmoothing with properly initialized weights.

New Transformer architecture prevents rank degeneracy in deep attention models.

ML models predict water table depth more accurately than PB models, especially in data-scarce regions.

Poly-GNNs achieve similar performance regardless of depth, highlighting graph noise's dominance.

CoT enhances transformer accuracy on serial tasks by enabling serial computation.

Uniform scaling limits in AdamW-trained transformers converge to ODEs.

GoTube verifies neural networks over time, scaling to large horizons.

A new depth measure for non-convex data supports, faster than halfspace depth.