This paper analyzes Stochastic Depth regularization in ResNets.
problem Improving generalization in ResNets through regularization.
method Hybrid analysis combining perturbation and signal propagation.
result Principled guidelines for choosing survival rates in SD.
We develop a scalable method for Bayesian neural networks with stochastic differential equations.
problem Uncertainty quantification in deep neural networks.
method Gradient-based stochastic variational inference in continuous-depth Bayesian neural networks.
result Gradient estimator with zero variance as the approximation improves.
Paper introduces MCSD, a method for uncertainty estimation in deep learning.
problem Need for reliable uncertainty quantification in deep neural networks.
method Theoretical connection to variational inference and empirical benchmarking of MCSD.
result MCSD achieves competitive predictive accuracy and improves uncertainty ranking.
Neural operators improve solving Helmholtz equation for various wave speeds.
problem Neural operators struggle with out-of-distribution scenarios for high-frequency waves.
method Proposed a subfamily of neural operators with stochastic depth for enhanced approximation of the Helmholtz equation.
result Neural operators with stochastic depth outperform standard models in out-of-distribution scenarios.
This paper explores how neural network width and depth behave as they approach infinity.
problem Understanding the behavior of neural functions as width and depth go to infinity.
method Formal definition of commutativity framework, study of neural covariance kernel, novel proof techniques.
result Taking width and depth to infinity in a deep neural network with skip connections results in the same covariance structure, regardless of the order of taking limits.
New Transformer architecture prevents rank degeneracy in deep attention models.
problem Rank degeneracy in deep attention models.
method Modified Softmax-based attention model with skip connections, centered at identity, and scaled logits.
result Existence of a stable SDE implies well-behaved covariance structure, preventing rank degeneracy.
We propose an analytically tractable class of models for the dynamics of a limit order book, described through a stochastic partial differential equation (SPDE) with multiplicative noise for the order book centered at the mid-price, along with stochastic dynamics for the mid-price which is consistent with the order flo…
Study on neural network initialization with shaped infinite depth-and-width networks.
problem Understanding the distribution of random covariance matrices in shaped infinite-depth-and-width networks.
method Introduced the Neural Covariance SDE to model the distribution of the random covariance matrix.
result Identified the precise scaling of the activation function necessary for a non-trivial limit.
We propose a generic framework to calibrate accuracy and confidence of a prediction in deep neural networks through stochastic inferences. We interpret stochastic regularization using a Bayesian model, and analyze the relation between predictive uncertainty of networks and variance of the prediction scores obtained by …
Poly-GNNs achieve similar performance regardless of depth, highlighting graph noise's dominance.
problem Performance of poly-GNNs in semi-supervised node classification.
method Analysis of poly-GNNs under a contextual stochastic block model (CSBM).
result For a sufficiently large graph, depth k>1 poly-GNNs exhibit the same rate of separation as depth k=1 counterparts. GoTube verifies neural networks over time, scaling to large horizons.
problem Verifying the robustness of time-continuous neural networks.
method Solves Go problems to construct a conservative execution set.
result Substantially outperforms existing tools in size, speed, and scalability.
Recent advances in bandit tools and techniques for sequential learning are steadily enabling new applications and are promising the resolution of a range of challenging related problems. We study the game tree search problem, where the goal is to quickly identify the optimal move in a given game tree by sequentially sa…
A study on the depth of graph neural networks on sparse graphs, revealing a dichotomy based on the Kesten-Stigum ratio.
problem Determining the optimal depth of graph neural networks for sparse graphs.
method Analyzing the sparse contextual stochastic block model with a message-passing classifier.
result The value of depth is governed by the Kesten-Stigum ratio, with thresholds dividing performance into geometric and branching processes.
We show that the standard stochastic gradient decent (SGD) algorithm is guaranteed to learn, in polynomial time, a function that is competitive with the best function in the conjugate kernel space of the network, as defined in Daniely, Frostig and Singer. The result holds for log-depth networks from a rich family of ar…
This work generalizes bounds on the number of linear regions in CPWL NNs.
problem Determining the number of linear regions in CPWL neural networks is challenging.
method Generalized bounds on the maximal number of linear regions for arbitrary CPWL activation functions.
result Depth significantly increases the number of linear regions, but not exponentially.
Paper tackles active labeling for partial supervision.
problem Accessing stochastic gradients with partial supervision.
method Streaming technique to minimize generalization error.
result Proves minimization of generalization error ratio.
Deep ResNets exhibit distinct scaling properties with depth, challenging neural ODE models.
problem Understanding the scaling properties of deep ResNets and their relation to neural ODEs.
method Detailed numerical experiments on weights trained by stochastic gradient descent.
result Deep ResNets can exhibit different scaling regimes, including stochastic differential equations or neither, challenging the neural ODE model.
Transformer learns to search through reinforcement learning, mimicking DFS.
problem Understanding how transformers learn search capabilities in RL.
method Two-head transformer, depth-wise curriculum, discounted returns.
result Transformer policy generalizes depth and prioritizes high-probability branches.
Model quantifies market price of trading liquidity risk and market depth.
problem Analyzing the market price of trading liquidity risk and market depth.
method Introduced a framework to analyze market price of liquidity risk, derived inhomogeneous Bernoulli ODE, obtained closed form solutions.
result Market depth encapsulates the market price of liquidity risk.
New tool for parallel and private stochastic convex optimization reduces query complexity.
problem Parallel and private stochastic convex optimization with reduced query complexity.
method Reweighted Stochastic Query (ReSQue) estimator combined with ball oracle acceleration.
result Achieves state-of-the-art complexities for SCO in parallel and private settings.
Study on Kyle's model with stochastic liquidity impacts asset volatility.
problem Impact of stochastic volatility of noise trading on asset volatility.
method Construct equilibrium for continuous-time Kyle's model with stochastic liquidity.
result In equilibrium, Kyle's Lambda and its inverse are submartingales.
Develops EFT for ResNets, revealing limitations of kernel-only approach.
problem Limitations of kernel-only approach in deep neural networks.
method Collective kernel EFT for pre-activation ResNets based on G-only closure hierarchy. result Numerical findings show V4 equation residual accumulates to an O(1) error. Dropout and RaM become equivalent in large ResNets as depth and width increase.
problem Improving performance in deep learning models.
method Comparing Dropout and Random Gradient Masking in ResNets.
result Dropout and RaM converge to the same large-scale limiting dynamics in ResNets.
Study defends shallow neural networks from data-poisoning attacks.
problem Protecting shallow neural networks from adversarial attacks during training.
method Developed a non-gradient stochastic algorithm for depth-2 neural networks, proving near-optimal trade-offs.
result Demonstrated improved performance over stochastic gradient descent under various data distributions.
Study on deep multi-head self-attention dynamics, proving homogenized limits under specific scalings.
problem Understanding the behavior of deep multi-head self-attention models as depth increases.
method Random model of deep multi-head self-attention, viewing depth as time, and analyzing the residual stream as a particle system.
result Homogenized limit of the dynamics, leading to deterministic or stochastic behavior depending on scaling, with implications for representation collapse.
Optimal trade execution in a fluctuating market with stochastic liquidity.
problem Minimizing costs in a market with unpredictable liquidity.
method Developed a recursion to find the least costly trade execution strategy.
result Explicit recursion characterizes the least costly trade execution.
Study introduces a probabilistic framework for air-sea fluxes using neural networks.
problem Accurately quantifying air-sea fluxes for understanding interactions and improving weather/climate models.
method Gaussian distributions conditioned on input variables, artificial neural networks, eddy-covariance data, minimizing negative log-likelihood loss.
result Trained neural networks provide alternative mean flux estimates and quantify uncertainty.
The thesis examines stochastic calculus in option pricing with logistic models and numerical methods.
problem Exploring the application of stochastic calculus in option pricing.
method Monte-Carlo Simulation and machine learning algorithms.
result Insights from Peter Carr and Lorenzo Torricelli's convex duality in continuous models.
The study analyzes deep linear networks from random initialization, capturing dynamics and hyperparameter effects.
problem Understanding training dynamics in deep linear networks from random initialization.
method Theoretical analysis of gradient descent dynamics in deep linear networks with random initialization and large data.
result Captures the 'wider is better' effect and hyperparameter transfer effects, contrasting with neural-tangent parameterization.
New model shows negative resilience can improve trading efficiency.
problem Optimal trade execution in limit order books with negative resilience.
method Stochastic order book model with negative resilience.
result Negative resilience can lead to more efficient trading.
The paper studies how noise synchronizes tokens in deep transformer models.
problem Understanding synchronization in deep learning models with noise.
method Proves convergence to a stochastic particle system and identifies the limiting SDE.
result The limiting model displays synchronization by noise and exponential dissipation of interaction energy.
Develops a new theory for neural systems stability and width effects.
problem Stability and finite-width effects in deep neural systems.
method Gauge-covariant stochastic effective field theory using classical commuting fields.
result Predicts the edge of chaos and low-frequency spectral deformation.
Neural GDEs improve graph prediction by blending discrete structures and differential equations.
problem Dynamic graph prediction challenges in irregularly sampled data.
method Continuous-depth graph neural networks (GNNs) with Neural GDEs.
result Neural GDEs enhance performance across various applications.
Batch normalization makes deep neural networks' representations increasingly orthogonal.
problem Orthogonality of deep neural network representations.
method Random linear transformations in successive batch-normalizations.
result Orthogonality of representations improves SGD performance.
When the parameters are independently and identically distributed (initialized) neural networks exhibit undesirable properties that emerge as the number of layers increases, e.g. a vanishing dependency on the input and a concentration on restrictive families of functions including constant functions. We consider parame…
SGD fails to converge for deep ReLU networks with limited random initializations.
problem SGD convergence in deep neural networks with limited random initializations.
method Analysis of four discretization parameters: network architecture, training data, gradient steps, and random initializations.
result SGD fails to converge for ReLU networks with depth much larger than width.
A new depth measure for non-convex data supports, faster than halfspace depth.
problem Non-convex data supports in multivariate statistics.
method Extending halfspace depth to Reproducing Kernel Hilbert Space (RKHS).
result The new depth measure is consistent and can be computed faster.
The paper studies randomized approximations of Tukey's depth for log-concave isotropic data.
problem The challenge of approximating Tukey's depth in high dimensions.
method The study examines randomized algorithms for approximating Tukey's depth for log-concave isotropic data.
result Randomized algorithms correctly approximate maximal depth and close to zero depths but not intermediate depths.
This paper connects functional data analysis with machine learning techniques.
problem Lack of theoretical analysis for functional depths.
method Viewing functional depths as kernel mean embeddings in machine learning.
result Facilitates answers to open questions about functional depths.
Self-attention models benefit equally from width and depth, but beyond a certain point, depth becomes less efficient.
problem Understanding the optimal balance between depth and width in self-attention models.
method Theoretical predictions and empirical ablations on networks of varying depths and widths.
result An optimal width of 30K is recommended for a 1-Trillion parameter network, marking a significant width for self-attention models.
Derives token price process for AMM tokens, finds leverage effect and pricing discrepancies.
problem Derives token price process for AMM tokens.
method Derives CEV process for token price, derives closed-form option prices, introduces liquidity-adjusted Greeks.
result Token price process is CEV, with leverage effect and pricing discrepancies.
A new depth measure based on optimal control theory captures multi-modal data.
problem Statistical depths for high-dimensional data.
method Eikonal equations and optimal control theory.
result The new depth measure is robust under adversarial models.
Signature volatility models are analyzed for existence, arbitrage, completeness, and hedging-error decomposition.
problem Existence, arbitrage, completeness, and hedging-error decomposition of signature volatility models.
method Global existence and uniqueness of strong solutions, asset-pricing, market completeness, and hedging-error decomposition derived through structural results.
result Signature volatility models are structurally sound with existence, arbitrage, completeness, and hedging-error decomposition.
A new depth measure and median defined on Hadamard manifolds.
problem Statistical depth and median on Hadamard manifolds.
method Horospherical depth and Busemann median defined using renormalized distance functions.
result The Busemann median exists for every Borel probability measure on Hadamard manifolds.
New approach uses loss functions to extend data depth for anomaly detection.
problem Anomaly detection in high-dimensional data.
method Introducing loss depths to generalize halfspace depth.
result New loss depths improve anomaly detection efficiency and interpretability.
Paper proposes Sinkformers for Transformers with doubly stochastic attention.
problem Improving Transformer models' accuracy in vision and natural language processing.
method Using Sinkhorn's algorithm to make attention matrices doubly stochastic instead of SoftMax normalization.
result Sinkformers enhance model accuracy in vision and natural language processing tasks.
Proves depth 2 neural networks can't approximate certain functions as well as depth 3 networks.
problem Approximating functions with depth 2 networks in high dimensions.
method Lower bound proof using worst-to-average-case random self-reducibility.
result Proves depth 2 networks can't approximate certain functions as well as depth 3 networks, resolving an open problem.
New findings on depth vs. width in neural networks, showing depth can improve learnability.
problem Understanding the role of depth in neural networks, especially when width is unbounded.
method Analyzing sample complexity for learnability in norm-controlled depth-2 and depth-3 ReLU networks.
result Depth can improve learnability of functions that are otherwise unlearnable with depth-2 networks.