arXiv research

Study on feature learning dynamics in infinite-depth neural networks, focusing on ResNets.

Inverse depth scaling found in LLMs due to similar layers averaging error.

Improved neural network depth-width trade-offs via dynamical systems.

Transformers can learn noisy linear systems with depth and IID data.

Residual networks with depthwise hyperparameter scaling transfer optimal hyperparameters across width and depth.

Study reveals how Fisher information changes with network depth, finding it grows linearly.

ContinuousNet generalizes ResNets to continuous dynamical systems.

A leading hypothesis for the surprising generalization of neural networks is that the dynamics of gradient descent bias the model towards simple solutions, by searching through the solution space in an incremental order of complexity. We formally define the notion of incremental learning dynamics and derive the conditi…

Researchers dissect Neural ODEs to understand their dynamics.

We propose an analytically tractable class of models for the dynamics of a limit order book, described through a stochastic partial differential equation (SPDE) with multiplicative noise for the order book centered at the mid-price, along with stochastic dynamics for the mid-price which is consistent with the order flo…

New connection between DNNs and Sharkovsky's Theorem for depth-width trade-offs.

The study analyzes deep linear networks from random initialization, capturing dynamics and hyperparameter effects.

Attention-only transformers learn from context via two stages of inference.

This note explores the consequences of nonlinear price impact functions on price dynamics within the chartist-fundamentalist framework. Price impact functions may be nonlinear with respect to trading volume. As indicated by recent empirical studies, a given transaction may cause a large (small) price change if market d…

Study examines infinite limits of transformer dynamics, identifying key parameterizations.

Neural GDEs improve graph prediction by blending discrete structures and differential equations.

ADMP-GNN dynamically adjusts message-passing layers for better graph learning performance.

Study explores how dataset breadth and depth affect Siamese Neural Network performance.

AdaEnsemble learns adaptive feature interactions for CTR prediction.

Hypersolvers enable fast continuous-depth models for practical applications.

Transformer learns to search through reinforcement learning, mimicking DFS.

New insights into GNN optimization reveal skip connections and depth accelerate training.

Graph neural networks are extended to continuous-depth models using differential equations.

Dropout and RaM become equivalent in large ResNets as depth and width increase.

DUNs improve active learning by dynamically adjusting model complexity.

Study of infinitely deep but narrow neural networks using NTK theory.

UDN adapts depth to data complexity, outperforming standard neural networks.

New architectures improve KANs, making them more interpretable and accurate.

Despite the widespread practical success of deep learning methods, our theoretical understanding of the dynamics of learning in deep neural networks remains quite sparse. We attempt to bridge the gap between the theory and practice of deep learning by systematically analyzing learning dynamics for the restricted case o…

Study on deep multi-head self-attention dynamics, proving homogenized limits under specific scalings.

New ODE-Block handles stateful layers with continuous-in-depth functions using basis functions.

In order-driven markets, limit-order book (LOB) resiliency is an important microscopic indicator of market quality when the order book is hit by a liquidity shock and plays an essential role in the design of optimal submission strategies of large orders. However, the evolutionary behavior of LOB resilience around liqui…

New method uses higher-order Langevin dynamics for efficient parallel sampling.

New scaling framework for MoE architectures ensures stability and optimal performance at scale.

Uniform scaling limits in AdamW-trained transformers converge to ODEs.

Much sequential data exhibits highly non-uniform information distribution. This cannot be correctly modeled by traditional Long Short-Term Memory (LSTM). To address that, recent works have extended LSTM by adding more activations between adjacent inputs. However, the approaches often use a fixed depth, which is at the …

We study the dynamics of gradient descent on objective functions of the form $f(\prod_{i=1}^{k} w_i)$ (with respect to scalar parameters $w_1,\ldots,w_k$), which arise in the context of training depth-$k$ linear neural networks. We prove that for standard random initializations, and under mild assumptions on $f$, the n…

Investigates the relationship between ResNets and Neural ODEs, quantifying their closeness and providing training methods.

Global convergence of multilayer neural networks proven for any depth.

We refine toxicity bounds for dynamic liquidation incentives in CP-AMM systems.

Convolutional neural networks (CNN) are generally designed with a heuristic initialization of network architecture and trained for a certain task. This often leads to overparametrization after learning and induces redundancy in the information flow paths within the network. This robustness and reliability is at the inc…

Recent model-free reinforcement learning algorithms have proposed incorporating learned dynamics models as a source of additional data with the intention of reducing sample complexity. Such methods hold the promise of incorporating imagined data coupled with a notion of model uncertainty to accelerate the learning of c…

A new depth measure for non-convex data supports, faster than halfspace depth.

The paper studies randomized approximations of Tukey's depth for log-concave isotropic data.

New algorithms improve multi-task learning across different environments.

This paper connects functional data analysis with machine learning techniques.

Characterizing the dynamic interactive patterns of complex systems helps gain in-depth understanding of how components interrelate with each other while performing certain functions as a whole. In this study, we present a novel multimodal data fusion approach to construct a complex network, which models the interaction…

Deep neural networks can classify any dataset with exact solutions.