arXiv research

Study on feature learning dynamics in infinite-depth neural networks, focusing on ResNets.

AdaEnsemble learns adaptive feature interactions for CTR prediction.

Study examines infinite limits of transformer dynamics, identifying key parameterizations.

Residual networks with depthwise hyperparameter scaling transfer optimal hyperparameters across width and depth.

Researchers quantify the relationship between feature depth and performance in deep neural networks.

Inverse depth scaling found in LLMs due to similar layers averaging error.

Dropout and RaM become equivalent in large ResNets as depth and width increase.

Study explores how dataset breadth and depth affect Siamese Neural Network performance.

Study links neural network inductive bias, feature learning, and generalization on Boolean functions.

Improved neural network depth-width trade-offs via dynamical systems.

New algorithms improve multi-task learning across different environments.

Transformers can learn noisy linear systems with depth and IID data.

This paper improves neural network generalization by dynamically learning kernel parameters.

Study reveals how Fisher information changes with network depth, finding it grows linearly.

Aux-Net model handles dynamic systems with inconsistent inputs.

Scene recognition with RGB images has been extensively studied and has reached very remarkable recognition levels, thanks to convolutional neural networks (CNN) and large scene datasets. In contrast, current RGB-D scene data is much more limited, so often leverages RGB large datasets, by transferring pretrained RGB CNN…

Multimodal fusion frameworks for Human Action Recognition (HAR) using depth and inertial sensor data have been proposed over the years. In most of the existing works, fusion is performed at a single level (feature level or decision level), missing the opportunity to fuse rich mid-level features necessary for better cla…

DFR models dynamic distributional data with weighted Fréchet means.

This study presents a novel, deep, fully convolutional architecture which is optimized for the task of EEG-based neonatal seizure detection. Architectures of different depths were designed and tested; varying network depth impacts convolutional receptive fields and the corresponding learned feature complexity. Two deep…

ContinuousNet generalizes ResNets to continuous dynamical systems.

A leading hypothesis for the surprising generalization of neural networks is that the dynamics of gradient descent bias the model towards simple solutions, by searching through the solution space in an incremental order of complexity. We formally define the notion of incremental learning dynamics and derive the conditi…

Convolutional neural networks (CNN) are generally designed with a heuristic initialization of network architecture and trained for a certain task. This often leads to overparametrization after learning and induces redundancy in the information flow paths within the network. This robustness and reliability is at the inc…

We propose an analytically tractable class of models for the dynamics of a limit order book, described through a stochastic partial differential equation (SPDE) with multiplicative noise for the order book centered at the mid-price, along with stochastic dynamics for the mid-price which is consistent with the order flo…

Unified spectral framework for μP under joint width-depth scaling.

Neural network models have a reputation for being black boxes. We propose to monitor the features at every layer of a model and measure how suitable they are for classification. We use linear classifiers, which we refer to as "probes", trained entirely independently of the model itself. This helps us better understand …

A new data-level recombination strategy improves RGB-D salient object detection.

The study analyzes deep linear networks from random initialization, capturing dynamics and hyperparameter effects.

Attention-only transformers learn from context via two stages of inference.

This note explores the consequences of nonlinear price impact functions on price dynamics within the chartist-fundamentalist framework. Price impact functions may be nonlinear with respect to trading volume. As indicated by recent empirical studies, a given transaction may cause a large (small) price change if market d…

Study of urban lifestyles from mobility data of 1.2M people in 11 U.S. cities.

Method combines LD and Fermat Distance for neural network uncertainty.

Deeper networks are better for local labels, but shallower for global labels.

Defines complexity measure for neural networks and feature representations, revealing scaling patterns.

Continuous deep learning architectures have recently re-emerged as Neural Ordinary Differential Equations (Neural ODEs). This infinite-depth approach theoretically bridges the gap between deep learning and dynamical systems, offering a novel perspective. However, deciphering the inner working of these models is still a…

Neural networks' feature geometry evolves like discrete Ricci flow.

Improved isolation forest for better outlier detection.

Analysis of deep neural networks under various learning rules reveals dynamics of feature and prediction learning.

Neural GDEs improve graph prediction by blending discrete structures and differential equations.

We investigated the feature map inside deep neural networks (DNNs) by tracking the transport map. We are interested in the role of depth (why do DNNs perform better than shallow models?) and the interpretation of DNNs (what do intermediate layers do?) Despite the rapid development in their application, DNNs remain anal…

ADMP-GNN dynamically adjusts message-passing layers for better graph learning performance.

We prove the precise scaling, at finite depth and width, for the mean and variance of the neural tangent kernel (NTK) in a randomly initialized ReLU network. The standard deviation is exponential in the ratio of network depth to width. Thus, even in the limit of infinite overparameterization, the NTK is not determinist…

Expressivity is one of the most significant issues in assessing neural networks. In this paper, we provide a quantitative analysis of the expressivity for the deep neural network (DNN) from its dynamic model, where the Hilbert space is employed to analyze the convergence and criticality. We study the feature mapping of…

Hypersolvers enable fast continuous-depth models for practical applications.

Transformer learns to search through reinforcement learning, mimicking DFS.

Large neural networks learn low-dimensional representations that balance complexity and regularity.

New approach to abstract neural network representations using renormalization group.

New insights into GNN optimization reveal skip connections and depth accelerate training.

Graphs avoid oversmoothing with properly initialized weights.