On-device research index

arXiv research

A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.

168,695 papers · 148 categories

Trend · papers per month

count % of corpus

41 results for “infinite-depth”

On the infinite-depth limit of finite-width neural networksstat.ML

Study infinite-depth limits of neural networks with fixed width.

problem Understanding the behavior of neural networks as depth increases with fixed width.

method Analyzing finite-width residual networks with random Gaussian weights, focusing on the infinite-depth limit.

result The pre-activations converge to a zero-drift diffusion process, differing from the infinite-width limit.

neural networks infinite-depth finite-width residual networks random weights

2022-10-03abs ↗pdf ↗

The Neural Covariance SDE: Shaped Infinite Depth-and-Width Networks at Initializationstat.ML

Study on neural network initialization with shaped infinite depth-and-width networks.

problem Understanding the distribution of random covariance matrices in shaped infinite-depth-and-width networks.

method Introduced the Neural Covariance SDE to model the distribution of the random covariance matrix.

result Identified the precise scaling of the activation function necessary for a non-trivial limit.

neural networks infinite depth infinite width covariance matrix stochastic differential equation

2022-06-06abs ↗pdf ↗

Wide Neural Networks as Gaussian Processes: Lessons from Deep Equilibrium Modelscs.LG

Wide neural networks converge to Gaussian processes, improving generalization.

problem Understanding the generalization of wide neural networks, especially deep equilibrium models.

method Investigation of deep equilibrium models (DEQs) with infinite-depth layers, focusing on their convergence to Gaussian processes as width and depth approach infinity.

result Wide DEQs converge to Gaussian processes, maintaining generalization performance.

deep equilibrium models neural networks Gaussian processes benign overfitting infinite-depth networks

2023-10-16abs ↗pdf ↗

The Future is Log-Gaussian: ResNets and Their Infinite-Depth-and-Width Limit at Initializationstat.ML

ResNets approximate log-Gaussian at initialization, improving network performance.

problem Understanding the initialization behavior of deep neural networks like ResNets.

method Analyzing ReLU ResNets in the infinite-depth-and-width limit, showing log-Gaussian behavior.

result ResNets at initialization exhibit hypoactivation and interlayer correlations, which are not captured by Gaussian limits.

ResNets log-Gaussian infinite-depth-and-width hypoactivation interlayer correlations

2021-06-07abs ↗pdf ↗

Neural Tangent Kernel Analysis of Deep Narrow Neural Networkscs.LG

Study of infinitely deep but narrow neural networks using NTK theory.

problem Analyzing the role of depth in deep learning with overparameterized networks.

method Infinite-depth limit analysis of MLP and CNN using Neural Tangent Kernel (NTK) theory.

result Established trainability guarantee for infinitely deep but narrow neural networks.

Neural Tangent Kernel Deep Learning Overparameterization Infinite Depth Narrow Neural Networks

2022-02-07abs ↗pdf ↗

Feature Learning Dynamics in Infinite-Depth Neural Networkscs.LG

Study on feature learning dynamics in infinite-depth neural networks, focusing on ResNets.

problem Understanding how features evolve during training in deep neural networks, especially in the large-depth limit.

method Conditional Gaussian representations and SDE system with decoupled backward weights.

result Depth-induced suppression of forward-backward coupling in infinite-depth networks, leading to a decoupled forward-backward SDE system.

Feature Learning Neural Networks Infinite Depth ResNets SDE System

2025-12-24abs ↗pdf ↗

The Shaped Transformer: Attention Models in the Infinite Depth-and-Width Limitstat.ML

New Transformer architecture prevents rank degeneracy in deep attention models.

problem Rank degeneracy in deep attention models.

method Modified Softmax-based attention model with skip connections, centered at identity, and scaled logits.

result Existence of a stable SDE implies well-behaved covariance structure, preventing rank degeneracy.

Transformer attention models infinite depth-and-width covariance matrix stochastic differential equation

2023-06-30abs ↗pdf ↗

Classification Fields: Arbitrarily Fine Recursive Hierarchical Clustering From Few Examplesstat.ML

Generates infinite-depth hierarchical clusters from few examples.

problem Inadequate finite-sample clustering methods for fine-scale hierarchical structures.

method Classification fields generated by a local refinement rule, approximated by predictors.

result Learned predictors can approximate infinite-depth hierarchical structures.

Hierarchical Clustering Infinite Depth Local Refinement Finite-Sample Exponential Convergence

2026-05-08abs ↗pdf ↗

Global Convergence of Over-parameterized Deep Equilibrium Modelscs.LG

DEQs converge to optimal solutions with mild over-parameterization.

problem Training over-parameterized deep equilibrium models.

method Solves equilibrium point directly, uses gradient descent, and analyzes convergence via linear rate.

result Gradient descent converges to a globally optimal solution at a linear rate for quadratic loss.

deep equilibrium models gradient descent over-parameterization equilibrium point linear convergence

2022-05-27abs ↗pdf ↗

Proportional infinite-width infinite-depth limit for deep linear neural networksstat.ML

Study of deep linear neural networks with proportional width and depth.

problem Lack of descriptive power in Gaussian limit of deep linear neural networks.

method Proportional infinite-width infinite-depth limit for deep linear neural networks.

result Characterization of limiting distribution as a nontrivial mixture of Gaussians.

deep learning neural networks infinite-width infinite-depth Gaussian process

2024-11-22abs ↗pdf ↗

On the inductive bias of infinite-depth ResNets and the bottleneck rankcs.LG

Deep ResNets favor low bottleneck rank with proper hyperparameters.

problem Understanding the inductive bias of deep neural networks.

method Computed minimum-norm weights of a deep linear ResNet.

result Deep nonlinear ResNets have an inductive bias towards minimizing bottleneck rank.

deep learning residual networks inductive bias bottleneck rank

2025-01-31abs ↗pdf ↗

Variational Inference for Infinitely Deep Neural Networkscs.LG

UDN adapts depth to data complexity, outperforming standard neural networks.

problem Adapting neural network depth to data complexity.

method Variational inference for infinitely deep neural networks with a novel algorithm.

result UDN outperforms standard neural networks and other infinite-depth approaches.

Variational inference Infinite depth Neural networks Adaptive complexity Gradient optimization

2022-09-21abs ↗pdf ↗

Stable ResNetcs.LG

Stable ResNet stabilizes gradients in deep networks.

problem Gradient vanishing and exploding in deep ResNet architectures.

method Introducing Stable ResNet architectures with gradient stabilization and infinite depth expressivity.

result Stable ResNet maintains gradient stability and expressivity in deep networks.

ResNet Gradient stabilization Deep learning Expressivity

2020-10-24abs ↗pdf ↗

Dissecting Neural ODEscs.LG

Researchers dissect Neural ODEs to understand their dynamics.

problem Understanding the inner workings of Neural ODEs.

method Developing continuous-depth formulation to clarify design choices.

result Clarified the influence of design choices on Neural ODE dynamics.

Neural ODEs Continuous Deep Learning Dynamical Systems Design Choices Influence on Dynamics

2020-02-19abs ↗pdf ↗

Bayesian Interpolation with Deep Linear Networksstat.ML

Bayesian linear networks reveal optimal depth and width trade-offs.

problem Understanding how depth, width, and dataset size affect model quality in linear networks.

method Zero noise Bayesian inference with Gaussian weight priors and mean squared error.

result Optimal predictions at infinite depth and maximized Bayesian model evidence at infinite depth.

Bayesian inference Linear networks Depth-width trade-offs Model selection Effective depth

2022-12-29abs ↗pdf ↗

Infinitely deep neural networks as diffusion processesstat.ML

When the parameters are independently and identically distributed (initialized) neural networks exhibit undesirable properties that emerge as the number of layers increases, e.g. a vanishing dependency on the input and a concentration on restrictive families of functions including constant functions. We consider parame…

2019-05-27abs ↗pdf ↗

Width and Depth Limits Commute in Residual Networksstat.ML

Theoretical limits of deep residual networks show consistent covariance structures.

problem Understanding the limits of deep residual networks.

method Analyzing the behavior of deep residual networks with skip connections as width and depth approach infinity.

result Theoretical analysis confirms that the covariance structure remains consistent regardless of the order of width and depth.

residual networks deep learning covariance structure skip connections infinite width

2023-02-01abs ↗pdf ↗

Infinite Limits of Multi-head Transformer Dynamicsstat.ML

Study examines infinite limits of transformer dynamics, identifying key parameterizations.

problem Understanding the training dynamics of transformer models in the feature learning regime.

method Analysis of infinite scaling limits using dynamical mean field theory.

result Identified parameterizations that admit well-defined infinite width and depth limits.

transformer models feature learning infinite limits dynamical mean field theory attention layers

2024-05-24abs ↗pdf ↗

Hypersolvers: Toward Fast Continuous-Depth Modelscs.LG

Hypersolvers enable fast continuous-depth models for practical applications.

problem Infinite-depth models like Neural ODEs are computationally infeasible for large problems.

method Introducing hypersolvers, neural networks that solve ODEs efficiently with theoretical guarantees.

result Hypersolvers achieve comparable inference time to traditional discrete networks, making continuous-depth models practical.

Neural ODEs Continuous-depth models Hypersolvers Inference efficiency Computational scalability

2020-07-19abs ↗pdf ↗

Parametric machines: a fresh approach to architecture searchcs.LG

New framework combines simple machines into complex ones for better neural network performance.

problem Improving neural network performance with limited training data.

method Developed a framework using topology and functional analysis to combine simple machines into complex ones, and used kernel methods to find optimal architectures.

result Kernel-inspired networks can outperform classical neural networks when training data is small.

neural networks machine learning topology functional analysis kernel methods

2020-07-06abs ↗pdf ↗

Deep Linear Network Training Dynamics from Random Initialization: Data, Width, Depth, and Hyperparameter Transfercs.LG

The study analyzes deep linear networks from random initialization, capturing dynamics and hyperparameter effects.

problem Understanding training dynamics in deep linear networks from random initialization.

method Theoretical analysis of gradient descent dynamics in deep linear networks with random initialization and large data.

result Captures the 'wider is better' effect and hyperparameter transfer effects, contrasting with neural-tangent parameterization.

deep linear networks random initialization gradient descent hyperparameter transfer mean-field

2025-02-04abs ↗pdf ↗

Rethinking Graph Regularization for Graph Neural Networkscs.LG

Propagation-regularization improves GNN performance by infusing extra graph information.

problem The effectiveness of graph Laplacian regularization in GNNs is questioned and improved upon.

method Introducing Propagation-regularization (P-reg) to enhance GNN performance.

result P-reg boosts GNN performance on various tasks across multiple datasets.

Graph Neural Networks Graph Laplacian Propagation-regularization Graph Convolutional Networks Semi-supervised Learning

2020-09-04abs ↗pdf ↗

Exact Convergence Rates of the Neural Tangent Kernel in the Large Depth Limitstat.ML

Recent work by Jacot et al. (2018) has shown that training a neural network using gradient descent in parameter space is related to kernel gradient descent in function space with respect to the Neural Tangent Kernel (NTK). Lee et al. (2019) built on this result by establishing that the output of a neural network traine…

2019-05-31abs ↗pdf ↗

Deep Equilibrium Modelscs.LG

We present a new approach to modeling sequential data: the deep equilibrium model (DEQ). Motivated by an observation that the hidden layers of many existing deep sequence models converge towards some fixed point, we propose the DEQ approach that directly finds these equilibrium points via root-finding. Such a method is…

2019-09-03abs ↗pdf ↗

DiffEnc: Variational Diffusion with a Learned Encodercs.LG

DiffEnc improves diffusion models by adding flexibility and achieving better likelihood on CIFAR-10.

problem Improving the likelihood of diffusion models on image datasets.

method Introducing a data- and depth-dependent mean function and a free weight parameter for noise variance.

result Achieved statistically significant improvement in likelihood on CIFAR-10.

Diffusion models Variational autoencoders CIFAR-10 Flexibility Likelihood improvement

2023-10-30abs ↗pdf ↗

Approximating Continuous Functions by ReLU Nets of Minimal Widthstat.ML

This article concerns the expressive power of depth in deep feed-forward neural nets with ReLU activations. Specifically, we answer the following question: for a fixed $d_{in}\geq 1,$ what is the minimal width $w$ so that neural nets with ReLU activations, input dimension $d_{in}$ , hidden layer widths at most $w,$ and …

2017-10-31abs ↗pdf ↗

Depthwise Hyperparameter Transfer in Residual Networks: Dynamics and Scaling Limitstat.ML

Residual networks with depthwise hyperparameter scaling transfer optimal hyperparameters across width and depth.

problem The challenge of hyperparameter tuning in deep learning, especially for large models.

method Combining

μ

P parameterization with residual networks having a residual branch scale of

1/\sqrt{ ext{depth}}

result Optimal hyperparameters transfer across width and depth in residual networks trained with this parameterization.

hyperparameter tuning residual networks depthwise scaling mean field theory neural network dynamics

2023-09-28abs ↗pdf ↗

Expressive Power of Implicit Models: Rich Equilibria and Test-Time Scalingcs.LG

Implicit models can match or exceed explicit models with more test-time compute.

problem Understanding the expressive power and scaling of implicit models.

method Nonparametric analysis of expressive power, mathematical characterization of implicit operators, and test-time scaling experiments.

result Implicit models can progressively express more complex mappings through iteration, matching a richer function class with test-time compute.

implicit models expressive power test-time scaling nonparametric analysis iterative computation

2025-10-04abs ↗pdf ↗

On Second Order Behaviour in Augmented Neural ODEscs.LG

SONODEs and ANODEs improve learning of second order dynamics.

problem Learning dynamics governed by second order laws.

method Extended adjoint sensitivity method and theoretical analysis of ANODEs.

result SONODEs and ANODEs can learn higher order dynamics efficiently.

Second Order Neural ODEs Augmented Neural ODEs Dynamics Learning Inductive Biases Optimisation

2020-06-12abs ↗pdf ↗

Bottleneck Structure in Learned Features: Low-Dimension vs Regularity Tradeoffcs.LG

Large neural networks learn low-dimensional representations that balance complexity and regularity.

problem Understanding the tradeoff between low-dimensional representations and complexity in deep neural networks.

method Computed finite depth corrections to reveal a measure of regularity that bounds the pseudo-determinant of the Jacobian.

result Proved the conjectured bottleneck structure in learned features as network depth increases, showing almost all hidden representations are approximately low-dimensional and weight matrices have singular values close to 1.

Deep Learning Neural Networks Bottleneck Structure Regularization Rank

2023-05-30abs ↗pdf ↗

Stochastic Gradient Descent on a Tree: an Adaptive and Robust Approach to Stochastic Convex Optimizationstat.ML

Online minimization of an unknown convex function over the interval $[0,1]$ is considered under first-order stochastic bandit feedback, which returns a random realization of the gradient of the function at each query point. Without knowing the distribution of the random gradients, a learning algorithm sequentially choo…

2019-01-17abs ↗pdf ↗

Hamiltonian Mechanics of Feature Learning: Bottleneck Structure in Leaky ResNetsstat.ML

Study on feature learning in Leaky ResNets, explaining bottleneck structure.

problem Understanding feature learning in deep neural networks.

method Lagrangian and Hamiltonian reformulation of representation geodesics.

result Emergence of a bottleneck structure in large effective depth networks.

Leaky ResNets Feature Learning Hamiltonian Mechanics Representation Geodesics Bottleneck Structure

2024-05-27abs ↗pdf ↗

Gradient Descent Optimizes Infinite-Depth ReLU Implicit Networks with Linear Widthscs.LG

Gradient descent converges to a global minimum in nonlinear ReLU implicit networks with linear width.

problem Understanding convergence of gradient methods in nonlinear, infinitely deep ReLU networks.

method Introduced a scaling constant to ensure well-posedness of the equilibrium equation, proving convergence to a global minimum for linear width networks.

result Gradient descent converges to a global minimum at a linear rate for nonlinear ReLU implicit networks with linear width.

gradient descent ReLU networks implicit learning nonlinear optimization convergence analysis

2022-05-16abs ↗pdf ↗

Continuum Transformers Perform In-Context Learning by Operator Gradient Descentstat.ML

Continuum transformers learn operators in context via gradient descent.

problem Generalizing transformers to handle infinite-dimensional inputs for in-context learning.

method Gradient descent in an operator RKHS, leveraging generalized representer theorems and gradient flows.

result Operator learned in context is Bayes Optimal Predictor in infinite depth limit.

continuum transformers in-context learning gradient descent operator RKHS Bayes Optimal Predictor

2025-05-23abs ↗pdf ↗

Differential Equation Scaling Limits of Shaped and Unshaped Neural Networksstat.ML

New findings connect shaped and unshaped neural networks using differential equations.

problem Understanding the behavior of neural networks with different activation scaling methods.

method Deriving differential equation-based asymptotic characterizations for shaped and unshaped neural networks.

result Two types of unshaped networks converge to the same infinite-depth-and-width limit at initialization.

neural networks differential equations activation scaling shaped networks unshaped networks

2023-10-18abs ↗pdf ↗

Laminations with transverse measures in ordered abelian semigroupsmath.GT

We describe a construction of ordered algebraic structures (ordered abelian semigroups, ordered commutative semirings, etc.) and describe applications to codimension-1 laminations. For a suitable ordered semi- algebraic structure $\mathbb L$ and measurable space $X$ we define $\mathbb L$ -measures $ν$ on $X$ . If $L$ is …

2014-07-25abs ↗pdf ↗

Robust Implicit Networks via Non-Euclidean Contractionscs.LG

New framework improves robustness of implicit neural networks.

problem Ill-posedness and convergence instability in implicit neural networks.

method NEMON framework based on contraction theory for

\ell_{\infty}

norm, including well-posedness condition, average iteration, and input-output Lipschitz constant regularization.

result Improved accuracy and robustness of implicit models with smaller input-output Lipschitz bounds.

Implicit Neural Networks Non-Euclidean Norm Fixed Point Equations Contraction Theory Robustness

2021-06-06abs ↗pdf ↗

Deep Equilibrium Based Neural Operators for Steady-State PDEscs.LG

FNO-DEQ solves steady-state PDEs as fixed points, outperforming traditional FNOs.

problem Lack of understanding in designing neural network architectures for PDEs.

method Proposes FNO-DEQ, a deep equilibrium architecture that solves steady-state PDEs as fixed points.

result FNO-DEQ outperforms FNO-based architectures in predicting solutions to steady-state PDEs.

partial differential equations steady-state deep equilibrium fixed point universal approximation

2023-11-30abs ↗pdf ↗

Do ideas have shape? Idea registration as the continuous limit of artificial neural networksstat.ML

ResNets and their GP generalization align ideas via function space warping, revealing robust properties and connections to image registration.

problem Aligning abstract shapes (ideas) using neural networks.

method Introducing a generalization of ResNets as a GP, showing convergence to image registration variational algorithms, and revealing properties via a Hamiltonian interpretation.

result ResNets and their GP generalization align ideas via function space warping, revealing robust properties and connections to image registration.

ResNets GP generalization image registration functional limits Hamiltonian dynamics

2020-08-10abs ↗pdf ↗

ResNets of All Shapes and Sizes: Convergence of Training Dynamics in the Large-scale Limitstat.ML

ResNets converge to a limit model with improved error rates.

problem Understanding convergence of ResNets in the large-scale limit.

method Combining cavity method and propagation of chaos arguments on skeleton maps.

result Convergence rate of O(1/sqrt(D)) for ResNets in the large-scale limit.

ResNets Convergence Large-scale Limit Cavity Method Propagation of Chaos

2026-03-18abs ↗pdf ↗

Transformers through the lens of support-preserving maps between measurescs.CL

Transformers preserve support and can approximate any continuous map.

problem Understanding the mathematical properties of transformers.

method Characterizing maps between measures that can be represented as transformers and proving their properties.

result Transformers preserve support and have uniformly continuous Fréchet derivatives.

transformers maps between measures support preservation mean-field limit Vlasov flow

2025-09-30abs ↗pdf ↗