Extends active subspace analysis to infinite dimensions.
problem Dimension reduction in infinite dimensional functionals.
method Defines an operator for Hilbert space, extends Euclidean properties, proposes Monte Carlo procedure.
result Desirable properties extend to infinite dimensional setting.
Study infinite-depth limits of neural networks with fixed width.
problem Understanding the behavior of neural networks as depth increases with fixed width.
method Analyzing finite-width residual networks with random Gaussian weights, focusing on the infinite-depth limit.
result The pre-activations converge to a zero-drift diffusion process, differing from the infinite-width limit.
Derives time-averaged active inference from control principles.
problem Finite-horizon or discounted-surprise problems in active inference.
method Derives infinite-horizon, average-surprise active inference from optimal control principles.
result Unified objective functional for sensorimotor control.
Study on neural network initialization with shaped infinite depth-and-width networks.
problem Understanding the distribution of random covariance matrices in shaped infinite-depth-and-width networks.
method Introduced the Neural Covariance SDE to model the distribution of the random covariance matrix.
result Identified the precise scaling of the activation function necessary for a non-trivial limit.
Develops a numerical method for LRM strategies in BNS models with infinite active jumps.
problem Calculating locally risk-minimizing strategies for non-martingale BNS models with infinite active jumps.
method Modified Malliavin calculus expression and Monte Carlo method for non-martingale BNS models.
result Proposes a numerical method for LRM strategies in non-martingale BNS models with infinite active jumps.
Study of deep Stable neural networks with various activation functions.
problem Characterizing the infinitely wide limits of deep Stable neural networks.
method Investigation of large-width properties of deep Stable NNs with a generalized central limit theorem for heavy tails.
result Extension of characterization to a broader class of activation functions, including sub-linear, asymptotically linear, and super-linear functions.
Study shows how activation functions impact the storage capacity of treelike neural networks.
problem Understanding the role of activation functions in neural network expressive power.
method Analysis of treelike two-layer networks with various activation functions in the infinite-width limit.
result Activation functions affect storage capacity and robustness, with nonlinearity increasing capacity and decreasing robustness.
Characterizes neural kernel and NNGP for various activations.
problem Understanding neural kernels and NNGP for non-RELU activations.
method Characterization of RKHS for various activation functions.
result Broad class of non-infinitely smooth activations generate equivalent RKHSs at different depths.
Method extends option valuation for 2D Lévy models.
problem Valuation of European options under 2-asset infinite-activity Lévy models.
method Developed numerical method extending Wang et al. (2007) for 1D to 2D, using Fourier transform for integral term and semi-Lagrangian theta-method for temporal discretization.
result Favourable second-order convergence for Normal Tempered Stable dynamics.
Hybridizes physical and data-driven methods for predicting physicochemical properties.
problem Predicting physicochemical properties accurately using limited data.
method Distills physical method predictions into a prior model and combines with sparse experimental data using Bayesian inference.
result Significant improvements in predicting activity coefficients at infinite dilution compared to baselines and ensemble methods.
The paper connects neural networks to physics using probability theory.
problem Creating neural networks that follow physical laws.
method Applying the central limit theorem and Gaussian process theory to neural networks.
result Neural networks can be designed to obey physical laws by choosing appropriate activation functions.
Study on critical points in random neural networks, revealing three regimes based on activation function.
problem Investigating the expected number of critical points in random neural networks.
method Deriving asymptotic formulas for critical points under infinite-width limit and suitable regularity conditions.
result Three distinct regimes of critical points behavior depending on activation function.
We develop a polynomial method to optimize trading in markets with transaction costs.
problem Optimizing trading strategies in markets with proportional transaction costs.
method Polynomial approximation of the residual value function to determine optimal trading strategies.
result Identify the trade-off between trading frequency and trade sizes for satisfactory agreement with theoretically optimal strategies.
Unified framework for series representations and finite approximations of CRMs.
problem Challenges in exact simulation and scalable inference with infinite-activity CRMs.
method Unified framework based on size-biased sampling of Poisson point process.
result Novel series representations for generalized gamma and stable beta processes.
Piecewise linear activations create many spurious local minima in neural networks.
problem Understanding the loss surface of neural networks with piecewise linear activations.
method Proved the existence of infinite spurious local minima and partitioned the loss surface into smooth cells.
result Piecewise linear activations create many spurious local minima that are invariant under a continuous path.
Deep neural networks' infinite-width behavior approximated by Gaussian models.
problem Understanding the behavior of deep neural networks in the limit of infinite width.
method Using the Lindeberg exchange principle to approximate weights by Gaussian random variables.
result Quantitative bounds on the 2-Wasserstein distance between deep neural networks and Gaussian limits.
Complex-valued neural networks avoid spurious local minima.
problem Finding spurious local minima in neural networks.
method Proved no spurious local minima for shallow complex neural networks with quadratic activations.
result Complex-valued weights eliminate spurious local minima in neural networks.
The infinite Viterbi alignment is the limiting maximum a-posteriori estimate of the unobserved path in a hidden Markov model as the length of the time horizon grows. For models on state-space Rd satisfying a new ``decay-convexity'' condition, we develop an approach to existence of the infinite Viterbi ali…
In this article, we consider a Markov process X, starting from x and solving a stochastic differential equation, which is driven by a Brownian motion and an independent pure jump component exhibiting state-dependent jump intensity and infinite jump activity. A second order expansion is derived for the tail probability …
Study embeddings between Barron spaces with various activation functions, focusing on RePU.
problem Understanding the influence of activation functions on infinitely wide neural networks.
method Prove embeddings by constructing push-forward maps on measures representing functions.
result Barron spaces with RePU activation have a hierarchical structure similar to Sobolev spaces.
This work challenges the Neural Tangent Kernel's role in overparameterized neural networks, especially with large width and depth.
problem The Neural Tangent Kernel's behavior in overparameterized neural networks with large width and depth is unclear.
method Experimental and theoretical analysis of ReLU networks with large width and depth.
result The aggregate norm of hidden neuron deviations does not vanish in infinitely-wide ReLU networks, indicating non-trivial behavior.
Infinite CNNs lose spatial correlations, but can be restored by correlated weights.
problem Infinite CNNs lose spatial correlations, which are crucial for their performance.
method Introduced correlated weights to restore spatial correlations in infinite CNNs.
result Optimal performance is achieved with a moderate level of weight correlation.
Polynomial-time convex optimization for CNNs with ReLU activations.
problem Training Convolutional Neural Networks (CNNs) with ReLU activations.
method Developed a convex analytic framework using semi-infinite duality to formulate equivalent convex optimization problems for CNN architectures.
result Proved that two-layer CNNs can be globally optimized via an ℓ2 norm regularized convex program. We analyse the behaviour of the implied volatility smile for options close to expiry in the exponential Lévy class of asset price models with jumps. We introduce a new renormalisation of the strike variable with the property that the implied volatility converges to a non-constant limiting shape, which is a function of …
This paper studies large-width asymptotics for ReLU neural networks with α-Stable initializations.
problem Characterizing the large-width behavior of ReLU neural networks with α-Stable initializations.
method Analysis of the large-width distributions and training dynamics of ReLU neural networks initialized with α-Stable distributions.
result For ReLU neural networks with α-Stable initializations, the large-width training dynamics achieve zero training error at a linear rate, characterized by a random kernel.
We develop a method to learn neural network activations with controlled Lipschitz constant.
problem Increase neural network capacity while controlling Lipschitz constant.
method Variational framework to learn activation functions with piecewise-linear constraints.
result Proves existence of solutions with continuous and piecewise-linear activations.
Analyzes feature learning in neural networks using a self-consistent dynamical field theory.
problem Feature learning in infinite-width neural networks.
method Constructs deterministic dynamical order parameters as inner-product kernels for hidden unit activations and gradients.
result Reveals the hidden layer activation distribution, neural tangent kernel evolution, and output predictions.
Study on function sensitivity in random DNNs using large deviation theory.
problem Understanding function sensitivity in finite-size deep neural networks.
method Large deviation theory and path integral analysis applied to random DNNs with ReLU and sign activations.
result Random DNNs with ReLU activations are more robust to parameter perturbations.
Wide neural networks can be closely approximated by Gaussian processes, with rates depending on the activation function's properties.
problem Approximating the behavior of wide neural networks using Gaussian processes.
method Established convergence rates for the central limit theorem in an infinite-dimensional functional space, using a transportation distance metric.
result Explicit convergence rates for neural networks approximated by Gaussian processes, varying based on the activation function's properties.
This paper characterizes how randomized neural networks generalize well in multi-dimensional tasks.
problem Understanding the generalization of randomized neural networks in multi-dimensional tasks.
method Characterizes RSNs as an IGAM formalized by an optimization problem with a regularization functional and loss.
result RSNs generalize well in multi-dimensional tasks, akin to spline regression under certain conditions.
Bayesian method corrects misspecified volatility estimation in high-frequency financial data.
problem Volatility estimation in financial data with infinite jump activity and microstructure noise.
method Proposes a misspecified posterior corrected by a simple estimate of the location shift and re-scaling of the log likelihood.
result Establishes a Bernstein-von Mises theorem for the adjusted posterior, showing asymptotic Gaussianity and consistent estimation.
Active learning selects optimal measurement times for inferring continuous paths from sparse data.
problem Inferring continuous probability paths from sparse snapshots in high-fidelity domains like single-cell biology.
method Extends active experimentation to the space of measures using Linearized Optimal Transport (LOT) for probabilistic surrogate modeling.
result Empirical results show that the proposed strategy outperforms uncertainty-agnostic baselines.
This paper analyzes convergence rates of neural networks in the deep learning regime.
problem Understanding convergence rates of neural networks in the deep learning regime.
method Analyzing the Neural Tangent Kernel (NTK) convergence rates in the large depth limit.
result Quantifies the impact of initialization and activation function on NTK convergence rates.
Theoretical limits of deep residual networks show consistent covariance structures.
problem Understanding the limits of deep residual networks.
method Analyzing the behavior of deep residual networks with skip connections as width and depth approach infinity.
result Theoretical analysis confirms that the covariance structure remains consistent regardless of the order of width and depth.
Observing prices of European put and call options, we calibrate exponential Lévy models nonparametrically. We discuss the efficient implementation of the spectral estimation procedures for Lévy models of finite jump activity as well as for self-decomposable Lévy models. Based on finite sample variances, confidence inte…
Universal approximation theorem for differentiable maps on infinite-dimensional manifolds
problem Approximation of differentiable maps on infinite-dimensional manifolds
method Weighted universal approximation theorem
result Universal approximation theorem for differentiable maps
Study deep maxout networks and their equivalence to Gaussian processes.
problem Understanding neural networks with infinite width.
method Derive equivalence between deep maxout networks and Gaussian processes, characterize maxout kernel, and provide efficient numerical implementation.
result Bayesian inference based on deep maxout network kernel leads to competitive results compared to finite-width counterparts and deep neural network kernels.
Study of deep linear neural networks with proportional width and depth.
problem Lack of descriptive power in Gaussian limit of deep linear neural networks.
method Proportional infinite-width infinite-depth limit for deep linear neural networks.
result Characterization of limiting distribution as a nontrivial mixture of Gaussians.
For any strictly positive martingale S=exp(X) for which X has a characteristic function, we provide an expansion for the implied volatility. This expansion is explicit in the sense that it involves no integrals, but only polynomials in the log strike. We illustrate the versatility of our expansion by computing t…
New method speeds up neural kernel computations for various activations.
problem Inefficient computation of neural kernels for general activations.
method Fast sketching method using truncated Hermite expansion.
result 106x speedup for approximate CNTK computation on CIFAR-10.
Generalizes neural network approximation to infinite-dimensional manifolds and derivatives.
problem Approximating differentiable maps on infinite-dimensional manifolds.
method Proves a weighted Nachbin theorem to establish universal approximation for differentiable maps, including derivatives.
result Linear functions of the signature can approximate path space functionals including their derivatives.
This study uses neural networks to solve interpolation problems with sparse, infinitely wide layers.
problem Exact data interpolation using sparse, infinitely wide neural networks.
method Atomic norm framework to derive convex hulls and equivalent convex formulations.
result Simple characterizations of convex hulls for different constraints on network weights and biases.
Paper explores activity recognition and prediction in real homes using sensor data and video.
problem Improving accuracy of activity recognition and prediction in real home environments.
method Binary sensor data, depth video data, field trial, probabilistic methods, LSTM networks, transfer learning, IIR filter.
result Achieved good accuracy in predicting next sensor event and its mean time of occurrence using LSTM model.
New reservoir computing approach handles infinite-dimensional systems.
problem Approximating and generalizing complex input/output systems.
method Randomly generated echo state networks with neural networks.
result Proves universal approximation properties for new class of systems.
The paper analyzes why ReLU and related functions are effective in neural networks.
problem Understanding why certain activation functions are effective in neural networks.
method Using spline theory, the paper provides theoretical characterizations of activation functions.
result The paper explains the importance of activation functions and related strategies in neural network design.
Two-layer neural networks can approximate functions with fractal singularities.
problem Characterizing functions that can be represented by infinitely wide two-layer neural networks.
method Representation formulas and pointwise properties analysis.
result Functions with fractal or curved singularities cannot be represented by two-layer networks with finite path-norm.
AI-assisted framework detects and predicts rare extreme events.
problem Detecting rare extreme events in complex systems.
method Combining BED with DNOs for active learning and forecasting.
result Framework outperforms GPs and uncovers extremes without initial data.
This paper opens the series of articles supplemental to the series (hep-th/9405050,q-alg/9610026,q-alg/9611003,q-alg/9611019,funct-an/9611003), which also lies in lines of general ideology exposed in the review (mp_arc/96-477). The main purpose of the activity, which has its origin and motivation presumably in the auth…