arXiv research

The study sets limits on how well nonlinear models can generalize from training data.

SSMs combined with neural networks match Transformers in dynamic token selection.

Three-layer networks learn complex hierarchical polynomials of multiple nonlinear features.

Kernel methods are powerful tools to capture nonlinear patterns behind data. They implicitly learn high (even infinite) dimensional nonlinear features in the Reproducing Kernel Hilbert Space (RKHS) while making the computation tractable by leveraging the kernel trick. Classic kernel methods learn a single layer of nonl…

Random Transformers behave like polynomial models in ICL with asymptotic growth.

The approximation of nonlinear kernels via linear feature maps has recently gained interest due to their applications in reducing the training and testing time of kernel-based learning algorithms. Current random projection methods avoid the curse of dimensionality by embedding the nonlinear feature space into a low dim…

Theory explains deep nonlinear networks' plateaus and transitions.

Deep nonlinear models pose a challenge for fitting parameters due to lack of knowledge of the hidden layer and the potentially non-affine relation of the initial and observed layers. In the present work we investigate the use of information theoretic measures such as mutual information and Kullback-Leibler (KL) diverge…

DiffSlack learns neural networks with nonlinear constraints via learnable slack variables.

The paper studies how neural networks evolve representations, finding a unique fixed point for nonlinear activations.

CGNNs use wavelets for continuous function generation in infinite-dimensional spaces.

Three-layer networks learn more complex features than two-layer networks.

The paper proves neural networks with ReLU and softmax can approximate any function.

FPGA-based multi-layer equalizer adapts to changing channels.

Theory explains how deep nets learn features from data.

Reduces nonlinear electromechanical dynamics through quasi-steady state hypothesis.

Deep equilibrium models converge globally without explicit computation.

New analysis explains pathology of deep Gaussian processes.

For a long time, designing neural architectures that exhibit high performance was considered a dark art that required expert hand-tuning. One of the few well-known guidelines for architecture design is the avoidance of exploding gradients, though even this guideline has remained relatively vague and circumstantial. We …

Speeds up deep neural networks training by 10x using GPU concurrency.

The paper designs neural networks with assurance for controlling nonlinear systems.

FsNet selects features for high-dimensional biological data efficiently.

Paper analyzes Transformer learning dynamics, proving benign landscape for in-context learning.

A vast majority of computation in the brain is performed by spiking neural networks. Despite the ubiquity of such spiking, we currently lack an understanding of how biological spiking neural circuits learn and compute in-vivo, as well as how we can instantiate such capabilities in artificial spiking circuits in-silico.…

Unified neural network for linear and nonlinear dimension reduction.

Develops AMITE for analyzing neural network nonlinearities.

Multilayer bootstrap network builds a gradually narrowed multilayer nonlinear network from bottom up for unsupervised nonlinear dimensionality reduction. Each layer of the network is a nonparametric density estimator. It consists of a group of k-centroids clusterings. Each clustering randomly selects data points with r…

RFMs transition from linear to nonlinear under specific input-label correlation.

We investigate the equation $(-Δ_{\mathbb H^n})^γ w=f(w)\quad in \mathbb H^{n},$ where $(-Δ_{\mathbb H^n})^γ$ corresponds to the fractional Laplacian on hyperbolic space for $γ\in (0,1)$ and $f$ is a smooth nonlinearity that typically comes from a double well potential. We prove the existence of heteroclinic connecti…

Deep learning searches for nonlinear factors for predicting asset returns. Predictability is achieved via multiple layers of composite factors as opposed to additive ones. Viewed in this way, asset pricing studies can be revisited using multi-layer deep learners, such as rectified linear units (ReLU) or long-short-term…

Interpretability has become an important issue in the machine learning field, along with the success of layered neural networks in various practical tasks. Since a trained layered neural network consists of a complex nonlinear relationship between large number of parameters, we failed to understand how they could achie…

We propose a numerical method for solving high dimensional fully nonlinear partial differential equations (PDEs). Our algorithm estimates simultaneously by backward time induction the solution and its gradient by multi-layer neural networks, while the Hessian is approximated by automatic differentiation of the gradient…

Many real-world complex systems across natural, social, and economical domains consist of manifold layers to form multiplex networks. The multiple network layers give rise to nonlinear effect for the emergent dynamics of systems. Especially, weak layers that can potentially play significant role in amplifying the vulne…

Deep-RLS uses deep learning to improve PCA for better source separation.

Deep learning is a hierarchical inference method formed by subsequent multiple layers of learning able to more efficiently describe complex relationships. In this work, Deep Gaussian Mixture Models are introduced and discussed. A Deep Gaussian Mixture model (DGMM) is a network of multiple layers of latent variables, wh…

We give a polynomial-time algorithm for learning neural networks with one layer of sigmoids feeding into any Lipschitz, monotone activation function (e.g., sigmoid or ReLU). We make no assumptions on the structure of the network, and the algorithm succeeds with respect to {\em any} distribution on the unit ball in $n$ …

We investigate deep Bayesian neural networks with Gaussian weight priors and a class of ReLU-like nonlinearities. Bayesian neural networks with Gaussian priors are well known to induce an L2, "weight decay", regularization. Our results characterize a more intricate regularization effect at the level of the unit activat…

We study the geometry of deep (neural) networks (DNs) with piecewise affine and convex nonlinearities. The layers of such DNs have been shown to be {\em max-affine spline operators} (MASOs) that partition their input space and apply a region-dependent affine mapping to their input to produce their output. We demonstrat…

The theoretical explanation for deep neural network (DNN) is still an open problem. In this paper DNN is considered as a discrete-time dynamical system due to its layered structure. The complexity provided by the nonlinearity in the dynamics is analyzed in terms of topological entropy and chaos characterized by Lyapuno…

We address the structure identification and the uniform approximation of two fully nonlinear layer neural networks of the type $f(x)=1^T h(B^T g(A^T x))$ on $\mathbb R^d$ from a small number of query samples. We approach the problem by sampling actively finite difference approximations to Hessians of the network. Gathe…

Based on conservation laws for surface layer integrals for critical points of causal variational principles, it is shown how jet spaces can be endowed with an almost-complex structure. We analyze under which conditions the almost-complex structure can be integrated to a canonical complex structure. Combined with the sc…

Finslerian graph neural networks recover nonlinear diffusion geometry

Tensor decomposition methods are widely used for model compression and fast inference in convolutional neural networks (CNNs). Although many decompositions are conceivable, only CP decomposition and a few others have been applied in practice, and no extensive comparisons have been made between available methods. Previo…

Token-adaptive FFN design improves LLM expressivity.

Transformers with MLP heads outperform linear baselines in in-context learning.

Gradient descent converges to a global minimum in nonlinear ReLU implicit networks with linear width.

New NN design for nonlinear systems control with guarantees.

TTT improves model adaptation to test data, especially for nonlinear models.