arXiv research

RWTNs improve NTN performance in SRL tasks.

Tensor programs prove neural network limits for any architecture.

A new method to prune neural networks with iterative randomization improves efficiency.

We present a homogenization theorem for isotropically-distributed point defects, by considering a sequence of manifolds with increasingly dense point defects. The loci of the defects are chosen randomly according to a weighted Poisson point process, making it a continuous version of the first passage percolation model.…

RicciNets prunes neural networks by removing edges of low importance based on Ricci curvature, reducing FLOPs by 35%.

New methods train neural networks without changing weights, achieving similar or higher performance.

We address the structure identification and the uniform approximation of sums of ridge functions $f(x)=\sum_{i=1}^m g_i(a_i\cdot x)$ on ${\mathbb R}^d$, representing a general form of a shallow feed-forward neural network, from a small number of query samples. Higher order differentiation, as used in our constructive a…

Randomly chosen primary hidden units and derived secondary units reduce neural network complexity.

DRE combines DNN with random feature regression for efficient neural network design.

New method finds compact strong lottery tickets in partially frozen networks.

Randomly initialized networks can perform as well as pruned networks.

This paper is concerned with the sparsification of the input-hidden weights of ELM (Extreme Learning Machine). For ordinary feedforward neural networks, the sparsification is usually done by introducing certain regularization technique into the learning process of the network. But this strategy can not be applied for E…

Randomly guessing weights helps analyze RL benchmarks objectively.

Randomly initialized ReLU networks of depth two can approximate smooth functions well.

We obtain the first polynomial-time algorithm for exact tensor completion that improves over the bound implied by reduction to matrix completion. The algorithm recovers an unknown 3-tensor with $r$ incoherent, orthogonal components in $\mathbb R^n$ from $r\cdot \tilde O(n^{1.5})$ randomly observed entries of the tensor…

The lottery ticket hypothesis (Frankle and Carbin, 2018), states that a randomly-initialized network contains a small subnetwork such that, when trained in isolation, can compete with the performance of the original network. We prove an even stronger hypothesis (as was also conjectured in Ramanujan et al., 2019), showi…

Although the convolutional neural networks (CNNs) have become popular for various image processing and computer vision task recently, it remains a challenging problem to reduce the storage cost of the parameters for resource-limited platforms. In the previous studies, tensor decomposition (TD) has achieved promising co…

Optimizes wide low-rank neural networks for reduced parameters and cost.

SPIDER uses deep neural networks for streaming tensor factorization.

Study Gaussian approximation for deep neural networks with random weights.

Before training a neural net, a classic rule of thumb is to randomly initialize the weights so the variance of activations is preserved across layers. This is traditionally interpreted using the total variance due to randomness in both weights \emph{and} samples. Alternatively, one can interpret the rule of thumb as pr…

Low-rank structure emerges in neural networks during learning.

Echo state networks with random weights can approximate any continuous system.

Model for material elasticity and plasticity using networks.

A new tensor ring mixture model improves density estimation efficiency.

An interesting approach to analyzing neural networks that has received renewed attention is to examine the equivalent kernel of the neural network. This is based on the fact that a fully connected feedforward network with one hidden layer, a certain weight distribution, an activation function, and an infinite number of…

The study analyzes XRP transaction networks to understand market dynamics.

Defines and proves properties of weighted renormalized volume coefficients.

Tensor regression networks achieve high compression rate of neural networks while having slight impact on performances. They do so by imposing low tensor rank structure on the weight matrices of fully connected layers. In recent years, tensor regression networks have been investigated from the perspective of their comp…

The training of stochastic neural network models with binary ($\pm1$) weights and activations via continuous surrogate networks is investigated. We derive new surrogates using a novel derivation based on writing the stochastic neural network as a Markov chain. This derivation also encompasses existing variants of the s…

A novel Laplace-approximated Bayesian Tensor Network Kernel Machine (LA-TNKM) provides principled uncertainty estimates.

We establish connections between the problem of learning a two-layer neural network and tensor decomposition. We consider a model with feature vectors $\boldsymbol x \in \mathbb R^d$, $r$ hidden units with weights $\{\boldsymbol w_i\}_{1\le i \le r}$ and output $y\in \mathbb R$, i.e., $y=\sum_{i=1}^r σ( \boldsymbol w_i…

Study Gaussian-process limits of neural networks using tensor programs.

Wide neural networks become linear, but adding bottlenecks makes them bilinear or multilinear.

Tensor decomposition is a well-known tool for multiway data analysis. This work proposes using stochastic gradients for efficient generalized canonical polyadic (GCP) tensor decomposition of large-scale tensors. GCP tensor decomposition is a recently proposed version of tensor decomposition that allows for a variety of…

We introduce a new parameterization method for deep learning layers using spectral tensor train decomposition.

A new RNN architecture reduces model size and improves performance.

Random feature models approximate functions in Banach spaces efficiently.

In this paper, we consider one dimensional (shallow) ReLU neural networks in which weights are chosen randomly and only the terminal layer is trained. First, we mathematically show that for such networks L2-regularized regression corresponds in function space to regularizing the estimate's second derivative for fairly …

Recent years have witnessed the success of deep neural networks in dealing with a plenty of practical problems. Dropout has played an essential role in many successful deep neural networks, by inducing regularization in the model training. In this paper, we present a new regularized training approach: Shakeout. Instead…

We introduce tensor network contraction algorithms for the evaluation of the Jones polynomial of arbitrary knots. The value of the Jones polynomial of a knot maps to the partition function of a $q$-state Potts model defined as a planar graph with weighted edges that corresponds to the knot. For any integer $q$, we cast…

We analyze linear independence of rank one tensors produced by tensor powers of randomly perturbed vectors. This enables efficient decomposition of sums of high-order tensors. Our analysis builds upon [BCMV14] but allows for a wider range of perturbation models, including discrete ones. We give an application to recove…

Method learns topological states from randomized measurements.

In this paper we study the problem of learning the weights of a deep convolutional neural network. We consider a network where convolutions are carried out over non-overlapping patches with a single kernel in each layer. We develop an algorithm for simultaneously learning all the kernels from the training data. Our app…

Tensor-EM method learns MoLDS from complex, noisy data.

Paper reconstructs training data from a single gradient query.

Tensor networks constrain kernel machines to Gaussian processes.

This paper finds a new way to compress CNN weights, improving on pruning and quantization.