arXiv research

A framework for transformer attention layers derived from SVR.

New CNN initialization scheme derived from modern architectures.

Derives new orthogonal coordinates for evolving surfaces and curves.

A fundamental question in deep learning concerns the role played by individual layers in a deep neural network (DNN) and the transferable properties of the data representations which they learn. To the extent that layers have clear roles, one should be able to optimize them separately using layer-wise loss functions. S…

Introduces Causal Energy Minimization to understand Transformer layers.

Law derived for neural networks with sparse connections.

A neural network derived from first principles using MaxEnt.

It is challenging to develop stochastic gradient based scalable inference for deep discrete latent variable models (LVMs), due to the difficulties in not only computing the gradients, but also adapting the step sizes to different latent factors and hidden layers. For the Poisson gamma belief network (PGBN), a recently …

New deep network derived from rate reduction principles, explaining features and efficiency.

We propose the Limited Multi-Label (LML) projection layer as a new primitive operation for end-to-end learning systems. The LML layer provides a probabilistic way of modeling multi-label predictions limited to having exactly k labels. We derive efficient forward and backward passes for this layer and show how the layer…

In recent years there has been an increased interest in statistical analysis of data with multiple types of relations among a set of entities. Such multi-relational data can be represented as multi-layer graphs where the set of vertices represents the entities and multiple types of edges represent the different relatio…

The study sets limits on how well nonlinear models can generalize from training data.

A method to analyze neural network performance by measuring layer saturation.

In this paper, we consider one dimensional (shallow) ReLU neural networks in which weights are chosen randomly and only the terminal layer is trained. First, we mathematically show that for such networks L2-regularized regression corresponds in function space to regularizing the estimate's second derivative for fairly …

The paper generalizes equivariant neural networks on homogeneous spaces to the non-linear setting.

Multi-layered representation is believed to be the key ingredient of deep neural networks especially in cognitive tasks like computer vision. While non-differentiable models such as gradient boosting decision trees (GBDTs) are the dominant methods for modeling discrete or tabular data, they are hard to incorporate with…

Study on ion travel time on curved surfaces.

A neural network with a single hidden layer can't represent certain multivariable functions.

Proposes BN layers for neural networks on complex domains, improving training stability and accuracy.

CLAPS improves conformal regression by adaptively scaling interval widths based on last-layer Laplace uncertainty.

Whereas CNNs have demonstrated immense progress in many vision problems, they suffer from a dependence on monumental amounts of labeled training data. On the other hand, dictionary learning does not scale to the size of problems that CNNs can handle, despite being very effective at low-level vision tasks such as denois…

This work analyzes how bottleneck layers and skip connections affect linear denoising autoencoders' generalization.

Two-layer neural networks learn efficiently using kernel methods in mean-field analysis.

New GM layers improve neural network performance.

New bounds for neural networks without loss boundedness assumption.

EigenGAN discovers interpretable dimensions in GAN layers for semantic control.

Derives formulae for general permutation equivariant layers and presents a second order graph variational encoder.

Deep Convolutional Neural Networks (CNN) enforces supervised information only at the output layer, and hidden layers are trained by back propagating the prediction error from the output layer without explicit supervision. We propose a supervised feature learning approach, Label Consistent Neural Network, which enforces…

This paper explains double descent in linear neural networks, identifying new factors.

New method diagnoses criticality in deep neural networks, improving performance.

We introduce a guide to help deep learning practitioners understand and manipulate convolutional neural network architectures. The guide clarifies the relationship between various properties (input shape, kernel shape, zero padding, strides and output shape) of convolutional, pooling and transposed convolutional layers…

Channel Pruning, widely used for accelerating Convolutional Neural Networks, is an NP-hard problem due to the inter-layer dependency of channel redundancy. Existing methods generally ignored the above dependency for computation simplicity. To solve the problem, under the Bayesian framework, we here propose a layer-wise…

Gradient descent converges linearly for overparameterized linear networks.

Improved recommendation systems using multi-layer embeddings reduce model size while maintaining accuracy.

AIM models explain deep learning in attention layers, offering solvable insights.

Multiscale stochastic volatility models have been developed as an efficient way to capture the principle effects on derivative pricing and portfolio optimization of randomly varying volatility. The recent book Fouque, Papanicolaou, Sircar and Sølna (2011, CUP) analyzes models in which the volatility of the underlying i…

We consider the problem of reconstructing a signal from multi-layered (possibly) non-linear measurements. Using non-rigorous but standard methods from statistical physics we present the Multi-Layer Approximate Message Passing (ML-AMP) algorithm for computing marginal probabilities of the corresponding estimation proble…

Paper perfect clusters sparse, diverse multilayer networks.

This paper extends stability and generalization analysis of GD for multi-layer NNs.

Various forms of representations may arise in the many layers embedded in deep neural networks (DNNs). Of these, where can we find the most compact representation? We propose to use a pruning framework to answer this question: How compact can each layer be compressed, without losing performance? Most of the existing DN…

Paper proposes an efficient method to optimize neural networks without backpropagation.

A Gaussian restricted Boltzmann machine (GRBM) is a Boltzmann machine defined on a bipartite graph and is an extension of usual restricted Boltzmann machines. A GRBM consists of two different layers: a visible layer composed of continuous visible variables and a hidden layer composed of discrete hidden variables. In th…

An important issue in neural network research is how to choose the number of nodes and layers such as to solve a classification problem. We provide new intuitions based on earlier results by An et al. (2015) by deriving an upper bound on the number of nodes in networks with two hidden layers such that linear separabili…

Upper bounds on fixed points in PWL neural networks with hyperplane analysis.

Study on VC dimension of GCNNs with input resolution effects.

Derives asymptotic generalization error for large-margin classifiers.

Derives equations for deep learning biases and weights, showing data complexity reduction.

Designs a neural network to reduce training cost by mapping to higher dimensions.