The study shows removing fully connected output layers improves efficiency without sacrificing performance.
problem Large number of parameters in fully connected layers for high-category datasets.
method Examined architectures replacing fully connected output layers with fixed layers and compared performance.
result Fixed classifiers offer no additional benefit over removing the output layer and its parameters.
FCC-GAN combines fully connected and convolutional layers for improved GAN performance.
problem Lack of understanding in choosing GAN network architectures.
method Proposes FCC-GAN, a hybrid architecture combining fully connected and convolutional layers.
result FCC-GAN outperforms traditional GAN architectures in terms of learning speed and sample quality.
This article demonstrates that convolutional operation can be converted to matrix multiplication, which has the same calculation way with fully connected layer. The article is helpful for the beginners of the neural network to understand how fully connected layer and the convolutional layer work in the backend. To be c…
Stanza separates convolutional and fully connected layers for faster deep learning training.
problem Heavy data transfer between workers and servers in distributed deep learning.
method Layer separation: most nodes train convolutional layers, others train fully connected layers only.
result Significant acceleration of training time (1.34x--13.9x) over current systems.
A network architecture improves classification performance on various datasets.
problem Improving classification performance on diverse datasets.
method Divide fully connected layer into three levels, learn existing layers, cluster similar classes, reclassify using clustering masks.
result Achieved state-of-the-art performance with an error rate of 11.56% on Cifar-100.
Proposes a low-rank deep CNN for multi-task learning.
problem Multi-task learning with deep neural networks.
method Low-rank deep network with nuclear norm and sparsity penalties.
result Improves performance on multiple tasks compared to standard models.
Deep learning improves sparse representation for better classification.
problem Improving classification accuracy using sparse representation.
method A transductive deep learning network combining convolutional autoencoder and fully-connected layers.
result The proposed network achieves better classification results than state-of-the-art SRC methods.
There has been significant interest in the use of fully-connected graphical models and deep-structured graphical models for the purpose of structured inference. However, fully-connected and deep-structured graphical models have been largely explored independently, leaving the unification of these two concepts ripe for …
New graph CNN layers improve accuracy on graph datasets.
problem Graph data relations are better represented as graphs, not grids.
method Proposed new graph CNN layers for vertex and edge features.
result Improved classification accuracy on graph datasets.
Recently popularized graph neural networks achieve the state-of-the-art accuracy on a number of standard benchmark datasets for graph-based semi-supervised learning, improving significantly over existing approaches. These architectures alternate between a propagation layer that aggregates the hidden states of the local…
The fully connected layers of a deep convolutional neural network typically contain over 90% of the network parameters, and consume the majority of the memory required to store the network parameters. Reducing the number of parameters while preserving essentially the same predictive performance is critically important …
RIFLE improves deep transfer learning by reinitializing fully-connected layers.
problem Limited improvement in transfer learning accuracy with pre-trained models on small datasets.
method Re-Initializing fully-connected layers with random scratch during fine-tuning.
result Significant improvement in deep transfer learning accuracy across various datasets.
Trained MLPs' weights are exchangeable, leading to stable kernel behavior.
problem Assumptions of IID parameters in trained models are violated.
method Showed weights in MLPs are exchangeable and identified kernel stability.
result Layer-wise kernel of fully-connected layers remains approximately constant during training.
Proves DCNNs with expansive convolution are strongly universally consistent.
problem Theoretical consistency of deep convolutional neural networks (DCNNs).
method Empirical risk minimization on DCNNs with expansive convolution (with zero-padding).
result DCNNs with expansive convolution are strongly universally consistent.
New GM layers improve neural network performance.
problem Improving neural network performance.
method Employing Gaussian mixture models and Wasserstein gradient flows.
result GM layers achieve comparable performance to two-layer networks.
Deep Reinforcement Learning (RL) recently emerged as one of the most competitive approaches for learning in sequential decision making problems with fully observable environments, e.g., computer Go. However, very little work has been done in deep RL to handle partially observable environments. We propose a new architec…
Tensor regression networks achieve high compression rate of neural networks while having slight impact on performances. They do so by imposing low tensor rank structure on the weight matrices of fully connected layers. In recent years, tensor regression networks have been investigated from the perspective of their comp…
Recent DNN pruning algorithms have succeeded in reducing the number of parameters in fully connected layers, often with little or no drop in classification accuracy. However, most of the existing pruning schemes either have to be applied during training or require a costly retraining procedure after pruning to regain c…
Recently the generalization error of deep neural networks has been analyzed through the PAC-Bayesian framework, for the case of fully connected layers. We adapt this approach to the convolutional setting.
Deep neural networks can handle high dimensions without losing accuracy.
problem The curse of dimensionality in nonparametric regression.
method Least squares estimates on fully connected neural networks with ReLU activation functions.
result Similar convergence results to deep learning models with sparsity constraints can be achieved without such constraints.
The paper calculates bounds on the local Lipschitz constants of neural network layers.
problem Understanding the Lipschitz constants of neural network layers for robustness analysis.
method Analytical approach to determine upper bounds on local Lipschitz constants of affine-ReLU functions.
result The method produces tighter bounds than the standard conservative bound, especially for small perturbations.
Transformers excel at sparse token selection, surpassing FCNs in both worst and average cases.
problem Sparse token selection task
method One-layer transformer trained with gradient descent
result Transformers learn sparse token selection and exhibit strong out-of-distribution length generalization
A simple 2-layer linear network outperforms neural networks in learning sparse targets.
problem Learning sparse targets from a sparse input with gradient descent.
method A 2-layer linear network with fully connected input layer and sparse targets.
result The 2-layer linear network achieves a lower expected square loss than neural networks.
We propose and evaluate new techniques for compressing and speeding up dense matrix multiplications as found in the fully connected and recurrent layers of neural networks for embedded large vocabulary continuous speech recognition (LVCSR). For compression, we introduce and study a trace norm regularization technique f…
A deep learning network was used to predict future blood glucose levels, as this can permit diabetes patients to take action before imminent hyperglycaemia and hypoglycaemia. A sequential model with one long-short-term memory (LSTM) layer, one bidirectional LSTM layer and several fully connected layers was used to pred…
Deep neural networks converge to Gaussian mixtures as layer width increases.
problem Understanding the distribution of outputs from deep neural networks.
method Proof and experiments with a simple model showing the convergence of neural network outputs to Gaussian mixtures.
result Neural networks converge to Gaussian mixtures as the width of the last hidden layer increases.
This paper extends ResNet theory to infinitely deep networks, linking them to diffusion processes.
problem Training infinitely deep ResNets with i.i.d. initializations leads to undesirable properties.
method Introduced doubly infinite ResNets with i.i.d. initializations, linking to diffusion processes.
result The dynamics of quantities of interest converge to deterministic limits in the limit of infinite depth.
We improve neural network explainability by bypassing batch normalization.
problem Lack of transparency in neural networks.
method Layer-wise Relevance Propagation with a method to include normalization layers.
result Heatmaps are more accurate for convolutional layers with our method.
Study on memorization vs. generalization in overparameterized networks.
problem Understanding the trade-off between memorization and generalization in neural networks.
method Examined fully-connected and convolutional networks trained to minimize reconstruction error.
result Different architectures exhibit distinct inductive biases, affecting generalization from a single training example.
This paper presents two unsupervised learning layers (UL layers) for label-free video analysis: one for fully connected layers, and the other for convolutional ones. The proposed UL layers can play two roles: they can be the cost function layer for providing global training signal; meanwhile they can be added to any re…
We demonstrate that a very deep ResNet with stacked modules with one neuron per hidden layer and ReLU activation functions can uniformly approximate any Lebesgue integrable function in d dimensions, i.e. ℓ1(Rd). Because of the identity mapping inherent to ResNets, our network has alternating layers of…
Structured linear substitutions improve both efficiency and accuracy in neural networks.
problem Improving neural network efficiency and accuracy tradeoff.
method Replacing linear components in pointwise convolutions with structured linear decompositions.
result Structured layers provide Pareto-optimal benefits in efficiency/accuracy.
Paper introduces DropFilter and DropFilter-PLUS for CNN regularization.
problem Overfitting in Convolutional Neural Networks (CNNs).
method Randomly modifies convolution filters in CNNs.
result Improves performance on image classification tasks.
We propose a new notion of `non-linearity' of a network layer with respect to an input batch that is based on its proximity to a linear system, which is reflected in the non-negative rank of the activation matrix. We measure this non-linearity by applying non-negative factorization to the activation matrix. Considering…
Proposes a new deep learning framework for financial stock trading.
problem Lack of effective techniques to fuse multi-channel financial time-series data.
method Inspired by convolution transform learning, SDCF processes channels through 1-D convolutions, fuses outputs with fully-connected layers, and applies softmax classification.
result Proposed framework yields better results than state-of-the-art techniques for stock trading.
Bounds neural network output distribution to Gaussian for random initialization.
problem Quantifying the distribution of randomly initialized deep neural networks.
method Quantitative Gaussian approximation using quadratic Wasserstein distance.
result Explicit inequalities show how network sizes affect Gaussian behavior.
The paper sets information-theoretic lower bounds for neural networks' parameter recovery and excess risk.
problem Establishing sample complexity lower bounds for neural network parameters and excess risk.
method Using information-theoretic tools, the paper proves lower bounds by constructing a generative network.
result Proves information-theoretic lower bounds for exact parameter recovery and positive excess risk.
New method diagnoses criticality in deep neural networks, improving performance.
problem Improving theoretical understanding and practical initialization of deep neural networks.
method Introducing partial Jacobians and deriving recurrence relations for their norms to analyze criticality.
result Proper stacking of LayerNorm and residual connections leads to a critical architecture for any initialization.
In our previous work we have shown that resistive cross point devices, so called Resistive Processing Unit (RPU) devices, can provide significant power and speed benefits when training deep fully connected networks as well as convolutional neural networks. In this work, we further extend the RPU concept for training re…
New method speeds up deep neural networks inference.
problem Inference speed of deep neural networks.
method Maximum volume algorithm for reduced-order modeling.
result Convolutional layers can be replaced with smaller fully-connected layers with minimal accuracy loss.
A new method compresses NLP networks by using multiple subspaces instead of a single one.
problem Large errors in compressing NLP networks using a single subspace.
method Projective clustering to find multiple subspaces that minimize squared distances.
result Networks are more accurate and smaller compared to standard matrix factorization.
DropCluster clusters features in convolutional layers to prevent overfitting.
problem Dropout's effectiveness in convolutional layers is limited due to feature spatial correlation.
method DropCluster clusters features in convolutional layers and randomly drops clusters during training.
result DropCluster controls overfitting better than other approaches on various datasets.
Deep neural networks are powerful learning models that achieve state-of-the-art performance on many computer vision, speech, and language processing tasks. In this paper, we study a fundamental question that arises when designing deep network architectures: Given a target network architecture can we design a smaller ne…
This paper proposes structurally sparse RNNs to reduce computational and memory costs.
problem Heavy computational and memory burden in fully connected RNNs.
method Study structurally sparse RNNs, reducing recurrent operations and weights.
result Structurally sparse RNNs achieve competitive performance with reduced costs.
IRNet improves material property prediction from composition and crystal structure.
problem Predicting material properties from composition and crystal structure.
method Deep residual regression network with individual residual learning.
result IRNet outperforms state-of-the-art machine learning approaches in predicting material properties.
The paper addresses differentially private learning for neural networks, focusing on risk bounds and algorithm feasibility.
problem Achieving differentially private learning for neural networks with theoretical guarantees.
method Developed algorithms and theoretical analysis for differentially private stochastic optimization of neural networks.
result Established theoretical bounds for excess population risk in differentially private learning of neural networks.
Enhances neural networks in hyperbolic space for better data structure capture.
problem Capturing hierarchical data structures efficiently.
method Unified hyperbolic model for neural network components.
result Superior parameter efficiency and outperformance over Euclidean methods.
Paper explores offline and online image recognition using neural networks.
problem Challenges in evolving image recognition through different settings.
method Used Convolutional Neural Networks and Multi-layer Perceptrons.
result Encouraging preliminary results in offline and online image classification.