Convolutional networks outperform fully-connected ones in certain tasks.
problem Understanding the computational advantage of convolutional networks over fully-connected networks.
method Demonstrated a computational advantage through a specific problem class.
result Convolutional networks can solve certain problems that fully-connected networks cannot, even with gradient descent.
The study shows removing fully connected output layers improves efficiency without sacrificing performance.
problem Large number of parameters in fully connected layers for high-category datasets.
method Examined architectures replacing fully connected output layers with fixed layers and compared performance.
result Fixed classifiers offer no additional benefit over removing the output layer and its parameters.
Convolutional layers can be mathematically equated to fully connected layers.
problem Understanding the equivalence between convolutional and fully connected layers for neural networks.
method Demonstrated that convolutional operations can be converted to matrix multiplication, showing equivalence.
result Convolutional layers and fully connected layers are mathematically equivalent in linear cases.
This paper improves deep neural network approximation for fully connected networks, achieving optimal convergence rates.
problem Improving approximation of fully connected deep neural networks for optimal convergence rates.
method Deriving approximation bounds specifically for a narrower fully connected deep neural network.
result Achieves an optimal rate (up to a logarithmic factor) for fully connected deep neural networks.
FCC-GAN combines fully connected and convolutional layers for improved GAN performance.
problem Lack of understanding in choosing GAN network architectures.
method Proposes FCC-GAN, a hybrid architecture combining fully connected and convolutional layers.
result FCC-GAN outperforms traditional GAN architectures in terms of learning speed and sample quality.
Convolutional nets require fewer samples than fully-connected nets for image classification.
problem Understanding why convolutional nets are more sample-efficient than fully-connected nets.
method Construction of a natural distribution and target function to demonstrate a sample complexity gap.
result Convolutional nets require O(1) samples for a single target function, while fully-connected nets require Ω(d2) samples. There has been significant interest in the use of fully-connected graphical models and deep-structured graphical models for the purpose of structured inference. However, fully-connected and deep-structured graphical models have been largely explored independently, leaving the unification of these two concepts ripe for …
Efficiently infers fully-connected CRFs using low-rank approximation.
problem Inference on fully-connected CRFs is challenging due to long-range contextual relationships.
method Developed an efficient algorithm based on SDP and low-rank approximation of the similarity matrix.
result Demonstrated that the method can solve CRFs that were previously unsolvable.
PAC-Bayesian bounds show fully connected DNNs with Gaussian priors match minimax rates.
problem Theoretical limits of fully connected deep neural networks with Gaussian priors.
method PAC-Bayesian bounds for fully connected Bayesian DNNs with Gaussian priors.
result PAC-Bayesian bounds match minimax-optimal rates in Besov space for nonparametric regression and binary classification.
Stanza separates convolutional and fully connected layers for faster deep learning training.
problem Heavy data transfer between workers and servers in distributed deep learning.
method Layer separation: most nodes train convolutional layers, others train fully connected layers only.
result Significant acceleration of training time (1.34x--13.9x) over current systems.
FC-GAGA forecasts traffic using a novel gating mechanism.
problem Forecasting multivariate time-series, especially with graph relationships.
method Learnable fully connected hard graph gating mechanism for fully connected time-series forecasting.
result Competitive or better performance than existing algorithms without graph knowledge.
A network architecture improves classification performance on various datasets.
problem Improving classification performance on diverse datasets.
method Divide fully connected layer into three levels, learn existing layers, cluster similar classes, reclassify using clustering masks.
result Achieved state-of-the-art performance with an error rate of 11.56% on Cifar-100.
Deep learning improves sparse representation for better classification.
problem Improving classification accuracy using sparse representation.
method A transductive deep learning network combining convolutional autoencoder and fully-connected layers.
result The proposed network achieves better classification results than state-of-the-art SRC methods.
Transformers excel at sparse token selection, surpassing FCNs in both worst and average cases.
problem Sparse token selection task
method One-layer transformer trained with gradient descent
result Transformers learn sparse token selection and exhibit strong out-of-distribution length generalization
Proposes a low-rank deep CNN for multi-task learning.
problem Multi-task learning with deep neural networks.
method Low-rank deep network with nuclear norm and sparsity penalties.
result Improves performance on multiple tasks compared to standard models.
This paper explains spectral clustering and its equivalence to PCA, breaking it into fully connected and multi-connected cases.
problem Understanding the mathematics behind spectral clustering and its equivalence to PCA.
method Dividing spectral clustering into two categories based on graph connectivity and proving the equivalence to PCA.
result Spectral clustering and PCA are equivalent, with specific proofs for fully connected and multi-connected graphs.
This work uses decision trees to encode relevant features and their interactions into neural networks, improving model performance.
problem Overfitting in neural networks with many irrelevant variables.
method Defines a mapping to encode decision tree extracted relationships into a neural network.
result The approach outperforms fully connected neural networks and tree-based methods.
This paper proposes structurally sparse RNNs to reduce computational and memory costs.
problem Heavy computational and memory burden in fully connected RNNs.
method Study structurally sparse RNNs, reducing recurrent operations and weights.
result Structurally sparse RNNs achieve competitive performance with reduced costs.
New communication topologies improve deep reinforcement learning efficiency.
problem Optimizing communication topology for faster and more robust learning in deep reinforcement learning.
method Introduced alternative network topologies (Erdos-Renyi random graphs) and compared their performance with fully-connected and star topologies.
result Erdos-Renyi random graphs outperform fully-connected networks in deep reinforcement learning tasks.
We study alignment in linear neural networks and its relation to gradient descent.
problem Understanding alignment in linear neural networks and its impact on training.
method Defined alignment for fully connected networks, analyzed alignment under gradient descent, and compared gradient descent to projected gradient descent for layer-constrained networks.
result Gradient descent can converge linearly to a global minimum when alignment is invariant, and alignment is impossible with large datasets in layer-constrained networks.
New method learns convolution-like structures from scratch.
problem Learning convolution-like structures from data.
method Minimum description length principle and β-LASSO algorithm. result Learned architectures achieve state-of-the-art accuracies.
USN improves neural networks with uniform sparse connectivity.
problem Overfitting and limited scalability in classical neural networks.
method Uniform sparse network (USN) with even and sparse connectivity.
result USN outperforms state-of-the-art sparse network models in accuracy, speed, and robustness.
Pruning FCNs reveals sub-networks that match CNNs' performance.
problem Understanding the inductive bias of pruning in neural networks.
method Iterative magnitude pruning of a simple FCN followed by analysis of the resulting architecture.
result Pruned FCNs exhibit key features of CNNs, suggesting new architectural biases.
Trained MLPs' weights are exchangeable, leading to stable kernel behavior.
problem Assumptions of IID parameters in trained models are violated.
method Showed weights in MLPs are exchangeable and identified kernel stability.
result Layer-wise kernel of fully-connected layers remains approximately constant during training.
Paper proposes a technique to reduce deep neural network parameters without sacrificing accuracy.
problem Designing smaller networks that approximate the operation of larger, more powerful networks.
method Randomized tensor sketching technique applied to convolutional and fully connected layers.
result Smaller networks trained with sketching technique achieve comparable accuracy to original networks.
Proves DCNNs with expansive convolution are strongly universally consistent.
problem Theoretical consistency of deep convolutional neural networks (DCNNs).
method Empirical risk minimization on DCNNs with expansive convolution (with zero-padding).
result DCNNs with expansive convolution are strongly universally consistent.
Study examines noise sensitivity of DNNs for binary classification.
problem Understanding non-robustness of DNN classifiers under noise.
method Defined and extended noise sensitivity and stability concepts for Boolean functions, applied to DNN models.
result Sorted out the relation between definitions and properties of DNN architectures under noise.
New communication topologies improve deep reinforcement learning performance.
problem Improving performance of learning agents in distributed reinforcement learning.
method Examined four graph families for communication topologies and found Erdos-Renyi random graphs to outperform fully connected topologies.
result Erdos-Renyi random graphs can improve performance of distributed learning agents.
Deep neural networks estimate regression functions on manifolds.
problem Estimating regression functions on manifolds from data.
method Fully connected deep neural networks with ReLU activation, analyzing convergence rates.
result Estimates achieve a rate of convergence dependent on manifold dimension, not predictor dimension.
A new neural network architecture reduces parameters by 94% while maintaining performance.
problem Reduction of trainable parameters in neural networks.
method Spatially-coupled sparse construction to allocate trainable parameters efficiently.
result Performance comparable to traditional neural networks with 94% fewer parameters.
We consider the effects of the global financial crisis through a local Korean financial market around the 2008 crisis. We analyze 185 individual stock prices belonging to the KOSPI (Korea Composite Stock Price Index), cosidering three time periods: the time before, during, and after the crisis. The complex networks gen…
RIFLE improves deep transfer learning by reinitializing fully-connected layers.
problem Limited improvement in transfer learning accuracy with pre-trained models on small datasets.
method Re-Initializing fully-connected layers with random scratch during fine-tuning.
result Significant improvement in deep transfer learning accuracy across various datasets.
Graph neural network simplifies to linear model with attention mechanisms.
problem Semi-supervised learning with limited labeled data.
method Proposes a graph neural network removing fully-connected layers and replacing them with attention mechanisms.
result Attention-based graph neural network outperforms existing methods on benchmark datasets.
The fully connected layers of a deep convolutional neural network typically contain over 90% of the network parameters, and consume the majority of the memory required to store the network parameters. Reducing the number of parameters while preserving essentially the same predictive performance is critically important …
New graph CNN layers improve accuracy on graph datasets.
problem Graph data relations are better represented as graphs, not grids.
method Proposed new graph CNN layers for vertex and edge features.
result Improved classification accuracy on graph datasets.
Improves deep RL for partially observable environments.
problem Handling partially observable environments in deep RL.
method Action-specific Deep Recurrent Q-Network (ADRQN) architecture.
result Demonstrates effectiveness in partially observable domains.
New method shows fully-connected networks can learn convolutional structures from data.
problem How to learn convolutional structures from translation-invariant data.
method Data-driven emergence of convolutional structure in neural networks.
result Initially fully-connected networks can learn convolutional structures from their inputs.
Stable processes emerge as limits of deep neural networks with symmetric stable distributions.
problem Understanding the behavior of deep neural networks as they become infinitely wide.
method Analyzing fully connected feed-forward deep neural networks with symmetric stable distributions and showing the limit as a stable process.
result The infinite wide limit of the network is a stable process with multivariate stable distributions.
Adapts PAC-Bayesian analysis to convolutional neural networks.
problem Generalization error of convolutional neural networks.
method PAC-Bayesian framework applied to convolutional layers.
result Margin bounds for convolutional neural networks.
RPU devices accelerate LSTM training with significant power and speed benefits.
problem Training large-scale LSTM networks efficiently using resistive cross-point devices.
method Extending RPU concept to LSTMs, studying imperfections, and optimizing input signal resolution.
result RPU devices can provide large acceleration factors for RNNs, with 1500 times larger models than fully connected networks.
A fast pruning algorithm for DNNs with GE guarantees.
problem Efficiently pruning DNNs without sacrificing accuracy.
method FeTa algorithm based on DC optimization, with GE analysis.
result FeTa is orders of magnitude faster and maintains GE.
Quantitative CLTs show neural network distributions converge to Gaussian as width increases.
problem Understanding the distribution of fully connected neural networks with random weights and biases.
method Analyzing the distribution of a fully connected neural network with random Gaussian weights and biases, proving quantitative bounds on normal approximations.
result The distance between a random fully connected network and the corresponding infinite width Gaussian process scales like n−γ for γ>0. New techniques for faster and more compact speech recognition models.
problem Efficiency and compactness in speech recognition neural networks.
method Trace norm regularization for low rank factoring and ARM optimized kernels for faster inference.
result 3x to 7x speed up in inference on ARM processors compared to gemmlowp.
Overparameterized autoencoders can memorize training examples.
problem Understanding generalization in overparameterized neural networks.
method Analyzing autoencoders of varying depths and types.
result Autoencoders concentrate learned functions around training examples.
LSTM networks improve top jet tagging at the LHC.
problem Boosted top quark tagging at the LHC.
method Incorporating Long Short-Term Memory (LSTM) networks into jet constituent analysis.
result Best LSTM network achieves 100 background rejection at 50% signal efficiency.
Paper introduces DropFilter and DropFilter-PLUS for CNN regularization.
problem Overfitting in Convolutional Neural Networks (CNNs).
method Randomly modifies convolution filters in CNNs.
result Improves performance on image classification tasks.
The paper sets information-theoretic lower bounds for neural networks' parameter recovery and excess risk.
problem Establishing sample complexity lower bounds for neural network parameters and excess risk.
method Using information-theoretic tools, the paper proves lower bounds by constructing a generative network.
result Proves information-theoretic lower bounds for exact parameter recovery and positive excess risk.
This paper extends ResNet theory to infinitely deep networks, linking them to diffusion processes.
problem Training infinitely deep ResNets with i.i.d. initializations leads to undesirable properties.
method Introduced doubly infinite ResNets with i.i.d. initializations, linking to diffusion processes.
result The dynamics of quantities of interest converge to deterministic limits in the limit of infinite depth.