arXiv research

Convolutional networks outperform fully-connected ones in certain tasks.

The study shows removing fully connected output layers improves efficiency without sacrificing performance.

This paper improves deep neural network approximation for fully connected networks, achieving optimal convergence rates.

Convolutional nets require fewer samples than fully-connected nets for image classification.

There has been significant interest in the use of fully-connected graphical models and deep-structured graphical models for the purpose of structured inference. However, fully-connected and deep-structured graphical models have been largely explored independently, leaving the unification of these two concepts ripe for …

PAC-Bayesian bounds show fully connected DNNs with Gaussian priors match minimax rates.

Generative Adversarial Networks (GANs) are a powerful class of generative models. Despite their successes, the most appropriate choice of a GAN network architecture is still not well understood. GAN models for image synthesis have adopted a deep convolutional network architecture, which eliminates or minimizes the use …

This article demonstrates that convolutional operation can be converted to matrix multiplication, which has the same calculation way with fully connected layer. The article is helpful for the beginners of the neural network to understand how fully connected layer and the convolutional layer work in the backend. To be c…

Recurrent neural networks (RNNs) are omnipresent in sequence modeling tasks. Practical models usually consist of several layers of hundreds or thousands of neurons which are fully connected. This places a heavy computational and memory burden on hardware, restricting adoption in practical low-cost and low-power devices…

The paper connects link symmetries to finite subgroups of O(3).

This paper explains spectral clustering and its equivalence to PCA, breaking it into fully connected and multi-connected cases.

Conditional Random Fields (CRF) have been widely used in a variety of computer vision tasks. Conventional CRFs typically define edges on neighboring image pixels, resulting in a sparse graph such that efficient inference can be performed. However, these CRFs fail to model long-range contextual relationships. Fully-conn…

FC-GAGA forecasts traffic using a novel gating mechanism.

H. Masur and J. Smillie proved precisely which singularity index lists arise from pseudo-Anosov mapping classes. In search of an analogous theorem for outer automorphisms of free groups, Handel and Mosher ask: Is each connected, simplicial, (2r-1)-vertex graph the ideal Whitehead graph of a fully irreducible outer auto…

In this paper, we propose a novel multi-task learning method based on the deep convolutional network. The proposed deep network has four convolutional layers, three max-pooling layers, and two parallel fully connected layers. To adjust the deep network to multi-task learning problem, we propose to learn a low-rank deep…

USN improves neural networks with uniform sparse connectivity.

Transformers excel at sparse token selection, surpassing FCNs in both worst and average cases.

Artificial Neural Networks form the basis of very powerful learning methods. It has been observed that a naive application of fully connected neural networks to data with many irrelevant variables often leads to overfitting. In an attempt to circumvent this issue, a prior knowledge pertaining to what features are relev…

Study on solvable Lie groups with specific Weyl connections.

New method learns convolution-like structures from scratch.

Deep Reinforcement Learning (RL) recently emerged as one of the most competitive approaches for learning in sequential decision making problems with fully observable environments, e.g., computer Go. However, very little work has been done in deep RL to handle partially observable environments. We propose a new architec…

In this empirical paper, we investigate how learning agents can be arranged in more efficient communication topologies for improved learning. This is an important problem because a common technique to improve speed and robustness of learning in deep reinforcement learning and many other machine learning algorithms is t…

Pruning FCNs reveals sub-networks that match CNNs' performance.

We study alignment in linear neural networks and its relation to gradient descent.

The past few years have witnessed the fast development of different regularization methods for deep learning models such as fully-connected deep neural networks (DNNs) and Convolutional Neural Networks (CNNs). Most of previous methods mainly consider to drop features from input data and hidden layers, such as Dropout, …

We propose Sparse Neural Network architectures that are based on random or structured bipartite graph topologies. Sparse architectures provide compression of the models learned and speed-ups of computations, they can also surpass their unstructured or fully connected counterparts. As we show, even more compact topologi…

In this paper, we will describe a network architecture that demonstrates high performance on various sizes of datasets. To do this, we will perform an architecture search by dividing the fully connected layer into three levels in the existing network architecture. The first step is to learn existing CNN layer and exist…

The parameter server architecture is prevalently used for distributed deep learning. Each worker machine in a parameter server system trains the complete model, which leads to a hefty amount of network data transfer between workers and servers. We empirically observe that the data transfer has a non-negligible impact o…

Let $u$ denote a solution to a rotationally invariant Hessian equation $F(D^2u)=0$ on a bounded simply connected domain $Ω\subset R^2$, with constant Dirichlet and Neumann data on $\partial Ω$. In this paper we prove that if $u$ is real analytic and not identically zero, then $u$ is radial and $Ω$ is a disk. The fully …

We present a transductive deep learning-based formulation for the sparse representation-based classification (SRC) method. The proposed network consists of a convolutional autoencoder along with a fully-connected layer. The role of the autoencoder network is to learn robust deep features for classification. On the othe…

A common technique to improve learning performance in deep reinforcement learning (DRL) and many other machine learning algorithms is to run multiple learning agents in parallel. A neglected component in the development of these algorithms has been how best to arrange the learning agents involved to improve distributed…

In this work, we leverage advances in sparse coding techniques to reduce the number of trainable parameters in a fully connected neural network. While most of the works in literature impose $\ell_1$ regularization, DropOut or DropConnect techniques to induce sparsity, our scheme considers feature importance as a criter…

We discuss dense embeddings of surface groups and fully residually free groups in topological groups. We show that a compact topological group contains a nonabelian dense free group of finite rank if and only if it contains a dense surface group. Also, we obtain a characterization of those Lie groups which admit a dens…

How well does a classic deep net architecture like AlexNet or VGG19 classify on a standard dataset such as CIFAR-10 when its width --- namely, number of channels in convolutional layers, and number of nodes in fully-connected internal layers --- is allowed to increase to infinity? Such questions have come to the forefr…

Recently popularized graph neural networks achieve the state-of-the-art accuracy on a number of standard benchmark datasets for graph-based semi-supervised learning, improving significantly over existing approaches. These architectures alternate between a propagation layer that aggregates the hidden states of the local…

Recently the generalization error of deep neural networks has been analyzed through the PAC-Bayesian framework, for the case of fully connected layers. We adapt this approach to the convolutional setting.

Proves DCNNs with expansive convolution are strongly universally consistent.

BayGo framework optimizes decentralized learning in multi-agent networks.

Study shows partially-typed NER datasets can match fully-typed ones in model performance.

Study examines noise sensitivity of DNNs for binary classification.

For any finite collection $f_i$ of fully irreducible automorphisms of the free group $F_n$ we construct a connected $δ$-hyperbolic $Out(F_n)$-complex in which each $f_i$ has positive translation length.

Deep neural networks estimate regression functions on manifolds.

Dropout is a popular regularization technique in deep learning. Yet, the reason for its success is still not fully understood. This paper provides a new interpretation of Dropout from a frame theory perspective. By drawing a connection to recent developments in analog channel coding, we suggest that for a certain famil…

We consider the effects of the global financial crisis through a local Korean financial market around the 2008 crisis. We analyze 185 individual stock prices belonging to the KOSPI (Korea Composite Stock Price Index), cosidering three time periods: the time before, during, and after the crisis. The complex networks gen…

Belief propagation (BP) can do exact inference in loop-free graphs, but its performance could be poor in graphs with loops, and the understanding of its solution is limited. This work gives an interpretable belief propagation rule that is actually minimization of a localized $α$-divergence. We term this algorithm as $α…

RIFLE improves deep transfer learning by reinitializing fully-connected layers.

The fully connected layers of a deep convolutional neural network typically contain over 90% of the network parameters, and consume the majority of the memory required to store the network parameters. Reducing the number of parameters while preserving essentially the same predictive performance is critically important …

Classifies connections on Galilei manifolds, generalizing known results.