arXiv research

Graph neural networks suffer from oversmoothing, but adding residual connections helps.

Classifies connected components of meromorphic differentials with residue conditions.

Constructs currents representing Baum-Bott residues for foliations.

Residual connections significantly boost the performance of deep neural networks. However, there are few theoretical results that address the influence of residuals on the hypothesis complexity and the generalization ability of deep neural networks. This paper studies the influence of residual connections on the hypoth…

In this paper, we propose a novel unsupervised learning method to learn the brain dynamics using a deep learning architecture named residual D-net. As it is often the case in medical research, in contrast to typical deep learning tasks, the size of the resting-state functional Magnetic Resonance Image (rs-fMRI) dataset…

We present the Video Ladder Network (VLN) for efficiently generating future video frames. VLN is a neural encoder-decoder model augmented at all layers by both recurrent and feedforward lateral connections. At each layer, these connections form a lateral recurrent residual block, where the feedforward connection repres…

This paper classifies components of meromorphic differential strata.

The paper establishes a Poisson Poincaré-Dulac theorem for Poisson-flat connections.

New spectral functionals related to noncommutative residue and torsion Dirac operators.

We introduce a new deep convolutional neural network, CrescendoNet, by stacking simple building blocks without residual connections. Each Crescendo block contains independent convolution paths with increased depths. The numbers of convolution layers and parameters are only increased linearly in Crescendo blocks. In exp…

GT-DDP optimizer trains residual networks using game theory.

Compact manifolds with specific cover properties are hyperbolic.

The paper establishes principles for initializing and designing GNNs with ReLU activations to avoid oversmoothing and correlation collapse.

Residual Neural Networks (ResNets) achieve state-of-the-art performance in many computer vision problems. Compared to plain networks without residual connections (PlnNets), ResNets train faster, generalize better, and suffer less from the so-called degradation problem. We introduce simplified (but still nonlinear) vers…

New findings show mapping class groups of certain high-dimensional manifolds are not residually finite.

Batch normalization makes deep residual networks train faster.

Gradient oversmoothing and expansion hinder deep GNN training, solved with normalization.

Let $M$ be a smooth manifold and $G$ a compact connected Lie group acting on $M$ by isometries. In this paper, we study the equivariant cohomology of ${\bf X}=T^\ast M$, and relate it to the cohomology of the Marsden-Weinstein reduced space via certain residue formulae. In case that $\bf X$ is a compact symplectic mani…

Theoretical limits of deep residual networks show consistent covariance structures.

Due to the success of residual networks (resnets) and related architectures, shortcut connections have quickly become standard tools for building convolutional neural networks. The explanations in the literature for the apparent effectiveness of shortcuts are varied and often contradictory. We hypothesize that shortcut…

One of the ways to train deep neural networks effectively is to use residual connections. Residual connections can be classified as being either identity connections or bridge-connections with a reshaping convolution. Empirical observations on CIFAR-10 and CIFAR-100 datasets using a baseline Resnet model, with bridge-c…

The skip-connections used in residual networks have become a standard architecture choice in deep learning due to the increased training stability and generalization performance with this architecture, although there has been limited theoretical understanding for this improvement. In this work, we analyze overparameter…

This paper proposes a deep speech enhancement method which exploits the high potential of residual connections in a wide neural network architecture, a topology known as Wide Residual Network. This is supported on single dimensional convolutions computed alongside the time domain, which is a powerful approach to proces…

Residually finite groups found in manifold automorphisms.

Recent results in the literature indicate that a residual network (ResNet) composed of a single residual block outperforms linear predictors, in the sense that all local minima in its optimization landscape are at least as good as the best linear predictor. However, these results are limited to a single residual block …

Using the Wodzicki residue, we build Wodzicki-Chern-Simons (WCS) classes in $H^{2k-1}(LM)$ associated to the residue Chern character on the loop space $LM$ of a Riemannian manifold $M^{2k-1}$. These WCS classes are associated to the $L^2$ connection and the Sobolev $s=1$ connections on $LM.$ The WCS classes detect seve…

Materials discovery is crucial for making scientific advances in many domains. Collections of data from experiments and first-principle computations have spurred interest in applying machine learning methods to create predictive models capable of mapping from composition and crystal structures to materials properties. …

Researchers prove Transformers are Turing-complete and analyze their components.

The study examines Hopfian properties of conjugation quandles and their underlying groups.

The study proves residual finiteness for certain lattice extensions and negatively curved projective varieties.

RDL-Net improves speech enhancement with fewer parameters and better performance.

We show that Residual Networks (ResNet) is equivalent to boosting feature representation, without any modification to the underlying ResNet training algorithm. A regret bound based on Online Gradient Boosting theory is proved and suggests that ResNet could achieve Online Gradient Boosting regret bounds through neural n…

New method diagnoses criticality in deep neural networks, improving performance.

We discuss dense embeddings of surface groups and fully residually free groups in topological groups. We show that a compact topological group contains a nonabelian dense free group of finite rank if and only if it contains a dense surface group. Also, we obtain a characterization of those Lie groups which admit a dens…

We identify and study two common failure modes for early training in deep ReLU nets. For each we give a rigorous proof of when it occurs and how to avoid it, for fully connected and residual architectures. The first failure mode, exploding/vanishing mean activation length, can be avoided by initializing weights from a …

Training deep recurrent neural network (RNN) architectures is complicated due to the increased network complexity. This disrupts the learning of higher order abstracts using deep RNN. In case of feed-forward networks training deep structures is simple and faster while learning long-term temporal information is not poss…

Paper introduces a new multilinear functional for spectral triples and computes its properties.

We add size factor to CAPM and normalize residuals by Volatility Index.

Residual networks (ResNets) are a deep learning architecture that substantially improved the state of the art performance in certain supervised learning tasks. Since then, they have received continuously growing attention. ResNets have a recursive structure $x_{k+1} = x_k + R_k(x_k)$ where $R_k$ is a neural network cal…

In this paper, we introduce transformations of deep rectifier networks, enabling the conversion of deep rectifier networks into shallow rectifier networks. We subsequently prove that any rectifier net of any depth can be represented by a maximum of a number of functions that can be realized by a shallow network with a …

Proves a stack of G-bundles with logarithmic connections is finite type.

On a complex curve, we establish a correspondence between integrable connections with irregular singularities, and Higgs bundles such that the Higgs field is meromorphic with poles of any order. The moduli spaces of these objects are obtained by fixing at each singularity the polar part of the connection. We prove that…

In this work, we propose a deep neural network architecture motivated by primal-dual splitting methods from convex optimization. We show theoretically that there exists a close relation between the derived architecture and residual networks, and further investigate this connection in numerical experiments. Moreover, we…

The existing graph neural networks (GNNs) based on the spectral graph convolutional operator have been criticized for its performance degradation, which is especially common for the models with deep architectures. In this paper, we further identify the suspended animation problem with the existing GNNs. Such a problem …

Nontrivial connectivity has allowed the training of very deep networks by addressing the problem of vanishing gradients and offering a more efficient method of reusing parameters. In this paper we make a comparison between residual networks, densely-connected networks and highway networks on an image classification tas…

We regard pre-trained residual networks (ResNets) as nonlinear systems and use linearization, a common method used in the qualitative analysis of nonlinear systems, to understand the behavior of the networks under small perturbations of the input images. We work with ResNet-56 and ResNet-110 trained on the CIFAR-10 dat…

Paper studies ResNet dynamics using NTH, reducing width requirement.

Residual Network (ResNet) is undoubtedly a milestone in deep learning. ResNet is equipped with shortcut connections between layers, and exhibits efficient training using simple first order algorithms. Despite of the great empirical success, the reason behind is far from being well understood. In this paper, we study a …