A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
Paper analyzes the free energy of CNNs with skip connections in Bayesian learning.
Skip connections make adversarial examples more transferable in ResNets.
The paper proves skip connections help neural networks avoid shallow local minima.
New insights into GNN optimization reveal skip connections and depth accelerate training.
This paper examines how skip connections prevent rank collapse in sequence models.
This work analyzes how bottleneck layers and skip connections affect linear denoising autoencoders' generalization.
DenseNets introduce concatenation-type skip connections that achieve state-of-the-art accuracy in several computer vision tasks. In this paper, we reveal that the topology of the concatenation-type skip connections is closely related to the gradient propagation which, in turn, enables a predictable behavior of DNNs' te…
DARTS- improves robustness by factoring out skip connections' advantage.
Theory explains why DARTS favors deep architectures over shallow ones.
Study on autoencoder denoising in high dimensions.
This paper explains GCNs using NTKs and improves their performance.
Backpropagation is the workhorse of deep learning, however, several other biologically-motivated learning rules have been introduced, such as random feedback alignment and difference target propagation. None of these methods have produced a competitive performance against backpropagation. In this paper, we show that bi…
In this work, we extend the SchNet architecture by using weighted skip connections to assemble the final representation. This enables us to study the relative importance of each interaction block for property prediction. We demonstrate on both the QM9 and MD17 dataset that their relative weighting depends strongly on t…
A long-standing obstacle to progress in deep learning is the problem of vanishing and exploding gradients. Although, the problem has largely been overcome via carefully constructed initializations and batch normalization, architectures incorporating skip-connections such as highway and resnets perform much better than …
CAggNet improves medical image segmentation by fusing coarse and fine features.
Proposes PSCs for UQ in deep nets without retraining.
Theoretical limits of deep residual networks show consistent covariance structures.
We introduce a novel approach, requiring only mild assumptions, for the characterization of deep neural networks at initialization. Our approach applies both to fully-connected and convolutional networks and easily incorporates batch normalization and skip-connections. Our key insight is to consider the evolution with …
The behavior of the gradient descent (GD) algorithm is analyzed for a deep neural network model with skip-connections. It is proved that in the over-parametrized regime, for a suitable initialization, with high probability GD can find a global minimum exponentially fast. Generalization error estimates along the GD path…
Paper studies ResNet dynamics using NTH, reducing width requirement.
While enormous progress has been made to Variational Autoencoder (VAE) in recent years, similar to other deep networks, VAE with deep networks suffers from the problem of degeneration, which seriously weakens the correlation between the input and the corresponding latent codes, deviating from the goal of the representa…
OSA overcomes instability in skipless Transformers.
Wide CNNs outperform infinite width networks, revealing scaling laws.
New framework explains deep learning using signal processing techniques.
Random feature maps improve forecasting of chaotic dynamical systems.
To discover powerful yet compact models is an important goal of neural architecture search. Previous two-stage one-shot approaches are limited by search space with a fixed depth. It seems handy to include an additional skip connection in the search space to make depths variable. However, it creates a large range of per…
One of the main difficulties in analyzing neural networks is the non-convexity of the loss function which may have many bad local minima. In this paper, we study the landscape of neural networks for binary classification tasks. Under mild assumptions, we prove that after adding one special neuron with a skip connection…
NoisyDARTS injects random noise to improve neural architecture search.
This manuscript addresses the problem of the automatic lesion boundary detection in dermoscopy, using deep neural networks. An approach is based on the adaptation of the U-net convolutional neural network with skip connections for lesion boundary segmentation task. I hope this paper could serve, to some extent, as an e…
New research shows non-bottlenecked autoencoders can outperform bottlenecked ones for anomaly detection.
Proposes -Nets, polynomial neural networks, for improved representation power.
New framework explains deep neural networks using variational spline theory.
DNAS disentangles neural architecture search for better interpretability and performance.
New neural architectures with multivariate nonlinearities are optimal in function space.
Deep Heaviside networks are limited but can be improved with connections or linear neurons.
This paper explores how neural network width and depth behave as they approach infinity.
Deep generative models parameterized by neural networks have recently achieved state-of-the-art performance in unsupervised and semi-supervised learning. We extend deep generative models with auxiliary variables which improves the variational approximation. The auxiliary variables leave the generative model unchanged b…
We present a framework for translating unlabeled images from one domain into analog images in another domain. We employ a progressively growing skip-connected encoder-generator structure and train it with a GAN loss for realistic output, a cycle consistency loss for maintaining same-domain translation identity, and a s…
PS8-Net improves eight-state protein secondary structure prediction accuracy.
Recent results in the literature indicate that a residual network (ResNet) composed of a single residual block outperforms linear predictors, in the sense that all local minima in its optimization landscape are at least as good as the best linear predictor. However, these results are limited to a single residual block …
We empirically evaluate common assumptions about neural networks that are widely held by practitioners and theorists alike. In this work, we: (1) prove the widespread existence of suboptimal local minima in the loss landscape of neural networks, and we use our theory to find examples; (2) show that small-norm parameter…
We introduce the HSIC (Hilbert-Schmidt independence criterion) bottleneck for training deep neural networks. The HSIC bottleneck is an alternative to the conventional cross-entropy loss and backpropagation that has a number of distinct advantages. It mitigates exploding and vanishing gradients, resulting in the ability…
NeuraLUT maps neural networks to lookup tables, reducing latency and improving expressivity.
This work investigates how neural collapse improves transfer learning for large-scale models.
Encoder-decoder networks using convolutional neural network (CNN) architecture have been extensively used in deep learning literatures thanks to its excellent performance for various inverse problems. However, it is still difficult to obtain coherent geometric view why such an architecture gives the desired performance…
Transformers without skip connections collapse token representations to a single direction.
Inverse problems in imaging such as denoising, deblurring, superresolution (SR) have been addressed for many decades. In recent years, convolutional neural networks (CNNs) have been widely used for many inverse problem areas. Although their indisputable success, CNNs are not mathematically validated as to how and what …
New model uses attention for in-context learning of categorical data.