A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
Paper analyzes the free energy of CNNs with skip connections in Bayesian learning.
The paper proves skip connections help neural networks avoid shallow local minima.
Skip connections are an essential component of current state-of-the-art deep neural networks (DNNs) such as ResNet, WideResNet, DenseNet, and ResNeXt. Despite their huge success in building deeper and more powerful DNNs, we identify a surprising security weakness of skip connections in this paper. Use of skip connectio…
New insights into GNN optimization reveal skip connections and depth accelerate training.
This paper examines how skip connections prevent rank collapse in sequence models.
This work analyzes how bottleneck layers and skip connections affect linear denoising autoencoders' generalization.
DenseNets introduce concatenation-type skip connections that achieve state-of-the-art accuracy in several computer vision tasks. In this paper, we reveal that the topology of the concatenation-type skip connections is closely related to the gradient propagation which, in turn, enables a predictable behavior of DNNs' te…
DARTS- improves robustness by factoring out skip connections' advantage.
Theory explains why DARTS favors deep architectures over shallow ones.
Study on autoencoder denoising in high dimensions.
This paper explains GCNs using NTKs and improves their performance.
We show that the skip-gram formulation of word2vec trained with negative sampling is equivalent to a weighted logistic PCA. This connection allows us to better understand the objective, compare it to other word embedding methods, and extend it to higher dimensional models.
Backpropagation is the workhorse of deep learning, however, several other biologically-motivated learning rules have been introduced, such as random feedback alignment and difference target propagation. None of these methods have produced a competitive performance against backpropagation. In this paper, we show that bi…
In this work, we extend the SchNet architecture by using weighted skip connections to assemble the final representation. This enables us to study the relative importance of each interaction block for property prediction. We demonstrate on both the QM9 and MD17 dataset that their relative weighting depends strongly on t…
A long-standing obstacle to progress in deep learning is the problem of vanishing and exploding gradients. Although, the problem has largely been overcome via carefully constructed initializations and batch normalization, architectures incorporating skip-connections such as highway and resnets perform much better than …
Variational autoencoders learn distributions of high-dimensional data. They model data with a deep latent-variable model and then fit the model by maximizing a lower bound of the log marginal likelihood. VAEs can capture complex distributions, but they can also suffer from an issue known as "latent variable collapse," …
CAggNet improves medical image segmentation by fusing coarse and fine features.
We introduce a novel approach, requiring only mild assumptions, for the characterization of deep neural networks at initialization. Our approach applies both to fully-connected and convolutional networks and easily incorporates batch normalization and skip-connections. Our key insight is to consider the evolution with …
Proposes PSCs for UQ in deep nets without retraining.
Theoretical limits of deep residual networks show consistent covariance structures.
Paper studies ResNet dynamics using NTH, reducing width requirement.
Self-supervised skip-tree training improves mathematical reasoning in language models.
TabSODA improves imputation of surveys with skips and ordinal data.
The behavior of the gradient descent (GD) algorithm is analyzed for a deep neural network model with skip-connections. It is proved that in the over-parametrized regime, for a suitable initialization, with high probability GD can find a global minimum exponentially fast. Generalization error estimates along the GD path…
While enormous progress has been made to Variational Autoencoder (VAE) in recent years, similar to other deep networks, VAE with deep networks suffers from the problem of degeneration, which seriously weakens the correlation between the input and the corresponding latent codes, deviating from the goal of the representa…
While increasingly deep networks are still in general desired for achieving state-of-the-art performance, for many specific inputs a simpler network might already suffice. Existing works exploited this observation by learning to skip convolutional layers in an input-dependent manner. However, we argue their binary deci…
OSA overcomes instability in skipless Transformers.
Boost-and-Skip generates minority samples without guidance, faster and more effectively.
Wide CNNs outperform infinite width networks, revealing scaling laws.
New framework explains deep learning using signal processing techniques.
Federated Learning aims to train distributed deep models without sharing the raw data with the centralized server. Similarly, in distributed inference of neural networks, by partitioning the network and distributing it across several physical nodes, activations and gradients are exchanged between physical nodes, rather…
Random feature maps improve forecasting of chaotic dynamical systems.
We present a probabilistic language model for time-stamped text data which tracks the semantic evolution of individual words over time. The model represents words and contexts by latent trajectories in an embedding space. At each moment in time, the embedding vectors are inferred from a probabilistic version of word2ve…
Corrected CBOW performs similarly to Skip-gram.
Since the invention of word2vec, the skip-gram model has significantly advanced the research of network embedding, such as the recent emergence of the DeepWalk, LINE, PTE, and node2vec approaches. In this work, we show that all of the aforementioned models with negative sampling can be unified into the matrix factoriza…
To discover powerful yet compact models is an important goal of neural architecture search. Previous two-stage one-shot approaches are limited by search space with a fixed depth. It seems handy to include an additional skip connection in the search space to make depths variable. However, it creates a large range of per…
PAM models generate dependent random distributions across groups with overlapping clusters.
New framework explains deep neural networks using variational spline theory.
Deep Heaviside networks are limited but can be improved with connections or linear neurons.
New neural architectures with multivariate nonlinearities are optimal in function space.
We show that removing sigmoid transformation in the skip-gram with negative sampling (SGNS) objective does not harm the quality of word vectors significantly and at the same time is related to factorizing a squashed shifted PMI matrix which, in turn, can be treated as a connection probabilities matrix of a random graph…
One of the main difficulties in analyzing neural networks is the non-convexity of the loss function which may have many bad local minima. In this paper, we study the landscape of neural networks for binary classification tasks. Under mild assumptions, we prove that after adding one special neuron with a skip connection…
Paper explores variable skipping to speed up range density estimation.
NoisyDARTS injects random noise to improve neural architecture search.
This manuscript addresses the problem of the automatic lesion boundary detection in dermoscopy, using deep neural networks. An approach is based on the adaptation of the U-net convolutional neural network with skip connections for lesion boundary segmentation task. I hope this paper could serve, to some extent, as an e…
InfiniteWalk connects deep network embeddings to spectral graph theory with a nonlinear transformation.
New research shows non-bottlenecked autoencoders can outperform bottlenecked ones for anomaly detection.
Proposes -Nets, polynomial neural networks, for improved representation power.