DFS dynamically decides bitwidths for layers to balance accuracy and efficiency.
problem Balancing model accuracy and inference speed for deep networks.
method Dynamic Fractional Skipping (DFS) framework that assigns bitwidths to layers for input-adaptive inference.
result DFS achieves superior tradeoff between computational cost and model accuracy.
New insights into GNN optimization reveal skip connections and depth accelerate training.
problem Understanding and optimizing the training of Graph Neural Networks (GNNs).
method Analysis of gradient dynamics in linearized GNNs and empirical validation.
result GNNs are implicitly accelerated by skip connections, more depth, and good label distribution during training.
A new method learns dynamic graph representations from time-varying data.
problem Learning dynamic graph representations from time-varying data.
method Higher-order skip-gram with negative sampling (HOSGNS) for tensor factorization.
result HOSGNS outperforms state-of-the-art methods in downstream tasks.
We introduce a method which enables a recurrent dynamics model to be temporally abstract. Our approach, which we call Adaptive Skip Intervals (ASI), is based on the observation that in many sequential prediction tasks, the exact time at which events occur is irrelevant to the underlying objective. Moreover, in many sit…
Random feature maps improve forecasting of chaotic dynamical systems.
problem Forecasting chaotic dynamical systems with high accuracy.
method Data-driven random feature maps with tanh activation, skip connections, and localization.
result Effective forecasting skill for dynamical systems with dimensions up to 512.
We present a probabilistic language model for time-stamped text data which tracks the semantic evolution of individual words over time. The model represents words and contexts by latent trajectories in an embedding space. At each moment in time, the embedding vectors are inferred from a probabilistic version of word2ve…
Paper analyzes the free energy of CNNs with skip connections in Bayesian learning.
problem Dependency of CNNs with skip connections on the number of parameters.
method Examines the Bayesian free energy of CNNs with and without skip connections.
result The upper bound of free energy of Bayesian CNN with skip connections does not depend on overparametrization.
Skip connections make adversarial examples more transferable in ResNets.
problem Transferability of adversarial examples in ResNets with skip connections.
method Skip Gradient Method (SGM) to craft more transferable adversarial examples.
result SGM improves transferability of adversarial examples in various DNNs.
Paper studies ResNet dynamics using NTH, reducing width requirement.
problem Understanding ResNet dynamics and improving training efficiency.
method Uses Neural Tangent Hierarchy (NTH) to analyze ResNet dynamics.
result Reduces width requirement from quartic to cubic for ResNet.
Self-supervised skip-tree training improves mathematical reasoning in language models.
problem Improving logical reasoning in language models for formal mathematics.
method Self-supervised language modeling on mathematical formulas, skip-tree task.
result Models trained on skip-tree task outperform standard models in mathematical reasoning tasks.
TabSODA improves imputation of surveys with skips and ordinal data.
problem Handling structural skips and ordinal responses in survey data.
method TabSODA uses an Elucidated Diffusion Model with skip pattern detection and ordinal awareness.
result TabSODA reduces ordinal missing-at-random (MACE) by up to 23.7% and improves categorical accuracy by up to 9%.
The paper proves skip connections help neural networks avoid shallow local minima.
problem Understanding how skip connections affect the loss landscape of deep neural networks.
method Theoretical analysis of the topology of loss landscapes of deep ReLU neural networks with skip connections.
result Skip connections help control the connectedness of sub-level sets, avoiding shallow local minima.
Wide CNNs outperform infinite width networks, revealing scaling laws.
problem Understanding the performance difference between finite and infinite width convolutional networks.
method Diagrammatic approach to derive asymptotic width dependence for various quantities.
result The difference in performance between finite and infinite width models vanishes at a definite rate with respect to model width.
This paper examines how skip connections prevent rank collapse in sequence models.
problem Rank collapse in sequence models, leading to reduced expressivity and training instabilities.
method Analytical and ablation studies of lambda-skip connections in SSMs.
result A sufficient condition to prevent rank collapse across various architectures.
Boost-and-Skip generates minority samples without guidance, faster and more effectively.
problem Generating minority samples in low-density regions of a data manifold.
method Boost-and-Skip approach with variance-boosted initialization and timestep skipping.
result Boost-and-Skip effectively promotes the emergence of underrepresented minority features.
Network representation learning, as an approach to learn low dimensional representations of vertices, has attracted considerable research attention recently. It has been proven extremely useful in many machine learning tasks over large graph. Most existing methods focus on learning the structural representations of ver…
Corrected CBOW performs similarly to Skip-gram.
problem CBOW embeddings underperform Skip-gram embeddings in word2vec.
method Fixed a bug in CBOW gradient update to improve performance.
result Corrected CBOW embeddings are competitive with Skip-gram on various tasks.
This work analyzes how bottleneck layers and skip connections affect linear denoising autoencoders' generalization.
problem Understanding the generalization of linear denoising autoencoders in overparameterized regimes.
method Analyzes two-layer linear denoising autoencoders with a bottleneck layer and skip connection, deriving test risk formulas.
result Bottleneck layers introduce an additional complexity measure, while skip connections can mitigate variance.
PAM models generate dependent random distributions across groups with overlapping clusters.
problem Generating dependent random distributions across multiple groups.
method Atom skipping in an infinite mixture model.
result Interpretable posterior inference of cluster exclusivity and sharing.
DenseNets introduce concatenation-type skip connections that achieve state-of-the-art accuracy in several computer vision tasks. In this paper, we reveal that the topology of the concatenation-type skip connections is closely related to the gradient propagation which, in turn, enables a predictable behavior of DNNs' te…
DARTS- improves robustness by factoring out skip connections' advantage.
problem Performance collapse in DARTS architecture search.
method Factor out skip connections' advantage with an auxiliary skip connection.
result Significantly improved robustness across various datasets.
Variational autoencoders learn distributions of high-dimensional data. They model data with a deep latent-variable model and then fit the model by maximizing a lower bound of the log marginal likelihood. VAEs can capture complex distributions, but they can also suffer from an issue known as "latent variable collapse," …
We show that the skip-gram formulation of word2vec trained with negative sampling is equivalent to a weighted logistic PCA. This connection allows us to better understand the objective, compare it to other word embedding methods, and extend it to higher dimensional models.
Paper explores variable skipping to speed up range density estimation.
problem Efficiently estimating range densities over high-dimensional data.
method Variable skipping technique to accelerate range density estimation.
result 10-100x efficiency improvements in challenging high-quantile error metrics.
Theory explains why DARTS favors deep architectures over shallow ones.
problem DARTS selects architectures with dominated skip connections, leading to performance degradation.
method Theoretical analysis of operations' effects on network optimization; introduces sparse binary gates and path-depth-wise regularization.
result Theoretical proof that architectures with more skip connections converge faster.
Generates low-dimensional node vectors for graphs with privacy while preserving structural preferences.
problem Publishing graph node vectors can leak sensitive individual information.
method SE-PrivGEmb, a skip-gram based technique with a unified noise tolerance mechanism and negative sampling probabilities.
result Our method outperforms existing methods in structural equivalence and link prediction tasks.
Simplified Bayesian neural networks reduce model complexity and improve interpretability.
problem Over-parameterization and interpretability issues in deep learning models.
method Input-skip Latent Binary Bayesian Neural Networks (LBBNNs) that allow covariates to skip layers or be excluded.
result Significant reduction in model complexity (over 99%) with minimal loss in accuracy and uncertainty.
LOBDIF predicts limit order book events using a diffusion model.
problem Predicting the timing and type of events in a dynamic market system.
method LOBDIF uses a diffusion model to learn the complex time-event distribution in limit order book streams.
result LOBDIF significantly outperforms existing methods in real-world data experiments.
Study on autoencoder denoising in high dimensions.
problem Denoising data from Gaussian mixtures.
method Two-layer non-linear autoencoder with skip connection in high-dimensional limit.
result Closed-form expressions for denoising mean-squared test error.
This paper explains GCNs using NTKs and improves their performance.
problem GCNs' performance degrades with depth, and skip connections marginally improve it.
method Derive NTKs for GCNs, validate with simulations, propose NTK as a surrogate model.
result Suitable normalisation can prevent drastic performance drop with depth.
Backpropagation is the workhorse of deep learning, however, several other biologically-motivated learning rules have been introduced, such as random feedback alignment and difference target propagation. None of these methods have produced a competitive performance against backpropagation. In this paper, we show that bi…
In this work, we extend the SchNet architecture by using weighted skip connections to assemble the final representation. This enables us to study the relative importance of each interaction block for property prediction. We demonstrate on both the QM9 and MD17 dataset that their relative weighting depends strongly on t…
We describe a method for learning word embeddings with data-dependent dimensionality. Our Stochastic Dimensionality Skip-Gram (SD-SG) and Stochastic Dimensionality Continuous Bag-of-Words (SD-CBOW) are nonparametric analogs of Mikolov et al.'s (2013) well-known 'word2vec' models. Vector dimensionality is made dynamic b…
This work proposes a novel neural network architecture, called the Dynamically Controlled Recurrent Neural Network (DCRNN), specifically designed to model dynamical systems that are governed by ordinary differential equations (ODEs). The current state vectors of these types of dynamical systems only depend on their sta…
ResiliNet improves distributed neural network inference resilience.
problem Physical node failures in distributed neural networks cause performance drops.
method Skip hyperconnection and failout technique.
result ResiliNet provides inference resiliency for distributed neural networks.
Proposes SNML for selecting word2vec Skip-gram dimensionality.
problem Selecting optimal dimensionality for word2vec Skip-gram models.
method Information criteria (AIC, BIC, SNML) applied to SG and SG Negative Sampling models.
result SNML outperforms AIC and BIC, selecting closer optimal dimensionality.
A long-standing obstacle to progress in deep learning is the problem of vanishing and exploding gradients. Although, the problem has largely been overcome via carefully constructed initializations and batch normalization, architectures incorporating skip-connections such as highway and resnets perform much better than …
This work shows dimension regularization can replace skip-gram negative sampling for graph embeddings, improving efficiency and performance.
problem Efficiently enforcing dissimilarity among node embeddings in graph learning.
method Dimension regularization as an alternative to skip-gram negative sampling.
result Dimension regularization is a more efficient approach to enforcing dissimilarity in graph embeddings.
Without relevant human priors, neural networks may learn uninterpretable features. We propose Dynamics of Attention for Focus Transition (DAFT) as a human prior for machine reasoning. DAFT is a novel method that regularizes attention-based reasoning by modelling it as a continuous dynamical system using neural ordinary…
Eliciting semantic similarity between concepts in the biomedical domain remains a challenging task. Recent approaches founded on embedding vectors have gained in popularity as they risen to efficiently capture semantic relationships The underlying idea is that two words that have close meaning gather similar contexts. …
DVNet efficiently segments large neurovascular datasets using skip connections.
problem Challenges in segmenting large neurovascular datasets from high-throughput microscopy data.
method A fully-convolutional, deep, and densely-connected encoder-decoder network with skip connections.
result DVNet achieves superior performance in semantic segmentation of neurovascular datasets.
New RL method learns to skip states in linearly qπ-realizable MDPs, simplifying to linear MDPs.
problem Online RL in episodic MDPs with linearly qπ-realizable action-values. method Derives a novel algorithm that learns to skip states and applies a linear MDP algorithm.
result First polynomial-sample-complexity online RL algorithm for linearly qπ-realizable MDPs. CAggNet improves medical image segmentation by fusing coarse and fine features.
problem Medical image segmentation accuracy and efficiency.
method Crossing Aggregation Network with nested skip connections and weighted aggregation.
result CAggNet achieves more accurate and efficient segmentation compared to existing methods.
Attention-only transformers learn from context via two stages of inference.
problem Learning from corrupted token sequences in minimal transformers.
method Two-stage empirical Bayes interpretation: kernel-weighted posterior mean and particle dynamics.
result Effective denoising without explicit noise schedules, showing posterior-mean recovery under asymptotic conditions.
Recent work has demonstrated that embeddings of tree-like graphs in hyperbolic space surpass their Euclidean counterparts in performance by a large margin. Inspired by these results and scale-free structure in the word co-occurrence graph, we present an algorithm for learning word embeddings in hyperbolic space from fr…
Since the invention of word2vec, the skip-gram model has significantly advanced the research of network embedding, such as the recent emergence of the DeepWalk, LINE, PTE, and node2vec approaches. In this work, we show that all of the aforementioned models with negative sampling can be unified into the matrix factoriza…
The present paper develops a novel aggregated gradient approach for distributed machine learning that adaptively compresses the gradient communication. The key idea is to first quantize the computed gradients, and then skip less informative quantized gradient communications by reusing outdated gradients. Quantizing and…
DRew dynamically rewires message passing to improve long-range tasks.
problem Over-squashing in MPNNs due to local message passing.
method Layer-dependent rewiring and delay mechanism for gradual densification.
result DRew outperforms graph Transformers and multi-hop MPNNs on long-range tasks.