Boost-and-Skip generates minority samples without guidance, faster and more effectively.
problem Generating minority samples in low-density regions of a data manifold.
method Boost-and-Skip approach with variance-boosted initialization and timestep skipping.
result Boost-and-Skip effectively promotes the emergence of underrepresented minority features.
RNN operators solve Newton's equations with large timesteps for molecular dynamics.
problem Solving Newton's equations of motion with large timesteps for molecular dynamics simulations.
method Recurrent Neural Networks (RNN) operators to solve Newton's equations using past trajectory data.
result Significant speedup in molecular dynamics simulations with timesteps up to 4000 times larger.
Accurate forward modeling is important for solving inverse problems. An inaccurate wave-equation simulation, as a forward operator, will offset the results obtained via inversion. In this work, we consider the case where we deal with incomplete physics. One proxy of incomplete physics is an inaccurate discretization of…
Paper analyzes the free energy of CNNs with skip connections in Bayesian learning.
problem Dependency of CNNs with skip connections on the number of parameters.
method Examines the Bayesian free energy of CNNs with and without skip connections.
result The upper bound of free energy of Bayesian CNN with skip connections does not depend on overparametrization.
Skip connections make adversarial examples more transferable in ResNets.
problem Transferability of adversarial examples in ResNets with skip connections.
method Skip Gradient Method (SGM) to craft more transferable adversarial examples.
result SGM improves transferability of adversarial examples in various DNNs.
The paper improves model-based reinforcement learning by using multi-timestep objectives.
problem Compounding errors in one-step dynamics models as trajectory length increases.
method Developed a multi-timestep objective as a weighted sum of losses at various future horizons.
result Exponentially decaying weights significantly improve long-horizon performance.
Self-supervised skip-tree training improves mathematical reasoning in language models.
problem Improving logical reasoning in language models for formal mathematics.
method Self-supervised language modeling on mathematical formulas, skip-tree task.
result Models trained on skip-tree task outperform standard models in mathematical reasoning tasks.
TabSODA improves imputation of surveys with skips and ordinal data.
problem Handling structural skips and ordinal responses in survey data.
method TabSODA uses an Elucidated Diffusion Model with skip pattern detection and ordinal awareness.
result TabSODA reduces ordinal missing-at-random (MACE) by up to 23.7% and improves categorical accuracy by up to 9%.
The paper proves skip connections help neural networks avoid shallow local minima.
problem Understanding how skip connections affect the loss landscape of deep neural networks.
method Theoretical analysis of the topology of loss landscapes of deep ReLU neural networks with skip connections.
result Skip connections help control the connectedness of sub-level sets, avoiding shallow local minima.
DFS dynamically decides bitwidths for layers to balance accuracy and efficiency.
problem Balancing model accuracy and inference speed for deep networks.
method Dynamic Fractional Skipping (DFS) framework that assigns bitwidths to layers for input-adaptive inference.
result DFS achieves superior tradeoff between computational cost and model accuracy.
This paper examines how skip connections prevent rank collapse in sequence models.
problem Rank collapse in sequence models, leading to reduced expressivity and training instabilities.
method Analytical and ablation studies of lambda-skip connections in SSMs.
result A sufficient condition to prevent rank collapse across various architectures.
New insights into GNN optimization reveal skip connections and depth accelerate training.
problem Understanding and optimizing the training of Graph Neural Networks (GNNs).
method Analysis of gradient dynamics in linearized GNNs and empirical validation.
result GNNs are implicitly accelerated by skip connections, more depth, and good label distribution during training.
High order splitting schemes with complex timesteps are applied to Kolmogorov backward equations stemming from stochastic differential equations in Stratonovich form. In the setting of weighted spaces, the necessary analyticity of the split semigroups can be easily proved. A numerical example from interest rate theory,…
Corrected CBOW performs similarly to Skip-gram.
problem CBOW embeddings underperform Skip-gram embeddings in word2vec.
method Fixed a bug in CBOW gradient update to improve performance.
result Corrected CBOW embeddings are competitive with Skip-gram on various tasks.
Diffusion-GAN uses diffusion to improve GAN training stability and realism.
problem Stability and realism issues in training GANs.
method Diffusion-GAN employs a forward diffusion chain to generate Gaussian-mixture distributed instance noise, with adaptive diffusion process and timestep-dependent discriminator.
result Diffusion-GAN produces more realistic images with higher stability and data efficiency.
This work analyzes how bottleneck layers and skip connections affect linear denoising autoencoders' generalization.
problem Understanding the generalization of linear denoising autoencoders in overparameterized regimes.
method Analyzes two-layer linear denoising autoencoders with a bottleneck layer and skip connection, deriving test risk formulas.
result Bottleneck layers introduce an additional complexity measure, while skip connections can mitigate variance.
Novel model improves clinical risk prediction by transferring knowledge between tasks over time.
problem Negative transfer in multi-task learning for clinical risk prediction.
method Temporal Probabilistic Asymmetric Multi-Task Learning (TPAMTL).
result Significantly outperforms various deep learning models for time-series prediction.
PAM models generate dependent random distributions across groups with overlapping clusters.
problem Generating dependent random distributions across multiple groups.
method Atom skipping in an infinite mixture model.
result Interpretable posterior inference of cluster exclusivity and sharing.
DenseNets introduce concatenation-type skip connections that achieve state-of-the-art accuracy in several computer vision tasks. In this paper, we reveal that the topology of the concatenation-type skip connections is closely related to the gradient propagation which, in turn, enables a predictable behavior of DNNs' te…
DARTS- improves robustness by factoring out skip connections' advantage.
problem Performance collapse in DARTS architecture search.
method Factor out skip connections' advantage with an auxiliary skip connection.
result Significantly improved robustness across various datasets.
Variational autoencoders learn distributions of high-dimensional data. They model data with a deep latent-variable model and then fit the model by maximizing a lower bound of the log marginal likelihood. VAEs can capture complex distributions, but they can also suffer from an issue known as "latent variable collapse," …
We show that the skip-gram formulation of word2vec trained with negative sampling is equivalent to a weighted logistic PCA. This connection allows us to better understand the objective, compare it to other word embedding methods, and extend it to higher dimensional models.
Paper explores variable skipping to speed up range density estimation.
problem Efficiently estimating range densities over high-dimensional data.
method Variable skipping technique to accelerate range density estimation.
result 10-100x efficiency improvements in challenging high-quantile error metrics.
Theory explains why DARTS favors deep architectures over shallow ones.
problem DARTS selects architectures with dominated skip connections, leading to performance degradation.
method Theoretical analysis of operations' effects on network optimization; introduces sparse binary gates and path-depth-wise regularization.
result Theoretical proof that architectures with more skip connections converge faster.
Generates low-dimensional node vectors for graphs with privacy while preserving structural preferences.
problem Publishing graph node vectors can leak sensitive individual information.
method SE-PrivGEmb, a skip-gram based technique with a unified noise tolerance mechanism and negative sampling probabilities.
result Our method outperforms existing methods in structural equivalence and link prediction tasks.
Simplified Bayesian neural networks reduce model complexity and improve interpretability.
problem Over-parameterization and interpretability issues in deep learning models.
method Input-skip Latent Binary Bayesian Neural Networks (LBBNNs) that allow covariates to skip layers or be excluded.
result Significant reduction in model complexity (over 99%) with minimal loss in accuracy and uncertainty.
Study on autoencoder denoising in high dimensions.
problem Denoising data from Gaussian mixtures.
method Two-layer non-linear autoencoder with skip connection in high-dimensional limit.
result Closed-form expressions for denoising mean-squared test error.
This paper explains GCNs using NTKs and improves their performance.
problem GCNs' performance degrades with depth, and skip connections marginally improve it.
method Derive NTKs for GCNs, validate with simulations, propose NTK as a surrogate model.
result Suitable normalisation can prevent drastic performance drop with depth.
Regularized SB process speeds up generative modeling.
problem Slow sampling and training times in SB-based models.
method Regularization terms to reduce timesteps and training time.
result Faster sampling speed for generative modeling.
Backpropagation is the workhorse of deep learning, however, several other biologically-motivated learning rules have been introduced, such as random feedback alignment and difference target propagation. None of these methods have produced a competitive performance against backpropagation. In this paper, we show that bi…
In this work, we extend the SchNet architecture by using weighted skip connections to assemble the final representation. This enables us to study the relative importance of each interaction block for property prediction. We demonstrate on both the QM9 and MD17 dataset that their relative weighting depends strongly on t…
A new method learns dynamic graph representations from time-varying data.
problem Learning dynamic graph representations from time-varying data.
method Higher-order skip-gram with negative sampling (HOSGNS) for tensor factorization.
result HOSGNS outperforms state-of-the-art methods in downstream tasks.
ResiliNet improves distributed neural network inference resilience.
problem Physical node failures in distributed neural networks cause performance drops.
method Skip hyperconnection and failout technique.
result ResiliNet provides inference resiliency for distributed neural networks.
Proposes SNML for selecting word2vec Skip-gram dimensionality.
problem Selecting optimal dimensionality for word2vec Skip-gram models.
method Information criteria (AIC, BIC, SNML) applied to SG and SG Negative Sampling models.
result SNML outperforms AIC and BIC, selecting closer optimal dimensionality.
A long-standing obstacle to progress in deep learning is the problem of vanishing and exploding gradients. Although, the problem has largely been overcome via carefully constructed initializations and batch normalization, architectures incorporating skip-connections such as highway and resnets perform much better than …
This work shows dimension regularization can replace skip-gram negative sampling for graph embeddings, improving efficiency and performance.
problem Efficiently enforcing dissimilarity among node embeddings in graph learning.
method Dimension regularization as an alternative to skip-gram negative sampling.
result Dimension regularization is a more efficient approach to enforcing dissimilarity in graph embeddings.
XAI identifies key time steps for early crop classification.
problem Early crop classification with high accuracy and timeliness.
method Training a baseline model with LRP to identify important time steps.
result Identified a 21st April 2019 to 9th August 2019 timeframe with 0.75% accuracy loss.
We introduce a method which enables a recurrent dynamics model to be temporally abstract. Our approach, which we call Adaptive Skip Intervals (ASI), is based on the observation that in many sequential prediction tasks, the exact time at which events occur is irrelevant to the underlying objective. Moreover, in many sit…
Eliciting semantic similarity between concepts in the biomedical domain remains a challenging task. Recent approaches founded on embedding vectors have gained in popularity as they risen to efficiently capture semantic relationships The underlying idea is that two words that have close meaning gather similar contexts. …
Dealing with uncertainty in Bayesian Network structures using maximum a posteriori (MAP) estimation or Bayesian Model Averaging (BMA) is often intractable due to the superexponential number of possible directed, acyclic graphs. When the prior is decomposable, two classes of graphs where efficient learning can take plac…
DVNet efficiently segments large neurovascular datasets using skip connections.
problem Challenges in segmenting large neurovascular datasets from high-throughput microscopy data.
method A fully-convolutional, deep, and densely-connected encoder-decoder network with skip connections.
result DVNet achieves superior performance in semantic segmentation of neurovascular datasets.
New RL method learns to skip states in linearly qπ-realizable MDPs, simplifying to linear MDPs.
problem Online RL in episodic MDPs with linearly qπ-realizable action-values. method Derives a novel algorithm that learns to skip states and applies a linear MDP algorithm.
result First polynomial-sample-complexity online RL algorithm for linearly qπ-realizable MDPs. A new adaptive splitting method improves accuracy for Cox-Ingersoll-Ross model.
problem Improving numerical solution accuracy for Cox-Ingersoll-Ross model.
method Adaptive splitting method over deterministic and random meshes, with uniform moment bound and strong error results.
result Uniform moment bound and strong error results of order 1/4 in L1 and L2 for κθ>σ^2, and order 1 for large noise.
Memory-augmented neural networks consisting of a neural controller and an external memory have shown potentials in long-term sequential learning. Current RAM-like memory models maintain memory accessing every timesteps, thus they do not effectively leverage the short-term memory held in the controller. We hypothesize t…
CAggNet improves medical image segmentation by fusing coarse and fine features.
problem Medical image segmentation accuracy and efficiency.
method Crossing Aggregation Network with nested skip connections and weighted aggregation.
result CAggNet achieves more accurate and efficient segmentation compared to existing methods.
Recent work has demonstrated that embeddings of tree-like graphs in hyperbolic space surpass their Euclidean counterparts in performance by a large margin. Inspired by these results and scale-free structure in the word co-occurrence graph, we present an algorithm for learning word embeddings in hyperbolic space from fr…
Since the invention of word2vec, the skip-gram model has significantly advanced the research of network embedding, such as the recent emergence of the DeepWalk, LINE, PTE, and node2vec approaches. In this work, we show that all of the aforementioned models with negative sampling can be unified into the matrix factoriza…
Spacetimeformer learns spatiotemporal relationships from data alone.
problem Forecasting multivariate time series with distinct spatial relationships.
method Transformers with dynamic graph connections learning interactions between space, time, and value.
result Competitive results on various time series prediction benchmarks.