Due to the success of residual networks (resnets) and related architectures, shortcut connections have quickly become standard tools for building convolutional neural networks. The explanations in the literature for the apparent effectiveness of shortcuts are varied and often contradictory. We hypothesize that shortcut…
On-device research index
arXiv research
A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
168,742 papers · 148 categories
Trend · papers per month
48 results for “shortcut connections”
Shortcut connections in ResNet help avoid local optima, leading to efficient training.
problem Understanding why shortcut connections in ResNet lead to efficient training.
method Two-layer non-overlapping convolutional ResNet, gradient descent with proper normalization.
result Gradient descent avoids spurious local optima, converging to a global optimum.
In established network architectures, shortcut connections are often used to take the outputs of earlier layers as additional inputs to later layers. Despite the extraordinary effectiveness of shortcuts, there remain open questions on the mechanism and characteristics. For example, why are shortcuts powerful? Why do sh…
Sensor Fusion using Backward Shortcut Connections for Sleep Apnea Detection in Multi-Modal Datacs.LG
Improved sleep apnea detection using sensor fusion and backward shortcut connections.
problem Untreated sleep apnea leads to severe health consequences; automated detection is needed.
method Late sensor fusion using backward shortcut connections to improve deep learning models.
result Significant improvement in predictive performance over single sensor methods.
Demystifying ResNetcs.NE
The Residual Network (ResNet), proposed in He et al. (2015), utilized shortcut connections to significantly reduce the difficulty of training, which resulted in great performance boosts in terms of both training and generalization error. It was empirically observed in He et al. (2015) that stacking more layers of resid…
We propose a neural architecture search (NAS) algorithm, Petridish, to iteratively add shortcut connections to existing network layers. The added shortcut connections effectively perform gradient boosting on the augmented layers. The proposed algorithm is motivated by the feature selection algorithm forward stage-wise …
New method trains deep vanilla networks as fast as ResNets without shortcut connections.
problem Training very deep neural networks is challenging.
method Developed a new type of transformation compatible with Leaky ReLUs.
result Validation accuracies with deep vanilla networks are competitive with ResNets and significantly higher.
Improved speech recognition with faster training and inference.
problem Training very deep CNNs for speech recognition is difficult.
method Proposed SNDCNN using SELU activations instead of RELU and shortcut connections/BN.
result Achieved similar or lower WER with faster training and inference.
New method flattens decision boundary by targeting shortcut-aligned axes in disentangled latent space.
problem Shortcut learning in neural networks, leading to poor out-of-distribution generalization.
method Injects targeted anisotropic noise to regularize classifier sensitivity along shortcut-aligned axes.
result Achieves state-of-the-art OOD performance without shortcut labels or conflicting samples.
Study on shortcuts in deep networks, revealing their layer-wise distribution and impact.
problem Understudied impact of shortcuts on feature representations in deep networks.
method Layer-wise localization through counterfactual training on clean and skewed datasets.
result Shortcuts are distributed throughout the network, not localized in specific layers.
Default-ERM shortcut learning persists even without additional information.
problem Default-ERM shortcut learning in perception tasks despite stable feature sufficiency.
method Studied linear perception task; developed margin control (MARG-CTRL) loss functions.
result Margin control mitigates shortcut learning on various tasks.
Regularization methods can overregulate, suppressing causal features.
problem Mitigating shortcuts in models exploiting spurious correlations.
method Analysis of regularization methods and their effects on causal features.
result Regularization can overregulate, suppressing causal features.
Not All Neuro-Symbolic Concepts Are Created Equal: Analysis and Mitigation of Reasoning Shortcutscs.LG
This work identifies and mitigates reasoning shortcuts in Neuro-Symbolic models.
problem Neuro-Symbolic models can achieve high accuracy by using unintended concepts.
method Characterized reasoning shortcuts as unintended optima of the learning objective and identified four key conditions.
result Reasoning shortcuts are difficult to mitigate, casting doubt on NeSy solutions' trustworthiness and interpretability.
A network supporting deep unsupervised learning is presented. The network is an autoencoder with lateral shortcut connections from the encoder to decoder at each level of the hierarchy. The lateral shortcut connections allow the higher levels of the hierarchy to focus on abstract invariant features. While standard auto…
Shortcut Matrix Product States and its applicationscond-mat.stat-mech
Matrix Product States (MPS), also known as Tensor Train (TT) decomposition in mathematics, has been proposed originally for describing an (especially one-dimensional) quantum system, and recently has found applications in various applications such as compressing high-dimensional data, supervised kernel linear classifie…
Transformers learn to use induction heads or shortcuts based on data diversity.
problem How data diversity influences the behavior of transformers.
method Gradient-based training of a single-layer transformer on a minimal task.
result Data diversity steers transformers toward induction heads or shortcuts.
Transformers simulate finite-state automata with fewer layers.
problem How do shallow, non-recurrent Transformers simulate complex computations?
method Hierarchical reparameterization of recurrent dynamics to simulate automata.
result Polynomial-sized, -depth solutions exist and are common.
To have a superior generalization, a deep learning neural network often involves a large size of training sample. With increase of hidden layers in order to increase learning ability, neural network has potential degradation in accuracy. Both could seriously limit applicability of deep learning in some domains particul…
Attention weights may not accurately highlight important parts due to combinatorial shortcuts.
problem Inaccurate interpretation of attention weights in models.
method Theoretical analysis and design of experiments to show combinatorial shortcuts. Proposed two methods to mitigate this issue.
result Proposed methods improve the interpretability of attention mechanisms.
Study reveals DNNs prefer easy-to-learn cues over essential ones in image recognition.
problem DNNs learn easy-to-learn features that aren't essential to the task.
method WCST-ML training setup with shortcut cues on synthetic and face datasets.
result DNNs converge to solutions focusing on preferred cues, leading to flat minima.
Materials discovery is crucial for making scientific advances in many domains. Collections of data from experiments and first-principle computations have spurred interest in applying machine learning methods to create predictive models capable of mapping from composition and crystal structures to materials properties. …
Neurosymbolic predictors fail to model uncertainty under independence assumption.
problem Neurosymbolic predictors' reliance on independence assumption limits their ability to model uncertainty.
method Formal analysis of NeSy predictors under independence assumption.
result Assuming independence among symbolic concepts prevents NeSy predictors from representing uncertainty.
Deep vanilla transformers trained without shortcuts achieve similar performance to standard models.
problem Training deep vanilla transformers without shortcuts and normalizations.
method Parameter initializations, bias matrices, and location-dependent rescaling.
result Deep vanilla transformers can train at similar speeds and performance to standard models.
One pixel modification can make deep models unlearnable.
problem Protecting data from unauthorized training of deep neural networks.
method Perturbing only one pixel in each image to degrade model accuracy.
result Generated One-Pixel Shortcut (OPS) cannot be erased by adversarial training and strong augmentations.
Recently, we proposed to transform the outputs of each hidden neuron in a multi-layer perceptron network to have zero output and zero slope on average, and use separate shortcut connections to model the linear dependencies instead. We continue the work by firstly introducing a third transformation to normalize the scal…
New method discourages models from using bias shortcuts for better generalization.
problem Training models on biased data can lead to poor generalization when bias shifts.
method Train a de-biased representation by encouraging it to differ from biased representations.
result Improved generalization across synthetic and real-world biases.
The Dehn function of SL(n;Z)math.GR
We prove that when n >= 5, the Dehn function of SL(n;Z) is quadratic. The proof involves decomposing a disc in SL(n;R)/SO(n) into triangles of varying sizes. By mapping these triangles into SL(n;Z) and replacing large elementary matrices by "shortcuts," we obtain words of a particular form, and we use combinatorial tec…
Residual networks (ResNet) and weight normalization play an important role in various deep learning applications. However, parameter initialization strategies have not been studied previously for weight normalized networks and, in practice, initialization methods designed for un-normalized networks are used as a proxy.…
A shortcut to sign Incremental Value-at-Risk for risk allocationcond-mat.stat-mech
Approximate Incremental Value-at-Risk formulae provide an easy-to-use preliminary guideline for risk allocation. Both the cases of risk adding and risk pooling are examined and beta-based formulae achieved. Results highlight how much the conditions for adding new risky positions are stronger than those required for ris…
Machine learning reveals hidden features in knot classification.
problem Classifying the topology of closed curves.
method Investigating shortcut methods used by ML for knot classification.
result Developed a dataset and code to remove non-topological features.
Gradient-based framework for optimizing text prompts in diffusion models.
problem Efficiently optimizing prompts in text-to-image diffusion models with large domain space and non-differentiable embeddings.
method Formulated as discrete optimization over language space, designed compact subspaces, and introduced shortcut text gradient.
result Empirically discovered prompts that enhance or destroy image faithfulness.
A result of M. Ledoux is that a complete Riemannian manifold with non negative Ricci curvature satisfying the Euclidean Sobolev inequality is the Euclidean space. We present a shortcut of the proof. We also give a refinement of a result of B-L. Chen et X-P. Zhu about locally conformally flat manifolds with non negative…
This paper proposes a deep speech enhancement method which exploits the high potential of residual connections in a wide neural network architecture, a topology known as Wide Residual Network. This is supported on single dimensional convolutions computed alongside the time domain, which is a powerful approach to proces…
We give a self-contained introduction to the theory of Turaev's shadows as a tool to study 3 and 4-manifolds. The goal of the present paper twofold: on one side it is intended to be a shortcut to a basic use of the theory of shadows, on the other side it gives a sketchy overview of some of the recent results on shadows…
Ozsvath and Szabo have defined a knot concordance invariant tau that bounds the 4-ball genus of a knot. Here we discuss shortcuts to its computation. We include examples of Alexander polynomial one knots for which the invariant is nontrivial, including all iterated untwisted positive doubles of knots with nonnegative T…
Recent empirical results on long-term dependency tasks have shown that neural networks augmented with an external memory can learn the long-term dependency tasks more easily and achieve better generalization than vanilla recurrent neural networks (RNN). We suggest that memory augmented neural networks can reduce the ef…
The paper explains geometric correspondences for homothetic navigation.
problem Understanding geodesic and Jacobi field correspondences in homothetic navigation.
method Providing conceptual explanations and shortcuts to formulas.
result Directly seeing local correspondence between isoparametric functions or hypersurfaces.
In this paper, we study the Hausdorff dimension of the Floyd and Bowditch boundaries of a relatively hyperbolic group, and show that for the Floyd metric and shortcut metrics respectively, they are are both equal to a constant times the growth rate of the group. In the proof, we study a special class of conical points …
Hybrid model combines LSTM and ETS for mid-term electric load forecasting.
problem Mid-term electric load forecasting accuracy.
method Combines LSTM, ETS, and ensemble learning; uses dilated LSTM for long-term relationships.
result High performance and competitiveness compared to classical and machine learning models.
New diagonal knots found with non-torus structure.
problem Identifying knots with diagonal grid diagrams.
method Analysis of knots represented by diagonal grid diagrams.
result All diagonal knots are positive, and a new non-torus example is found.
This work explores non-negative low-rank matrix factorization based on regularized Poisson models (PF or "Poisson factorization" for short) for recommender systems with implicit-feedback data. The properties of Poisson likelihood allow a shortcut for very fast computations over zero-valued inputs, and oftentimes result…
Study highlights robustness issues in healthcare diagnostic models due to distribution shifts.
problem Robustness of diagnostic models in healthcare is compromised by distribution shifts.
method Theoretical analysis and simulation studies to understand and mitigate shortcuts learned by models.
result Ignoring covariates or using invariant learning approaches leads to non-robust predictors.
Towards Understanding Residual and Dilated Dense Neural Networks via Convolutional Sparse Codingcs.LG
New models explain residual and dilated dense neural networks using sparse coding.
problem Lack of theoretical understanding of residual and dilated dense neural networks.
method Proposed Res-CSC and MSD-CSC models, derived mathematical relationships, implemented ISTA.
result Mathematical understanding of residual and dilated dense neural networks.
SGD quickly learns a spurious XOR feature before the signal feature, revealing learning dynamics.
problem Over-reliance on spurious correlations in neural networks trained by SGD.
method Theoretical analysis of SGD on two-layer ReLU networks trained on XOR data.
result SGD learns the spurious feature first and exponentially fast, dominating the signal feature.
Martingale Doppelgänger-Eval benchmarks VLMs on candlestick evidence vs. trend extrapolation
problem Auditing whether VLMs use chart evidence or trend extrapolation
method Proving formal limitations and designing controlled mechanisms
result Identifying regression coefficients for evidence vs. trend
We define and study a discrete process that generalizes the convex-layer decomposition of a planar point set. Our process, which we call "homotopic curve shortening" (HCS), starts with a closed curve (which might self-intersect) in the presence of a set of point obstacles, and evolves in discrete…
SurvNAM explains survival model predictions using machine learning.
problem Explaining predictions of black-box survival models.
method SurvNAM is a modified Neural Additive Model (NAM) trained with a specific loss function based on Cox model and GAM.
result SurvNAM allows local and global explanation of survival model predictions.
New geometric regularizers improve deep learning generalization.
problem Improving deep learning models' ability to generalize to unseen data.
method Using Bregman divergence loss and bounded spectral products, we propose a novel geometric regularizer to enhance model generalization.
result Good generalization can be achieved by designing deep models with specific structural regularizers.