This paper analyzes shallow ViTs, providing sample complexity and SGD behavior insights.
problem Theoretical understanding of shallow ViTs, especially their sample complexity and SGD behavior.
method Data model with label-relevant and label-irrelevant tokens, theoretical analysis of shallow ViT training.
result Characterization of sample complexity for zero generalization error in shallow ViTs.
Paper compresses ViTs models by 60% with minimal accuracy loss.
problem Memory constraints in ViTs models.
method Activation-aware low-rank tensor approximations.
result 60% reduction in model size with <1% accuracy loss.
PACE explains ViTs by modeling patch-level concept distributions, surpassing existing methods.
problem Lack of trustworthy post-hoc explanations for Vision Transformers (ViTs)
method Variational Bayesian explanation framework (PACE)
result PACE surpasses state-of-the-art methods in meeting desiderata for ViT explanations.
VITS improves Thompson Sampling for contextual bandits with efficient approximate inference.
problem Intractable posterior sampling in traditional Thompson Sampling for contextual bandits.
method VITS uses Gaussian Variational Inference for efficient posterior approximations.
result VITS achieves sub-linear regret bound similar to traditional TS.
Study proposes a statistical test for Vision Transformer's attention mechanisms.
problem ViT's attention mechanisms may focus on irrelevant regions, leading to unreliable evidence.
method Selective inference framework to quantify statistical significance of attentions as p-values.
result Proposed method enables reliable quantification of false positive detection probability of attentions.
Vision Transformers show different internal representations compared to CNNs.
problem Understanding how Vision Transformers solve image classification tasks.
method Comparative analysis of ViT and CNN architectures on image classification benchmarks.
result ViT has more uniform representations across all layers, while CNNs have more varied representations.
Mobile V-MoEs scale down ViTs for resource-constrained vision tasks.
problem Scaling down Vision Transformers for resource-constrained applications.
method Sparse Mixture-of-Experts (MoEs) applied to entire images, with a stable training procedure.
result Mobile V-MoEs achieve better performance-efficiency trade-offs than dense ViTs.
B-cos transformers explain Vision Transformers' decisions.
problem Lack of holistic explanations for transformer outputs.
method Formulate each component as dynamic linear, allowing a single linear transform for summarization.
result Bcos-ViTs are highly interpretable and competitive on ImageNet.
Theoretical analysis of vision transformers' performance with MAE and CL objectives.
problem Understanding the distinct representations learned by vision transformers with MAE and CL objectives.
method Modeling visual data distribution and analyzing ViTs training dynamics with gradient descent.
result ViTs trained with MAE objectives learn both global and local features, while CL-trained ViTs favor global features.
CDAM improves attention maps for ViTs, making them more class-sensitive.
problem Existing attention maps in ViTs lack class sensitivity.
method Class-discriminative attention maps (CDAM) that scale attention scores by class relevance.
result CDAM provides more class-sensitive explanations than existing methods.
Paper analyzes why deeper layers of ViTs perform worse on out-of-distribution tasks.
problem Performance degradation of intermediate layers in ViTs under distribution shift.
method Extensive linear probing experiments across various benchmarks and fine-grained analysis of transformer modules.
result Probing feedforward network activations yields best performance under significant distribution shift.
Paper proposes ADC framework to reduce ViT SL training communication overhead.
problem Reducing communication overhead in ViT SL training.
method Two parallel compression strategies: class-agnostic merging and token discarding.
result Significantly reduces communication overhead without sacrificing accuracy.
Regularizes attention scores in vision transformers using bootstrapping.
problem Noisy and diffused attention maps in ViT limit interpretability.
method Statistical learning techniques, bootstrapping of attention scores.
result Improves shrinkage and sparsity of attention scores.
BoC probe assesses neural network confidence coherence, revealing architecture-specific uncertainty.
problem Poor calibration and OOD detection in neural networks.
method Bag-of-Coins (BoC) probe compares softmax confidence to pairwise dominance probabilities.
result BoC reveals clear ID/OOD separation for some architectures but not others.
ConViT combines CNN and ViT strengths, improving image classification.
problem Combining the strengths of CNNs and ViTs while avoiding their limitations.
method Introducing GPSA, a form of positional self-attention with a soft convolutional inductive bias.
result ConViT outperforms DeiT on ImageNet while offering improved sample efficiency.
Improved DNN calibration without sacrificing accuracy.
problem Poor calibration of over-parametrized DNNs in safety-critical applications.
method Decoupling feature extraction and classification layers, and applying Gaussian priors.
result Significant improvement in model calibration with minimal training cost.
BayesDLL offers a PyTorch library for Bayesian deep learning with large models.
problem Bayesian inference for large-scale deep networks.
method Variational inference, MC-dropout, stochastic-gradient MCMC, Laplace approximation.
result BayesDLL can handle Vision Transformers and pre-trained model weights as priors.
Shallow neural networks can represent polynomials efficiently.
problem Representing polynomials using shallow neural networks.
method Using shallow neural networks of width 2(R+d)d to represent d-variate polynomials of degree R. result Derives minimax optimal convergence rate for shallow networks to unknown univariate regression functions.
Wide and shallow networks approximate convex functions well.
problem Understanding why wide and shallow neural networks perform well.
method Analyzing the epigraph of the input-output map of shallow and wide neural networks.
result The epigraph of the input-output map approximates a convex function.
EpiMer merges models by solving Fréchet mean on a Riemannian manifold.
problem Integrating knowledge from multiple models without retraining.
method EpiMer casts model merging as solving the Fréchet mean on a Riemannian manifold, restricting computation to a low-rank subspace.
result EpiMer outperforms flat-geometry methods on image classification tasks.
Optimal rates for shallow ReLU networks in nonparametric regression.
problem Approximating smooth and non-smooth functions with shallow ReLU networks.
method Analysis of shallow ReLUk neural networks, using variation norms and deep learning theory. result Optimal approximation rates for shallow ReLU networks in nonparametric regression.
Deep learning models predict epileptic seizures with high accuracy.
problem Predicting epileptic seizures for better patient care.
method Developed Temporal Multi-Channel Transformer (TMC-T) and Vision Transformer (TMC-ViT) models for EEG signals.
result TMC-ViT model outperformed CNN in seizure prediction.
This paper reports the performances of shallow word-level convolutional neural networks (CNN), our earlier work (2015), on the eight datasets with relatively large training data that were used for testing the very deep character-level CNN in Conneau et al. (2016). Our findings are as follows. The shallow word-level CNN…
Deep ReLU networks can be simplified to a three-layer model.
problem Understanding the behavior of deep neural networks.
method Constructive proof and algorithm to transform deep networks into shallow ones.
result Deep ReLU networks can be represented by a simpler three-layer structure.
Deep neural networks can learn smooth functions without parameters.
problem Learning smooth functions from shallow ReLU neural networks.
method Using over-parameterized shallow ReLU neural networks with norm constraints.
result Least squares estimators based on shallow neural networks are minimax optimal.
CHOOSE enhances shallow Transformers for wireless symbol detection.
problem Improving wireless symbol detection with shallow Transformers.
method Introducing autoregressive latent reasoning steps within hidden space.
result Lightweight Transformers achieve comparable performance to deep models.
New bounds for shallow neural networks with deterministic parameters.
problem Developing generalisation bounds for shallow neural networks.
method PAC-Bayesian theory applied to shallow neural networks with deterministic parameters.
result Empirical non-vacuous bounds for shallow neural networks trained with vanilla SGD.
Sharp lower bounds on shallow neural networks' approximation rates are derived.
problem The efficiency of shallow neural networks in approximating functions.
method Lower bounding the L2-metric entropy and Kolmogorov n-widths of the convex hull of neural network basis functions. result Sharp lower bounds on the approximation rates for shallow neural networks are provided.
Study shows shallow ReLU networks struggle with high-dimensional Lipschitz functions.
problem Expressing high-dimensional Lipschitz functions with shallow ReLU networks.
method Established lower bounds on shallow network complexity for polynomial approximation.
result Shallow ReLU networks suffer from the curse of dimensionality for Lipschitz functions.
Proves existence of optimal shallow neural networks with ReLU activation.
problem Proving the existence of optimal shallow feedforward networks with ReLU activation.
method Proves existence of global minima in the loss landscape for continuous target functions using shallow feedforward neural networks with ReLU activation.
result Existence of global minima in the loss landscape for shallow feedforward networks with ReLU activation.
Gradient-trained shallow networks can generalize well but are vulnerable to small-radius adversarial attacks.
problem Adversarial robustness of gradient-trained shallow networks.
method Analysis of neuron alignment and polynomial ReLU activation.
result Gradient-trained shallow networks with polynomial ReLU activation are robust to small-radius adversarial attacks.
In this paper, we introduce transformations of deep rectifier networks, enabling the conversion of deep rectifier networks into shallow rectifier networks. We subsequently prove that any rectifier net of any depth can be represented by a maximum of a number of functions that can be realized by a shallow network with a …
LeJEPA provides a scalable, theory-driven approach to self-supervised learning.
problem Lack of practical guidance and theory in JEPAs.
method Identified optimal Gaussian distribution and introduced SIGReg objective.
result LeJEPA achieves state-of-the-art performance with minimal hyperparameters and heuristics.
A new algorithm for learning shallow neural networks with infinite width.
problem Learning shallow over-parameterized neural networks.
method Sinkhorn proximal algorithm approximating mean field learning dynamics.
result The algorithm performs gradient descent of the free energy associated with the risk functional.
Study shows limits on deep and shallow neural networks for approximating compact sets.
problem Understanding the limitations of deep and shallow neural networks in approximating compact sets.
method Proved Carl's type inequalities for approximation error, using Lipschitz widths.
result Lower bounds on approximation error for neural network outputs.
Paper studies shallow ReLU networks' approximation rates for Hölder functions.
problem Understanding shallow ReLU networks' efficiency in approximating Hölder functions.
method Analyzes rates of uniform approximation by ReLU shallow neural networks with m hidden neurons. result Shows ReLU shallow neural networks can uniformly approximate Hölder functions with rates close to optimal.
This paper focuses on a comparative evaluation of the most common and modern methods for text classification, including the recent deep learning strategies and ensemble methods. The study is motivated by a challenging real data problem, characterized by high-dimensional and extremely sparse data, deriving from incoming…
Deep ReLU networks approximate as well as shallow ones in kernel regimes.
problem Understanding the limitations of kernel methods for deep ReLU networks.
method Characterizing eigenvalue decays of kernels derived from deep ReLU networks.
result Deep ReLU networks and shallow two-layer networks have equivalent approximation properties in kernel regimes.
Deep networks without non-linearities are equivalent to shallow ones.
problem Training deep orthogonal linear networks with no non-linearity.
method Riemannian gradient descent and gradient descent on factorization.
result Training deep overparametrized networks is equivalent to shallow ones.
It is well established that neural networks with deep architectures perform better than shallow networks for many tasks in machine learning. In statistical physics, while there has been recent interest in representing physical data with generative modelling, the focus has been on shallow neural networks. A natural ques…
Unified theorem for deep and shallow joint-equivariant machines.
problem Universal approximation of joint-equivariant machines.
method Constructive universal approximation theorem based on ridgelet transform.
result Unified approximation of deep and shallow networks.
New method adapts neural networks without losing prior knowledge.
problem Understanding and enabling flexible adaptation of neural networks.
method Differential geometry framework, functionally invariant paths (FIP).
result Achieves comparable state-of-the-art performance on continual learning and sparsification tasks.
New method for efficient proximal mapping of 1-path-norm in shallow networks.
problem Efficiently handling the 1-path-norm of shallow neural networks.
method Closed-form proximal operator for efficient computation and upper bound on Lipschitz constant.
result Proximal mapping allows robust training against adversarial perturbations.
In this work, we present a comparison of a shallow and a deep learning architecture for the automated segmentation of white matter lesions in MR images of multiple sclerosis patients. In particular, we train and test both methods on early stage disease patients, to verify their performance in challenging conditions, mo…
Study shows overparameterization helps shallow neural networks recover signals in high dimensions.
problem Signal recovery in shallow neural networks with overparameterization.
method Gradient flow on population risk, Gaussian distribution assumption, high-dimensional limit analysis.
result Minimal overparameterization is sufficient for strong recovery of signals.
Deep networks learn hierarchical functions more efficiently than shallow ones.
problem Understanding the advantage of deep neural networks over shallow models.
method Analytical study of learning dynamics and generalization performance of deep networks compared to shallow ones.
result Deep networks reduce effective dimensionality, enabling learning with fewer samples.
Combines machine learning and convex limiting for accurate subgrid flux modeling in shallow-water equations.
problem Accurate subgrid flux modeling in shallow-water equations.
method Machine learning and flux limiting for property-preserving subgrid scale modeling.
result The proposed method produces meaningful closures even in untrained scenarios.
This paper considers the power of deep neural networks (deep nets for short) in realizing data features. Based on refined covering number estimates, we find that, to realize some complex data features, deep nets can improve the performances of shallow neural networks (shallow nets for short) without requiring additiona…