Two-layer CNNs can overfit well if initialized correctly.
problem Understanding the conditions for benign overfitting in over-parameterized CNNs.
method Extending analysis to fully trainable two-layer CNNs, examining initialization scaling effects.
result Initialization scaling of the output layer is crucial; large scales lead to fixed output behavior, small scales to complex interactions.
Two-layer CNNs can overfit without harm under certain conditions.
problem Understanding when and how overfitting occurs in neural networks.
method Theoretical analysis of a two-layer CNN trained by gradient descent.
result A sharp phase transition between benign and harmful overfitting based on signal-to-noise ratio.
DP-GD improves CNN training accuracy with privacy, especially with low signal-to-noise ratios.
problem Privacy-preserving training of neural networks with crowdsourced data.
method Differentially private gradient descent (DP-GD) algorithm applied to two-layer CNNs.
result DP-GD can achieve superior generalization performance compared to GD, especially with low signal-to-noise ratios.
SimCLR pre-training improves CNN performance with fewer labels.
problem Learning with fewer labeled data.
method SimCLR contrastive learning method combined with supervised fine-tuning.
result SimCLR pre-training with supervised fine-tuning achieves almost optimal test loss with fewer labeled data.
Convolutional neural networks (CNNs) have achieved remarkable performance in various fields, particularly in the domain of computer vision. However, why this architecture works well remains to be a mystery. In this work we move a small step toward understanding the success of CNNs by investigating the learning dynamics…
Theory captures feature learning effects in finite CNNs.
problem Feature learning in finite deep neural networks.
method Derive a self-consistent Gaussian Process theory.
result Good agreement with experiments and sharp transition between regimes.
Polynomial-time convex optimization for CNNs with ReLU activations.
problem Training Convolutional Neural Networks (CNNs) with ReLU activations.
method Developed a convex analytic framework using semi-infinite duality to formulate equivalent convex optimization problems for CNN architectures.
result Proved that two-layer CNNs can be globally optimized via an ℓ2 norm regularized convex program. In this paper, we analyze efficacy of the fast gradient sign method (FGSM) and the Carlini-Wagner's L2 (CW-L2) attack. We prove that, within a certain regime, the untargeted FGSM can fool any convolutional neural nets (CNNs) with ReLU activation; the targeted FGSM can mislead any CNNs with ReLU activation to classify a…
Batch normalization biases linear models towards uniform margins, improving performance in binary classification.
problem Understanding the implicit bias of batch normalization in linear models and neural networks.
method Analyzing gradient descent convergence on linear models and two-layer CNNs with batch normalization.
result Gradient descent with batch normalization in linear models converges to a uniform margin classifier with an exponential convergence rate.
In this paper, we present a new location fingerprinting database comprised of Wi-Fi received signal strength (RSS) and geomagnetic field intensity measured with multiple devices at a multi-floor building in Xi'an Jiatong-Liverpool University, Suzhou, China. We also provide preliminary results of localization and trajec…
Paper analyzes weak-to-strong generalization in CNNs, identifying data-scarce and data-abundant regimes.
problem Weak-to-strong generalization in CNNs trained on weak models.
method Formal analysis of gradient descent dynamics in data-scarce and data-abundant regimes.
result Identifies two regimes and distinct mechanisms of generalization in each.
Convolutional Neural Networks (CNNs) have emerged as highly successful tools for image generation, recovery, and restoration. A major contributing factor to this success is that convolutional networks impose strong prior assumptions about natural images. A surprising experiment that highlights this architectural bias t…
We analyze why some models resist unlearning using linear stability theory.
problem Understanding and predicting when machine learning models resist unlearning.
method Linear stability theory applied to machine learning models, focusing on data coherence and optimization dynamics.
result Data coherence and signal-to-noise ratio (SNR) influence unlearning resistance; lower SNR makes unlearning easier.
Gradient descent with logistic loss can make two-layer networks interpolate binary classification data.
problem Training two-layer networks for binary classification.
method Gradient descent with logistic loss applied to two-layer networks.
result Gradient descent can drive training loss to zero under certain conditions.
Two-layer neural networks can approximate functions with fractal singularities.
problem Characterizing functions that can be represented by infinitely wide two-layer neural networks.
method Representation formulas and pointwise properties analysis.
result Functions with fractal or curved singularities cannot be represented by two-layer networks with finite path-norm.
Analytical method finds deeper optima in two-layer ReLU networks.
problem Training two-layer ReLU networks with analytical methods.
method Analytically finding critical points of the loss function for one layer while keeping the other fixed.
result Significantly smaller training loss values on real datasets compared to gradient descent methods.
Gradient descent converges to minimum Bayes risk for two-layer ReLU networks in mean field regime.
problem Training two-layer ReLU networks using gradient descent in the mean field regime.
method Describes a condition for convergence to minimum Bayes risk, extending previous results to ReLU-activated networks.
result The condition for convergence does not depend on initialization and concerns weak convergence of network realization.
Three-layer networks learn more complex features than two-layer networks.
problem Understanding feature learning in deep neural networks.
method Analysis of three-layer neural networks trained with gradient descent.
result Three-layer networks can learn functions that two-layer networks cannot.
We study cascades on a two-layer multiplex network, with asymmetric feedback that depends on the coupling strength between the layers. Based on an analytical branching process approximation, we calculate the systemic risk measured by the final fraction of failed nodes on a reference layer. The results are compared with…
Two-layer neural networks learn efficiently using kernel methods in mean-field analysis.
problem Feature learning ability of two-layer neural networks in the mean-field regime.
method Mean-field analysis through kernel methods, focusing on dynamics of the first layer's kernel.
result Two-layer neural networks can learn a union of multiple reproducing kernel Hilbert spaces more efficiently than kernel methods.
Two-layer networks trained on low-dimensional subspaces are vulnerable to adversarial examples.
problem Vulnerability of two-layer neural networks to adversarial examples on low-dimensional subspaces.
method Analysis of gradient behavior and effect of initialization scale and regularization.
result Decreasing initialization scale or adding L2 regularization can improve robustness to adversarial perturbations orthogonal to the data.
Two-layer neural networks must be robust, even with arbitrary weights.
problem Proving the robustness of two-layer neural networks with arbitrary weights.
method Developed a new function-space covering method to prove the robustness law, replacing parameter-space covering.
result Proved the conjectured law for two-layer networks with arbitrary real weights, biases, and affine skip connections.
This work shows linear convergence for two-layer neural networks in mean-field regime.
problem Optimizing two-layer neural networks in the mean-field regime.
method Mean-field analysis and continuous-time noisy gradient descent.
result Establishes linear convergence rate for two-layer neural networks.
We focus on estimating \emph{a priori} generalization error of two-layer ReLU neural networks (NNs) trained by mean squared error, which only depends on initial parameters and the target function, through the following research line. We first estimate \emph{a priori} generalization error of finite-width two-layer ReLU …
We study the limits and methods of training two-layer autoencoders.
problem Understanding the limits and methods of training two-layer autoencoders.
method Focus on non-linear two-layer autoencoders trained in the proportional regime, using gradient methods.
result Gradient methods achieve the minimizers of the population risk and reveal the structure of the features.
Study on SGD for overparameterized neural networks, focusing on convergence rates.
problem Understanding convergence rates of SGD in overparameterized two-layer neural networks.
method Combines NTK approximation with RKHS analysis to explore SGD dynamics.
result Established sharp convergence rates for SGD in overparameterized two-layer neural networks.
Study shows infoGAN's generalization error bound for two-layer networks.
problem Understanding generalization error in infoGAN for two-layer neural networks.
method Analyzes the difference between empirical and population objective functions, derives Rademacher complexity bounds.
result Derives error bound for infoGAN's generalization error in a two-layer network.
This paper presents a phase diagram for two-layer neural networks under different initialization scales.
problem Understanding the behavior of neural networks under varying scales of initialization.
method Analysis of a phase diagram for two-layer neural networks.
result Condensation of weight vectors on isolated orientations during training.
PHP connects to ReLU neural networks for scalable Bayesian inference.
problem Scalability and Bayesian inference in two-layer ReLU neural networks.
method PHP with Gaussian prior, decomposition propositions, annealed sequential Monte Carlo.
result PHP provides an alternative scalable representation for two-layer ReLU neural networks.
Adam's generalization performance is improved by batch size and weight decay in neural networks.
problem Understanding how batch size and weight decay affect Adam's generalization in neural networks.
method Theoretical analysis of two-layer over-parameterized CNNs on image data.
result Adam's mini-batch variants can achieve near-zero test error, unlike full-batch Adam.
This paper explains double descent in linear neural networks, identifying new factors.
problem Understanding double descent in linear neural networks.
method Gradient flow derivation and necessary conditions for double descent.
result Singular values of input-output covariance matrix are important for double descent in two-layer models.
Two-layer networks learn hard GLMs with SGD in high dimensions.
problem Learning hard generalized linear models with SGD in high-dimensional settings.
method Reduction of SGD dynamics to a stochastic process in lower dimensions, focusing on the role of stochasticity.
result Overparameterization enhances convergence by a constant factor, suggesting minimal role of stochasticity.
Study of two-layer NNs under Gaussian mixtures data, proving polynomial models equivalent to neural networks.
problem Training and generalization performance of two-layer NNs under structured Gaussian mixture data.
method Asymptotic analysis of two-layer NNs after one gradient descent step under Gaussian mixture data assumption.
result High-order polynomial models equivalent to nonlinear neural networks under certain conditions.
It has been shown that gradient descent can yield the zero training loss in the over-parametrized regime (the width of the neural networks is much larger than the number of data points). In this work, combining the ideas of some existing works, we investigate the gradient descent method for training two-layer neural ne…
Recently, several studies have proven the global convergence and generalization abilities of the gradient descent method for two-layer ReLU networks. Most studies especially focused on the regression problems with the squared loss function, except for a few, and the importance of the positivity of the neural tangent ke…
Signal models based on sparsity, low-rank and other properties have been exploited for image reconstruction from limited and corrupted data in medical imaging and other computational imaging applications. In particular, sparsifying transform models have shown promise in various applications, and offer numerous advantag…
ARFF reduces spectral bias in SGD-trained neural networks.
problem Spectral bias in two-layer neural networks.
method Comparison of SGD and ARFF on spectral bias and robustness.
result ARFF yields a closer to zero spectral bias compared to SGD.
Analyzes SGD dynamics in two-layer networks, bridging different regimes.
problem Understanding SGD dynamics in high-dimensional and mean-field settings.
method Rigorous analysis via deterministic low-dimensional description of sufficient statistics.
result Infinite-width dynamics remains close to a low-dimensional subspace.
Two-layer ReLU networks can overfit without harm, study finds.
problem Understanding when and how two-layer ReLU networks can overfit without harming generalization.
method Established algorithm-dependent risk bounds for two-layer ReLU convolutional neural networks with label-flipping noise.
result Gradient descent-trained ReLU networks can achieve near-zero training loss and Bayes optimal test risk.
Two-layer networks favor simple features, especially in complex datasets.
problem Simplicity bias in neural networks over-reliing on simple features.
method Characterization of two-layer neural networks with small weights and gradient flow.
result Features learned in middle training stages are more useful for out-of-distribution transfer.
New estimates for the population risk are established for two-layer neural networks. These estimates are nearly optimal in the sense that the error rates scale in the same way as the Monte Carlo error rates. They are equally effective in the over-parametrized regime when the network size is much larger than the size of…
Gradient descent struggles with high-dimensional data fitting.
problem Gradient descent struggles with high-dimensional data fitting.
method Gradient descent training of a two-layer neural network on empirical or population risk.
result Gradient descent training may not decrease population risk faster than t−4/(d−2) under mean field scaling. Develops a two-layer model to design mortgage assistance products.
problem Designing effective mortgage assistance products to improve household resilience.
method Two-layer approach: simulation and optimization.
result Shows how the approach can design and evaluate mortgage assistance products.
SGD converges globally to logistic loss minima for two-layer nets.
problem Global convergence of SGD for logistic loss on two-layer neural nets.
method Demonstrates existence of Frobenius norm regularized logistic loss functions as Villani functions, proving convergence and exponential rate.
result SGD converges globally to the global minima of appropriately regularized logistic empirical risk of depth 2 nets.
Two-layer neural networks can overfit without increasing risk when data is noisy.
problem Understanding why neural networks can overfit without increasing risk in noisy data.
method Combining bias and variance analysis in a high-dimensional setting.
result The excess learning risk of the interpolator decays under mild conditions.
New approximative kernels improve PDE-G-CNNs for geometric deep learning.
problem Inaccurate approximations of exact kernels in PDE-G-CNNs.
method Developed new approximative kernels that work regardless of spatial anisotropy.
result New kernels provide better error estimates and maintain reflectional symmetries.
In recent years, deep learning poses a deep technical revolution in almost every field and attracts great attentions from industry and academia. Especially, the convolutional neural network (CNN), one representative model of deep learning, achieves great successes in computer vision and natural language processing. How…
This paper explores how MoE layers improve deep learning performance.
problem Understanding the Mixture-of-Experts (MoE) layer in deep learning.
method Formal study of MoE layer's effectiveness and mechanism.
result MoE layer improves performance by leveraging cluster structure and non-linearity.