Study of two-layer NNs under Gaussian mixtures data, proving polynomial models equivalent to neural networks.
problem Training and generalization performance of two-layer NNs under structured Gaussian mixture data.
method Asymptotic analysis of two-layer NNs after one gradient descent step under Gaussian mixture data assumption.
result High-order polynomial models equivalent to nonlinear neural networks under certain conditions.
Enhances apparel attribute recognition with a two-layer ensemble method.
problem Improving accuracy in apparel attributes classification using deep neural networks.
method Proposes a two-layer mixture framework combining bagging and boosting for ensemble learning.
result The proposed method outperforms individual models and ensemble methods.
New GM layers improve neural network performance.
problem Improving neural network performance.
method Employing Gaussian mixture models and Wasserstein gradient flows.
result GM layers achieve comparable performance to two-layer networks.
Paper finds conditions for benign overfitting in neural networks.
problem Benign overfitting in leaky ReLU two-layer neural networks.
method Established directional convergence and classification error bounds.
result Benign overfitting occurs with high probability on mixture data.
Stochastic gradient descent converges to universal limits in high dimensions.
problem Statistical tasks in high dimensions with specific data projections.
method Stochastic gradient descent applied to mixture distributions, proving universality of limits.
result The ODE limits are universal for mixtures of arbitrary product distributions.
In this paper, the use of the Generalized Beta Mixture (GBM) and Horseshoe distributions as priors in the Bayesian Compressive Sensing framework is proposed. The distributions are considered in a two-layer hierarchical model, making the corresponding inference problem amenable to Expectation Maximization (EM). We prese…
The paper studies learning dynamics in two-layer neural networks.
problem Learning dynamics and time scales in two-layer neural networks.
method Gradient flow dynamics of a wide two-layer neural network in high-dimension, with data following a single-index model.
result The learning dynamics exhibit separation of timescales and intermittency.
Study shows different training methods yield varying prediction risks for two-layers neural networks.
problem Understanding prediction risks in two-layers neural networks under different training regimes.
method Examined three training regimes: random features, neural tangent, and fully trained neural network.
result There is a significant gap in prediction risk between the random features and neural tangent regimes when the number of neurons is smaller than the ambient dimension.
Study on autoencoder denoising in high dimensions.
problem Denoising data from Gaussian mixtures.
method Two-layer non-linear autoencoder with skip connection in high-dimensional limit.
result Closed-form expressions for denoising mean-squared test error.
Study improves flow-based model training from few samples.
problem Training flow-based models from limited data.
method Sharp analysis of two-layer autoencoder with finite sample complexity.
result Generative flow approximates target density with rate Θ_n(1/n).
Many different methods to train deep generative models have been introduced in the past. In this paper, we propose to extend the variational auto-encoder (VAE) framework with a new type of prior which we call "Variational Mixture of Posteriors" prior, or VampPrior for short. The VampPrior consists of a mixture distribu…
SGD learns a simple network for multi-class classification from mixtures of well-separated distributions.
problem Learning overparameterized neural networks for multi-class classification.
method Stochastic Gradient Descent (SGD) on structured data.
result SGD learns a network with small generalization error from mixtures of well-separated distributions.
This paper explores how MoE layers improve deep learning performance.
problem Understanding the Mixture-of-Experts (MoE) layer in deep learning.
method Formal study of MoE layer's effectiveness and mechanism.
result MoE layer improves performance by leveraging cluster structure and non-linearity.
Neural networks outperform kernel methods in classifying high-dimensional Gaussian mixtures.
problem Classifying high-dimensional Gaussian mixtures using kernel methods and neural networks.
method Theoretical analysis and derivation of learning dynamics for 2LNN and comparison with kernel methods.
result 2LNN can achieve near-optimal performance on high-dimensional Gaussian mixture classification tasks, surpassing kernel methods.
Gradient ascent method successfully removes specific data points from neural networks without retraining.
problem Addressing privacy and ethical concerns by removing specific data points from trained models.
method Gradient ascent approach to unlearning, leveraging the implicit bias of gradient descent towards margin maximization conditions.
result Gradient ascent method can successfully unlearn specific data points from two-layer ReLU neural networks without retraining.
This paper explores tradeoffs between standard and adversarial risks in distributionally adversarial training.
problem Understanding the impact of adversarial training on standard risk and adversarial risk.
method Study of distributionally adversarial training with different learning settings and models.
result Derives Pareto-optimal tradeoff curves between standard and adversarial risks.
AdaPT-GMM improves multiple testing power with covariates.
problem Powerful and robust multiple testing with covariates.
method Covariate-assisted Gaussian mixture model with adaptive thresholding.
result AdaPT-GMM delivers high power in various scenarios.
Gradient-free method improves predictive accuracy for probabilistic models.
problem Balancing computational efficiency and robust predictive performance in deep learning.
method CAVI-CMN, a gradient-free variational method for conditional mixture networks.
result CAVI-CMN achieves competitive and often superior predictive accuracy compared to MLE with backpropagation.
New algorithm reduces dynamic regret for MDPs with unknown transition and adversarial rewards.
problem Episodic linear mixture MDPs with unknown transition and adversarial rewards.
method Combines occupancy-measure-based global optimization and policy-based variance-aware value-targeted regression.
result Achieves near-optimal dynamic regret of O ~ ( d H 3 K + H K ( H + P ˉ K ) ) \widetilde{\mathcal{O}}(d \sqrt{H^3 K} + \sqrt{HK(H + \bar{P}_K)}) O ( d H 3 K + H K ( H + P ˉ K ) ) . The paper analyzes convergence rates of softmax gating in MoE models.
problem The effectiveness and scalability of machine learning models using MoE.
method Convergence analysis of parameter and expert estimation under MoE with softmax gating and its variants.
result Theoretical results show polynomially many data points are needed for strong identifiability conditions, while exponential points are required for linear experts.
Gradient descent with logistic loss can make two-layer networks interpolate binary classification data.
problem Training two-layer networks for binary classification.
method Gradient descent with logistic loss applied to two-layer networks.
result Gradient descent can drive training loss to zero under certain conditions.
Two-layer neural networks can approximate functions with fractal singularities.
problem Characterizing functions that can be represented by infinitely wide two-layer neural networks.
method Representation formulas and pointwise properties analysis.
result Functions with fractal or curved singularities cannot be represented by two-layer networks with finite path-norm.
New model predicts financial connectedness via COVID-19 spread.
problem Predicting financial connectedness during COVID-19 spread.
method Semiparametric matrix regression model with Bayesian hierarchical mixture prior.
result Model captures heterogeneity in network responses to risk factors.
Gradient descent achieves fast convergence for approximating functions with two-layer neural networks.
problem Approximating continuous functions with two-layer neural networks.
method Gradient descent combined with generic chaining technique from probability theory.
result Gradient descent yields an exponential convergence rate for two-layer neural networks without needing a large width relative to the number of data points.
Gradient descent learns two-layer networks well for classification problems.
problem Understanding the performance of gradient descent on classification problems.
method Refined convergence analysis of gradient descent for two-layer networks with smooth activations.
result Gradient descent can learn less over-parameterized networks for classification problems.
Analytical method finds deeper optima in two-layer ReLU networks.
problem Training two-layer ReLU networks with analytical methods.
method Analytically finding critical points of the loss function for one layer while keeping the other fixed.
result Significantly smaller training loss values on real datasets compared to gradient descent methods.
We provide a series of results for unsupervised learning with autoencoders. Specifically, we study shallow two-layer autoencoder architectures with shared weights. We focus on three generative models for data that are common in statistical machine learning: (i) the mixture-of-gaussians model, (ii) the sparse coding mod…
New algorithm learns two-layer neural networks under symmetric input distributions.
problem Learning two-layer neural networks with symmetric inputs.
method Method-of-moments framework, spectral algorithms.
result Guaranteed to recover parameters of ground-truth network under certain conditions.
Gradient descent converges to minimum Bayes risk for two-layer ReLU networks in mean field regime.
problem Training two-layer ReLU networks using gradient descent in the mean field regime.
method Describes a condition for convergence to minimum Bayes risk, extending previous results to ReLU-activated networks.
result The condition for convergence does not depend on initialization and concerns weak convergence of network realization.
Estimates generalization error for two-layer ReLU NNs through minimum norm solutions.
problem Estimating generalization error for two-layer ReLU NNs trained by mean squared error.
method Uses minimum norm solutions and Neural Tangent Kernel (NTK) regime to derive generalization error bounds.
result Derives an a priori generalization error bound for two-layer ReLU NNs without requiring exponentially large number of neurons.
Three-layer networks learn more complex features than two-layer networks.
problem Understanding feature learning in deep neural networks.
method Analysis of three-layer neural networks trained with gradient descent.
result Three-layer networks can learn functions that two-layer networks cannot.
We study cascades on a two-layer multiplex network, with asymmetric feedback that depends on the coupling strength between the layers. Based on an analytical branching process approximation, we calculate the systemic risk measured by the final fraction of failed nodes on a reference layer. The results are compared with…
Two-layer neural networks learn efficiently using kernel methods in mean-field analysis.
problem Feature learning ability of two-layer neural networks in the mean-field regime.
method Mean-field analysis through kernel methods, focusing on dynamics of the first layer's kernel.
result Two-layer neural networks can learn a union of multiple reproducing kernel Hilbert spaces more efficiently than kernel methods.
Two-layer neural networks must be robust, even with arbitrary weights.
problem Proving the robustness of two-layer neural networks with arbitrary weights.
method Developed a new function-space covering method to prove the robustness law, replacing parameter-space covering.
result Proved the conjectured law for two-layer networks with arbitrary real weights, biases, and affine skip connections.
Two-layer networks trained on low-dimensional subspaces are vulnerable to adversarial examples.
problem Vulnerability of two-layer neural networks to adversarial examples on low-dimensional subspaces.
method Analysis of gradient behavior and effect of initialization scale and regularization.
result Decreasing initialization scale or adding L2 regularization can improve robustness to adversarial perturbations orthogonal to the data.
This work shows linear convergence for two-layer neural networks in mean-field regime.
problem Optimizing two-layer neural networks in the mean-field regime.
method Mean-field analysis and continuous-time noisy gradient descent.
result Establishes linear convergence rate for two-layer neural networks.
Two-layer model sparsifies image residuals for CT image reconstruction.
problem Image reconstruction from limited and corrupted data.
method Pre-learning a two-layer sparsifying transform model with block coordinate descent optimization.
result Preliminary experiments show the two-layer model improves CT image reconstruction from low-dose measurements.
Study local geometry of mixture models via spectral theory, revealing transitions in training dynamics.
problem Understanding the local geometry of high-dimensional mixture models.
method Spectral theory of Hessian and information matrices, focusing on i.i.d. Gaussian mixtures.
result Exact formulas for limits of spectral distribution and outlier eigenvalues, connecting training dynamics to effective dynamics.
We study the limits and methods of training two-layer autoencoders.
problem Understanding the limits and methods of training two-layer autoencoders.
method Focus on non-linear two-layer autoencoders trained in the proportional regime, using gradient methods.
result Gradient methods achieve the minimizers of the population risk and reveal the structure of the features.
Study on SGD for overparameterized neural networks, focusing on convergence rates.
problem Understanding convergence rates of SGD in overparameterized two-layer neural networks.
method Combines NTK approximation with RKHS analysis to explore SGD dynamics.
result Established sharp convergence rates for SGD in overparameterized two-layer neural networks.
Private training and synthetic data generation using DP clustering.
problem Protecting sensitive data in deep neural networks training.
method Approximate input dataset with privately generated synthetic dataset using DP clustering.
result Simple two-layer neural network achieves SOTA classification accuracy on standard benchmark datasets.
Study shows infoGAN's generalization error bound for two-layer networks.
problem Understanding generalization error in infoGAN for two-layer neural networks.
method Analyzes the difference between empirical and population objective functions, derives Rademacher complexity bounds.
result Derives error bound for infoGAN's generalization error in a two-layer network.
Estimates for neural network risk nearly match Monte Carlo error rates.
problem Understanding the performance of two-layer neural networks.
method Established a priori estimates for the population risk of two-layer neural networks.
result The new estimates are nearly optimal and depend only on function norms, not model parameters.
Generalizes neural tangent kernel analysis for two-layer networks with noise and regularization.
problem Limitations of NTK analysis in deep learning practice.
method Generalized NTK analysis for two-layer neural networks with weight decay and gradient noise.
result Noisy gradient descent with weight decay exhibits 'kernel-like' behavior and converges linearly.
This paper presents a phase diagram for two-layer neural networks under different initialization scales.
problem Understanding the behavior of neural networks under varying scales of initialization.
method Analysis of a phase diagram for two-layer neural networks.
result Condensation of weight vectors on isolated orientations during training.
PHP connects to ReLU neural networks for scalable Bayesian inference.
problem Scalability and Bayesian inference in two-layer ReLU neural networks.
method PHP with Gaussian prior, decomposition propositions, annealed sequential Monte Carlo.
result PHP provides an alternative scalable representation for two-layer ReLU neural networks.
This paper explains double descent in linear neural networks, identifying new factors.
problem Understanding double descent in linear neural networks.
method Gradient flow derivation and necessary conditions for double descent.
result Singular values of input-output covariance matrix are important for double descent in two-layer models.
Two-layer CNNs can overfit well if initialized correctly.
problem Understanding the conditions for benign overfitting in over-parameterized CNNs.
method Extending analysis to fully trainable two-layer CNNs, examining initialization scaling effects.
result Initialization scaling of the output layer is crucial; large scales lead to fixed output behavior, small scales to complex interactions.