Paper finds wide minima are better for generalization and proposes a new learning rate schedule.
problem The challenge of finding optimal learning rates for model training.
method The paper introduces a new hypothesis about the density of wide minima and designs an explore-exploit learning rate schedule.
result The explore-exploit learning rate schedule improves model performance and reduces training time.
New learning rules for wide neural networks without backpropagation.
problem Training wide neural networks efficiently and without backpropagation.
method Input-weight alignment driven by gradient descent in the NTK regime.
result Biologically-motivated learning rules equivalent to backpropagation in wide networks.
Wide deep neural networks are easy to optimize without constraints.
problem Optimizing wide deep neural networks.
method Analysis of optimization landscapes and empirical-risk minimization.
result Wide neural networks have no confined points, making optimization easier.
Wide Learning automates feature engineering in healthcare datasets.
problem Time-consuming and expert-demanding feature engineering in ML.
method Proposes a Wide Learning architecture for automated feature recommendation.
result Wide Learning reaches 94.38% accuracy in PhysioNet Challenge 2016 dataset.
Generalized linear models with nonlinear feature transformations are widely used for large-scale regression and classification problems with sparse inputs. Memorization of feature interactions through a wide set of cross-product feature transformations are effective and interpretable, while generalization requires more…
Wide neural networks can benefit from multi-task learning in their infinite-width limit.
problem The generalization behavior of wide neural networks in multi-task learning settings.
method Optimizing wide ReLU neural networks with L2-regularization promotes multi-task learning in the infinite-width limit.
result An exact quantitative characterization of multi-task learning in the infinite-width limit of wide ReLU neural networks.
Wide neural networks converge linearly to zero loss with feature learning.
problem Optimizing wide neural networks with feature learning guarantees.
method Gradient flow analysis for wide shallow and multi-layer NNs.
result Training loss converges linearly to zero for wide NNs under GF, demonstrating feature learning and better generalization.
Wide neural networks can learn complex functions like gravitational force law.
problem Learning complex functions like gravitational force law with neural networks.
method Extending theoretical bounds to analytic functions on the sphere using SGD and ReLU networks.
result Wide ReLU networks can learn analytic functions efficiently with proportional number of samples.
Wide neural networks simplify to linear models under gradient descent.
problem Understanding the training dynamics of deep neural networks.
method Analyzing wide neural networks in the infinite width limit and showing they evolve as linear models.
result Gradient-based training of wide neural networks results in predictions from a Gaussian process with a specific kernel.
Wide and Deep GNN learns from distributed graphs and retrain online.
problem Decentralized graph support changes over time, causing mismatch between training and testing graphs.
method Wide and Deep GNN architecture with distributed online learning.
result Convergence guarantees for online retraining of the wide part of the GNN.
Study shows training input-to-hidden weights reduces generalization error in wide neural networks.
problem Understanding the mathematical properties of hierarchical neural networks in the infinite-width limit.
method Examined a three-layer neural network with a large number of hidden units, comparing training input-to-hidden weights to keeping them fixed.
result Training input-to-hidden weights yields a smaller generalization error and avoids singularities.
WideBNet learns inverse scattering from wide-band data efficiently and stably.
problem Learning the inverse scattering map from wide-band scattering data.
method Combines butterfly factorization, FFT, and deep learning.
result WideBNet requires fewer training points and has stable training dynamics.
The paper explains the richness scale of wide neural networks.
problem Understanding the behavior of overparameterized neural networks.
method Nonrigorous derivation and empirical evidence.
result Wide neural networks exhibit a richness scale from lazy kernel behavior to feature learning.
Wide CNNs with shared weights and max pooling have linearly independent features and can achieve zero training error.
problem Understanding the optimization landscape and expressiveness of deep CNNs.
method Analysis of loss landscape and expressiveness of practical deep CNNs with shared weights and max pooling layers.
result Wide CNNs can achieve zero training error and have a well-behaved loss surface with almost no bad local minima.
Wide hypernetworks don't guarantee convergence under gradient descent.
problem Theoretical guarantees for wide hypernetworks in over-parameterized settings.
method Analyzing infinitely wide hypernetworks and their convergence properties.
result Infinitely wide hypernetworks do not guarantee convergence to a global minimum under gradient descent.
Stochastic Gradient Descent finds wide but shallow minima due to undersampling, akin to energy-entropy competition.
problem The empirical effectiveness of Stochastic Gradient Descent in machine learning.
method Deriving a correspondence between parameter inference and free energy minimisation in statistical physics, where the degree of undersampling plays the role of temperature.
result Stochasticity in Stochastic Gradient Descent biases it towards wide minima, explaining its empirical effectiveness.
GNNs learn graph representations, with new theory on their power and limitations.
problem Understanding the capabilities and limitations of GNNs.
method Theoretical analysis of GNNs, focusing on approximation and learning properties.
result New insights into the representation, generalization, and extrapolation of GNNs.
Proposes a deep latent factor model for better recommendation systems.
problem Improving collaborative filtering in recommendation systems.
method Introduces a deeper latent factor model using deep learning.
result Significantly outperforms state-of-the-art techniques in experiments.
GCNIII combines Wide & Deep for better node classification.
problem Issues with graph convolutional networks in node classification tasks.
method Proposes GCNIII framework integrating Wide & Deep architecture and three techniques.
result Demonstrates improved performance in various node classification tasks.
Study improves kernel quadrature for infinitely wide models with faster approximation and estimation rates.
problem Efficiently approximating and estimating expectations in infinitely wide models.
method Developed general kernel quadrature (GKQ) for parameter distributions, achieving faster rates.
result Achieved a fast approximation rate of O ( e − p ) O(e^{-p}) O ( e − p ) and a fast estimation rate of O ~ ( 1 / n ) \widetilde{O}(1/n) O ( 1/ n ) . Wide Boosting improves GB's performance on multivariate output tasks.
problem Lack of flexibility in fitting probabilistic multi-dimensional outputs.
method Inserts matrix multiplication between GB output and loss function.
result Wide Boosting outperforms Gradient Boosting on multivariate output tasks.
This chapter introduces quaternion machine learning for 3D rotations.
problem Lack of quaternion machine learning for 3D rotations.
method Augmented statistics, widely linear models, quaternion calculus, mean square estimation.
result Foundation for quaternion machine learning.
Wide networks learn from adversarial perturbations effectively.
problem Understanding why adversarial examples deceive classifiers and transfer between models.
method Assumed wide two-layer networks, proved with theoretical analysis.
result Adversarial perturbations contain class-specific features for networks to generalize.
Deep and wide ReLU networks learn data-dependent features even in the lazy training regime.
problem Understanding the behavior of neural networks with finite depth and width.
method Analyzing the mean and variance of the neural tangent kernel (NTK) in a randomly initialized ReLU network.
result The NTK has a non-trivial evolution during training, with the mean of its first SGD update being exponential in the ratio of depth to width.
Single wide layer followed by a pyramidal structure ensures global convergence in deep networks.
problem Ensuring global convergence in deep neural networks with limited width constraints.
method Proves that a single wide layer followed by a pyramidal structure guarantees global convergence for over-parameterized networks.
result Single wide layer of width N N N suffices for global convergence in deep networks with constant-width remaining layers. Neural Episodic Control learns faster than other reinforcement learning agents.
problem Inefficient reinforcement learning methods requiring vast amounts of data.
method Uses a semi-tabular value function representation with a buffer of past experiences.
result Significantly faster learning across various environments.
An approach for learning ancestral causal relationships in high dimensions, validated on human genome-wide data.
problem Learning ancestral causal relationships in high-dimensional biological data.
method Supervised learning approach with discrete indicators treated as labels, scalable to large problems.
result The approach is highly effective and scalable to the human genome-wide setting, robust to perturbations of input information.
This work analyzes when contrastive models are close to PCA or kernel methods.
problem Understanding when contrastive models are equivalent to kernel methods or PCA.
method Analyzing the training dynamics of two-layer contrastive models with non-linear activation.
result Wide contrastive models with cosine similarity based losses are close to PCA.
Deep networks with a wide layer ensure sublevel set connectivity.
problem Ensuring connectivity of sublevel sets in deep learning.
method Analyzing the connectivity of sublevel sets in deep neural networks with a specific layer width.
result A single wide layer of width N + 1 N+1 N + 1 suffices to prove connectivity of sublevel sets. Machine learning risks in finance pricing and hedging
problem Understanding and managing risks in financial models
method Analyzing machine learning applications in finance, focusing on pricing and hedging of financial options
result Identifies various sources of risk and potential mitigation strategies
New research on limits of transfer learning, proving key selection and dependence requirements.
problem Insufficient theoretical foundation for transfer learning.
method Proved novel results on transfer learning, emphasizing selection of information and dependence between domains.
result Upper bound on improvement possible with transfer learning, highlighting the need for careful selection.
Wide neural networks ensure disconnected decision regions.
problem Ensuring neural networks produce disconnected decision regions.
method Analyzing conditions for connectedness of decision regions in neural networks with specific activation functions.
result For leaky ReLU and pyramidal structures, wide hidden layers are necessary for disconnected decision regions.
Wide neural networks with asymmetrical node scaling converge globally and learn features.
problem Global convergence and feature learning in over-parameterised shallow networks.
method Gradient-based optimisation of wide, shallow neural networks with asymmetrical node scaling.
result Gradient flow and gradient descent converge to a global minimum and learn features, unlike in the NTK parameterisation.
New method clusters stationary stochastic processes using covariance-based dissimilarity.
problem Clustering wide-sense stationary ergodic stochastic processes.
method Covariance-based dissimilarity measure with consistent algorithms for offline and online clustering.
result Asymptotically consistent algorithms for efficient clustering.
Demon improves neural network training with a decaying momentum approach.
problem Improving neural network training efficiency and robustness.
method Proposes a decaying momentum ( extsc{Demon}) rule for neural network optimization.
result Demon achieves the highest number of Top-1 and Top-3 finishes across various settings and architectures.
Deep learning detects genetic interactions in type 2 diabetes.
problem Detecting genetic interactions in complex diseases like type 2 diabetes.
method Stacked Autoencoder for non-linear epistatic interactions.
result Deep learning can uncover missing heritability in complex diseases.
Deep neural networks' Jacobian spectrum becomes well-conditioned with orthogonal weights.
problem Understanding and handling the Jacobian spectrum of deep neural networks.
method Applying free probability theory to show almost sure asymptotic freeness of Jacobians in the wide limit.
result Layer-wise Jacobians of deep neural networks with orthogonal weights are almost surely asymptotically free.
Novel mutual information bound improves statistical inference rates.
problem Improving statistical inference rates in Bayesian nonparametrics.
method Introduces a novel mutual information bound.
result Improved contraction rates for fractional posteriors.
Infinitely wide neural nets perform well on small datasets.
problem Performing well on small datasets with limited training samples.
method Using Neural Tangent Kernels (NTKs) for kernel regression.
result NTK SVM outperforms Random Forests and Convolutional NTK on CIFAR-10 with 10-640 training samples.
Bayesian inference for wide neural networks using Edgeworth expansion.
problem Analyzing the non-Gaussian behavior of wide neural networks in Bayesian inference.
method Proposed a non-Gaussian distribution using multivariate Edgeworth expansion for finite-width neural networks.
result Derived non-Gaussian posterior distribution in Bayesian regression tasks.
Study explores algorithmic collusion in repeated games using various learning dynamics.
problem Understanding algorithmic collusion in repeated games with different learning dynamics.
method Examines Q Q Q -learning, gradient learning, and other dynamics in a general repeated game setting. result Characterizes the set of payoff vectors achievable by these dynamics, revealing possibilities for collusion.
Deep learning predicts personalized drug responses from medical data.
problem Predicting personalized drug responses from medical data.
method Deep learning algorithms applied to large datasets.
result Deep learning improves prediction of personalized drug responses.
New MD algorithms using Tempesta logarithms for machine learning.
problem Optimization in machine learning with tailored hyperparameters.
method Developed Mirror Descent algorithms using Tempesta multi-parametric logarithms.
result Wide and flexible family of Mirror Descent and mirror-less updates.
Sparse coding improves reinforcement learning representations.
problem Improving representation learning in reinforcement learning.
method Developed a supervised sparse coding objective for policy evaluation.
result Sparse coding representations outperform tile-coding representations.
Kernel methods are widespread in machine learning; however, they are limited by the quadratic complexity of the construction, application, and storage of kernel matrices. Low-rank matrix approximation algorithms are widely used to address this problem and reduce the arithmetic and storage cost. However, we observed tha…
Study on symmetries in wide neural networks' dynamics without bias.
problem Understanding symmetries in the dynamics of wide two-layer neural networks.
method Analyzing symmetries in gradient flow on population risk for infinitely wide networks.
result Symmetries can simplify the dynamics of predictors and reduce the dimensionality of the problem.
Deep and wide networks are shown to be equivalent in terms of their capability.
problem The relationship between the width and depth of neural networks.
method Formulated transforms to map networks, used polynomial representations.
result Deep and wide networks are quasi-equivalent with an arbitrarily small error.
New algorithm learns Gaussian mixtures privately with optimal sample complexity.
problem Learning parameters of Gaussian mixtures under differential privacy constraints.
method Differentially private algorithm based on Achlioptas and McSherry's approach.
result Sample complexity matches non-private algorithm up to lower order terms.