Wide & Deep combines linear and deep learning for better app recommendations.
problem Sparse user-item interactions lead to less relevant recommendations.
method Jointly trains wide linear models and deep neural networks.
result Wide & Deep significantly increases app acquisitions compared to wide-only and deep-only models.
Wide deep neural networks are easy to optimize without constraints.
problem Optimizing wide deep neural networks.
method Analysis of optimization landscapes and empirical-risk minimization.
result Wide neural networks have no confined points, making optimization easier.
GCNIII combines Wide & Deep for better node classification.
problem Issues with graph convolutional networks in node classification tasks.
method Proposes GCNIII framework integrating Wide & Deep architecture and three techniques.
result Demonstrates improved performance in various node classification tasks.
Deep and wide networks are shown to be equivalent in terms of their capability.
problem The relationship between the width and depth of neural networks.
method Formulated transforms to map networks, used polynomial representations.
result Deep and wide networks are quasi-equivalent with an arbitrarily small error.
Single wide layer followed by a pyramidal structure ensures global convergence in deep networks.
problem Ensuring global convergence in deep neural networks with limited width constraints.
method Proves that a single wide layer followed by a pyramidal structure guarantees global convergence for over-parameterized networks.
result Single wide layer of width N suffices for global convergence in deep networks with constant-width remaining layers. Stable processes emerge as limits of deep neural networks with symmetric stable distributions.
problem Understanding the behavior of deep neural networks as they become infinitely wide.
method Analyzing fully connected feed-forward deep neural networks with symmetric stable distributions and showing the limit as a stable process.
result The infinite wide limit of the network is a stable process with multivariate stable distributions.
Wide neural networks have non-attracting local minima.
problem Understanding the impact of suboptimal local minima in deep and wide neural networks.
method Construction of non-attracting local minima and saddle points in wide neural networks.
result Wide neural networks have non-attracting local minima, even though they are not negatively impacted by suboptimal local optima.
Study of deep Stable neural networks with various activation functions.
problem Characterizing the infinitely wide limits of deep Stable neural networks.
method Investigation of large-width properties of deep Stable NNs with a generalized central limit theorem for heavy tails.
result Extension of characterization to a broader class of activation functions, including sub-linear, asymptotically linear, and super-linear functions.
Proposes a deep latent factor model for better recommendation systems.
problem Improving collaborative filtering in recommendation systems.
method Introduces a deeper latent factor model using deep learning.
result Significantly outperforms state-of-the-art techniques in experiments.
Gradient descent finds global min in wide deep linear networks.
problem Optimizing deep linear neural networks with limited width.
method Proved convergence rate for gradient descent with wide layers.
result Gradient descent converges linearly to global min in wide layers.
Wide CNNs with shared weights and max pooling have linearly independent features and can achieve zero training error.
problem Understanding the optimization landscape and expressiveness of deep CNNs.
method Analysis of loss landscape and expressiveness of practical deep CNNs with shared weights and max pooling layers.
result Wide CNNs can achieve zero training error and have a well-behaved loss surface with almost no bad local minima.
Deep networks with a wide layer ensure sublevel set connectivity.
problem Ensuring connectivity of sublevel sets in deep learning.
method Analyzing the connectivity of sublevel sets in deep neural networks with a specific layer width.
result A single wide layer of width N+1 suffices to prove connectivity of sublevel sets. Wide and Deep GNN learns from distributed graphs and retrain online.
problem Decentralized graph support changes over time, causing mismatch between training and testing graphs.
method Wide and Deep GNN architecture with distributed online learning.
result Convergence guarantees for online retraining of the wide part of the GNN.
Wide deep neural networks with Gaussian weights approximate Gaussian processes closely.
problem Understanding the approximation of deep neural networks with Gaussian weights to Gaussian processes.
method Established novel rates for the Gaussian approximation of random deep neural networks with Gaussian parameters and Lipschitz activation functions in the wide limit.
result The distance between the network output and the Gaussian approximation scales inversely with the width of the network.
The paper links deep neural networks to Gaussian processes, showing convergence under certain conditions.
problem Understanding theoretical properties of deep neural networks.
method Study of random, wide, fully connected feedforward networks and Gaussian processes with recursive kernels.
result As network width increases, random functions converge to Gaussian processes under broad conditions.
Deep neural networks' Jacobian spectrum becomes well-conditioned with orthogonal weights.
problem Understanding and handling the Jacobian spectrum of deep neural networks.
method Applying free probability theory to show almost sure asymptotic freeness of Jacobians in the wide limit.
result Layer-wise Jacobians of deep neural networks with orthogonal weights are almost surely asymptotically free.
Deep linear networks with arbitrary loss have all local minima as global minima.
problem Finding optimal solutions in deep linear networks with arbitrary convex losses.
method Provided a short and elementary proof for all local minima being global minima under specific conditions.
result All local minima are global minima for deep linear networks with certain layer widths.
Bayesian inference for wide neural networks using Edgeworth expansion.
problem Analyzing the non-Gaussian behavior of wide neural networks in Bayesian inference.
method Proposed a non-Gaussian distribution using multivariate Edgeworth expansion for finite-width neural networks.
result Derived non-Gaussian posterior distribution in Bayesian regression tasks.
The paper examines how deep linear neural networks behave as they become infinitely wide.
problem Understanding the behavior of deep linear neural networks as they approach infinite width.
method Analyzes the infinite-width limit of deep linear neural networks, proving convergence to deterministic models and providing precise laws for random weights.
result The training dynamics of deep linear neural networks converge to those of a deterministic model, and the weights' behavior is precisely described.
Proposes efficient training method for deep thin networks.
problem Deploying deep learning models with accuracy and compactness.
method Three-stage method: widen, warm up, fine tune.
result Deep thin networks trained with method outperform standard deep networks.
Gradient descent proves global convergence for deep networks with a single wide layer.
problem Proving global convergence of gradient descent for deep ReLU networks.
method Simplified proof using a single wide layer, leveraging ReLU's Lipschitz property.
result Gradient descent converges globally for networks with a single wide layer.
TR-Nets compress deep networks by 11x for LeNet-5 and 243x for Wide ResNet.
problem Large neural networks require excessive memory and computation.
method Tensor Ring factorization to compress fully connected and convolutional layers.
result TR-Nets can compress LeNet-5 by 11x and Wide ResNet by 243x with minimal accuracy loss.
Study on deep and wide echo state networks for forecasting complex time series.
problem Performance analysis of deep reservoir computing models.
method Investigates the impact of partitioning neurons and parallel pathways on forecasting accuracy.
result Wide and deep networks outperform shallow models in forecasting multiscale spatiotemporal data.
Deep learning predicts personalized drug responses from medical data.
problem Predicting personalized drug responses from medical data.
method Deep learning algorithms applied to large datasets.
result Deep learning improves prediction of personalized drug responses.
Deep RL algorithms generally generalize better than specialized schemes.
problem Deep RL agents fail to generalize beyond their training environments.
method Presented a benchmark and experimental protocol to systematically assess generalization in deep RL.
result Vanilla deep RL algorithms outperform specialized generalization schemes.
Deep and wide ReLU networks learn data-dependent features even in the lazy training regime.
problem Understanding the behavior of neural networks with finite depth and width.
method Analyzing the mean and variance of the neural tangent kernel (NTK) in a randomly initialized ReLU network.
result The NTK has a non-trivial evolution during training, with the mean of its first SGD update being exponential in the ratio of depth to width.
Memory split advantage: thinner networks outperform a single wide network.
problem Optimizing deep learning models with limited memory.
method Investigated training a single wide network vs. an ensemble of thinner networks with the same total number of parameters.
result An ensemble of several thinner networks outperforms a single wide network for large memory budgets.
Neural Episodic Control learns faster than other reinforcement learning agents.
problem Inefficient reinforcement learning methods requiring vast amounts of data.
method Uses a semi-tabular value function representation with a buffer of past experiences.
result Significantly faster learning across various environments.
Study on infinitely-wide CNNs and their adaptability to function spatial scales.
problem Understanding how CNNs efficiently learn high-dimensional functions and their adaptability to function spatial scales.
method Study infinitely-wide deep CNNs in the kernel regime, characterizing their spectrum and using generalisation bounds to prove adaptability.
result Deep CNNs adapt to the spatial scale of the target function, with error decay controlled by the effective dimensionality of function subsets.
Wide neural networks converge to Gaussian processes, improving generalization.
problem Understanding the generalization of wide neural networks, especially deep equilibrium models.
method Investigation of deep equilibrium models (DEQs) with infinite-depth layers, focusing on their convergence to Gaussian processes as width and depth approach infinity.
result Wide DEQs converge to Gaussian processes, maintaining generalization performance.
Wide neural networks with narrow bottlenecks behave like deep Gaussian processes.
problem Understanding the behavior of neural networks with narrow layers in the wide limit.
method Analyzing the wide limit of BNNs with narrow bottlenecks, showing they behave like a composition of GPs.
result Wide neural networks with narrow bottlenecks form a composition of GPs, termed a bottleneck NNGP.
Wide neural networks' last hidden layers split into groups of redundant neurons.
problem Understanding why wide neural networks generalize well despite overfitting.
method Analyzed the last hidden layer representations of various convolutional neural networks.
result Wide hidden layers split into groups of redundant neurons, which help generalize.
DeepFM combines deep learning and factorization machines for CTR prediction.
problem Maximizing CTR in recommender systems through sophisticated feature interactions.
method End-to-end learning framework combining DNN and PNN with shared feature input.
result DeepFM-D outperforms existing models, achieving over 10% improvement in CTR.
New algorithm tackles saddle points in deep learning.
problem Saddle points proliferation in nonconvex optimization.
method Hessian-free optimization, addressing computational complexity and saddle point issue.
result First step towards applying Newton's method to nonconvex optimization.
Wide and deep neural network predicts Alzheimer's progression from shape and clinical data.
problem Predicting Alzheimer's disease progression from shape and clinical data.
method Fused anatomical shape and tabular clinical data in a neural network, employing survival analysis loss.
result The model outperforms shape and clinical models individually.
This paper analyzes deep Stable neural networks, showing convergence rates under different growth settings.
problem Analyzing the behavior of deep Stable neural networks as width increases.
method Large-width asymptotic analysis and convergence rates for fully connected feed-forward deep Stable NNs.
result The rescaled deep Stable NN converges weakly to a Stable SP under joint growth, with sup-norm convergence rates established.
WideBNet learns inverse scattering from wide-band data efficiently and stably.
problem Learning the inverse scattering map from wide-band scattering data.
method Combines butterfly factorization, FFT, and deep learning.
result WideBNet requires fewer training points and has stable training dynamics.
This paper introduces an efficient method for optimizing deep learning hyperparameters.
problem The high dependency of deep learning algorithms on hyper-parameters.
method Orthogonal Array Tuning Method (OATM) for deep learning hyper-parameter tuning.
result The proposed OATM method significantly saves tuning time compared to state-of-the-art methods.
Proposes SSM to improve CTR prediction with deep neural networks.
problem Improving CTR prediction with deep neural networks.
method Designs an orthogonal base convolution and pooling model to learn multi-scale base semantic representation.
result Demonstrates superior performance in CTR prediction.
Empirical study finds deep learning assumptions often incorrect.
problem Widespread assumptions about neural networks are often incorrect.
method Empirical evaluation of neural network assumptions using various techniques.
result Proves existence of suboptimal local minima and demonstrates practical implications.
Paper proposes knockoff-based methods to simplify deep neural networks by controlling false discovery rates.
problem High-dimensional deep neural networks with many irrelevant parameters and inputs.
method Knockoff methods combined with regularized neural networks for variable screening.
result Proposed algorithms show satisfactory performance in controlling false discovery rates.
Deep learning predicts hospital readmission risk from notes.
problem Predicting hospital-wide readmission risk from doctors' notes.
method Deep learning and natural language processing on unstructured text.
result Model predicts readmission with c-statistic 0.70.
Article presents QR and LQ decomposition algorithms for various matrix sizes and ranks.
problem Solving least squares problems in machine learning and computer vision.
method Developed novel matrix backpropagation algorithms for QR and LQ decompositions of different matrix sizes and ranks.
result Numerical stability and computational efficiency of the proposed methods.
The paper proves neural networks' consistency and optimal convergence rates for various function classes.
problem Proving neural networks' consistency and optimal convergence rates for diverse function classes.
method Analyzes wide and deep ReLU neural networks trained on logistic loss and Kolmogorov-Donoho optimal function classes.
result Proves universal consistency and minimax optimal convergence rates for neural networks.
Deep learning detects genetic interactions in type 2 diabetes.
problem Detecting genetic interactions in complex diseases like type 2 diabetes.
method Stacked Autoencoder for non-linear epistatic interactions.
result Deep learning can uncover missing heritability in complex diseases.
Deep Gaussian Processes are reinterpreted as deep trigonometric networks for tractable inference.
problem Challenging inference in DGPs due to intractable marginalization in latent function space.
method Viewing DGPs as deep trigonometric networks with Bochner's theorem, and using the wide limit with a bottleneck to translate DGPs into deep trigonometric networks.
result The weight space view yields the same effective covariance functions as obtained in function space, and varying prior distributions over network parameters is equivalent to employing different kernels.
Enhanced volatility model using LSTM and realized volatility.
problem Volatility modeling in financial markets.
method Combining deep learning (LSTM) and realized volatility measures in a Bayesian framework.
result Superior predictive performance compared to benchmark models.
Deep linear ResNets converge globally with certain transformations.
problem Global convergence of training deep linear ResNets.
method Gradient descent and stochastic gradient descent for training L-hidden-layer linear ResNets. result GD and SGD can converge to global minimum for deep linear ResNets with specific transformations.