Improves deep neural network training and accuracy with adaptive basis approach.
problem Gap between theoretical and practical performance of deep neural networks.
method Adaptive basis viewpoint, novel initializations, hybrid optimizer.
result Dramatic increases in accuracy and convergence rate for various DNN applications.
We present a novel deep Recurrent Neural Network (RNN) model for acoustic modelling in Automatic Speech Recognition (ASR). We term our contribution as a TC-DNN-BLSTM-DNN model, the model combines a Deep Neural Network (DNN) with Time Convolution (TC), followed by a Bidirectional Long Short-Term Memory (BLSTM), and a fi…
Deep neural networks often fit low-frequency functions, contrary to conventional numerical schemes.
problem Understanding the implicit bias of deep neural networks in fitting training data.
method Fourier analysis perspective applied to DNNs training process.
result Deep neural networks tend to fit training data by low-frequency functions, contrary to conventional numerical schemes.
Paper tackles DNN defects by analyzing data flow footprints.
problem Detecting defects in DNN models due to low precision.
method Developed DeepMorph to analyze data flow footprints for root cause identification.
result DeepMorph can guide DNN developers to improve model quality.
RadiX-Net generates diverse sparse neural topologies.
problem Sparse neural networks require more efficient storage and training.
method Deterministically generates RadiX-Nets from sparse topologies.
result RadiX-Nets can train to the same precision as dense DNNs at lower cost.
Deep-Lock secures DNN models with secret keys.
problem Preventing unauthorized usage of DNN models.
method Uses S-Boxes and key scheduling to encrypt DNN parameters.
result Ensures correct functioning only with the correct secret key.
New definition of interpretability for deep neural networks.
problem Vague definition of interpretability for deep neural networks.
method Proposed a new definition of human predictability for interpretability of DNNs.
result Our definition will help to the research of interpretability of DNNs.
The paper develops generalization bounds for deep compound Gaussian neural networks.
problem Developing theoretical guarantees for the performance of deep neural networks.
method Novel generalization error bounds using a compound Gaussian prior and Dudley's integral.
result Theoretical bounds show generalization error scales O ( n ln ( n ) ) \mathcal{O}(n\sqrt{\ln(n)}) O ( n ln ( n ) ) in signal dimension and O ( ( N e t w o r k S i z e ) 3 / 2 ) \mathcal{O}((Network Size)^{3/2}) O (( N e tw or k S i z e ) 3/2 ) in network size. Introduces a probabilistic view of deep learning for better understanding and explaining neural networks.
problem Explaining the behavior and properties of deep neural networks.
method Introduces a probabilistic representation of deep learning, linking neurons, hidden layers, and the whole architecture to Gibbs distributions and Bayesian neural networks.
result Demonstrates the hierarchy and generalization properties of deep learning through a probabilistic lens.
New theory predicts deep neural networks can operate in an extended critical regime without fine-tuning.
problem Understanding the dynamics and computational principles of deep neural networks.
method Combining theories of heavy-tailed random matrices and non-equilibrium statistical physics.
result Deep neural networks can operate in an extended critical regime without fine-tuning parameters.
DRE combines DNN with random feature regression for efficient neural network design.
problem Designing and training deep neural networks (DNN) efficiently and effectively.
method DRE architecture with two-layer neural networks, randomly drawn input and output weights trained with linear ridge regression.
result DRE outperforms state-of-the-art DNN in many data sets with lower computational cost.
EKF improves DNN uncertainty quantification with less computation.
problem Uncertainty quantification in deep neural networks.
method Extended Kalman Filtering applied to DNNs.
result EKF reduces computational overhead for uncertainty quantification.
Paper analyzes deep neural networks with dependent data, establishing convergence rates and error bounds.
problem Statistical analysis of deep neural networks under dependent data.
method Establishes rates of convergence and L 2 \mathcal{L}^{2} L 2 -error bounds for nonparametric sieve estimators of DNNs. result Non-asymptotic probability bounds on L 2 \mathcal{L}^{2} L 2 -errors for DNN estimators under stationary β \beta β -mixing data. Embedding principle explains loss landscape of deep neural networks.
problem Understanding the structure of loss landscapes in deep neural networks.
method Proposed an embedding principle that critical points of narrower DNNs can be embedded to critical points of wider DNNs.
result Wide DNNs are often attracted by highly-degenerate critical points embedded from narrower DNNs.
Framework enhances DNNs against malware attacks.
problem Adversarial malware detection in DNNs.
method Hashing Transformation Deep Neural Networks (HashTran-DNN) with DAE regularizer.
result HashTran-DNN effectively defends against all four known attacks.
The paper rigorously investigates the Frequency Principle in deep neural networks.
problem Understanding the training dynamics of deep neural networks.
method Theoretical investigation of Frequency Principle at three stages of training.
result Theorem providing quantitative understanding of Frequency Principle for general DNNs.
Study shows DNNs often extract redundant features, influenced by network size and activation function.
problem Redundancy in deep neural network features.
method Hierarchical clustering of features based on cosine distances, varying network sizes and activation functions.
result Network size and activation function are key factors in DNN redundancy.
Proposes a Bayesian approach to explain, justify, and quantify uncertainty in DNNs.
problem Lack of transparency and confidence in DNNs for critical applications.
method Bayesian approach to extract explanations, justifications, and uncertainty estimates from black box DNNs.
result Improves interpretability and reliability of DNNs, validated on CIFAR-10.
Paper proposes a method to predict deep neural network confidences with guarantees.
problem Quantifying uncertainty in deep neural networks for safety-critical applications.
method Uses Clopper-Pearson confidence intervals and histogram binning for calibrated prediction.
result Demonstrates the effectiveness of predicted confidences in improving DNN performance and safety.
DeepDyve uses simpler neural networks to verify DNNs for faults.
problem Faults in DNNs, especially those not crafted by attackers.
method Pre-trained, simpler neural networks for dynamic verification.
result Reduces 90% of risks with 10% overhead.
This paper examines how different data normalization techniques affect DNN performance in time series forecasting.
problem Improving DNN performance for time series forecasting with nonlinear, dynamic data.
method Different data normalization techniques were applied to time series data before feeding it into a DRNN model for forecasting.
result Data normalization significantly impacts the DNN's performance in time series forecasting.
Paper proposes a new black-box attack method for deep neural networks.
problem Understanding and defending against adversarial attacks on deep neural networks.
method The algorithm finds a probability distribution over a region around the input, making a sample likely an adversarial example.
result The method outperforms state-of-the-art black-box or white-box attack methods for most test cases.
Neural networks can memorize random labels just as well as true labels.
problem Understanding how neural networks memorize random labels.
method Empirical experiments and similarity measurement of learned patterns.
result DNNs have a 'One way to Learn' but 'N ways to Memorize'.
Proposes MOPED method for choosing priors in Bayesian DNNs.
problem Challenges in specifying meaningful priors for deep neural networks.
method Two-stage hierarchical modeling with empirical Bayes.
result MOPED enables scalable variational inference and reliable uncertainty quantification.
Deep neural networks are proven universally powerful using Koopman operator.
problem Proving the universality of deep neural networks.
method Formal deep network as a dual voice transform with Koopman operator, using group actions and Schur's lemma.
result Simple proof of the universality of DNNs.
Deep networks can be approximated as Gaussian processes, linking training methods and kernel learning.
problem Understanding the relationship between deep neural networks and Gaussian processes.
method Showed that certain Gaussian posterior approximations for Bayesian DNNs are equivalent to GP posteriors, linking solutions and iterations of a deep-learning algorithm to GP inference.
result Obtained a GP kernel and nonlinear feature map while training a DNN, with the kernel being the neural tangent kernel.
New framework analyzes deep neural networks using feature probabilities.
problem Degenerate situation in over-parameterized DNNs.
method Mean-field framework representing DNNs by feature probabilities and functions.
result Global convergence proof for over-parameterized Res-Net training.
Sparse DNNs simplify DNN mathematics and reveal exact solutions.
problem Sparse DNNs to overcome memory limitations in large, complex models.
method Associative array algebra to simplify DNN mathematics and construct exact solutions.
result Construct exact solutions and perturbation models for ReLU DNN equations.
A new DNN structure reduces complexity for wireless tasks.
problem Reducing complexity in training deep neural networks for wireless tasks.
method Proposes a DNN with special structure using permutation invariant a priori information.
result The proposed DNN structure reduces training complexity and model parameters.
This work accelerates uncertainty propagation in DNNs using active subspace.
problem Challenges in propagating uncertainty in DNN predictions from real-world data.
method Gradient-based active subspace method and response surface technique.
result Efficient uncertainty propagation in DNN predictions at low computational cost.
Proposes a method to learn sparse deep neural networks with theoretical guarantees.
problem Over-parameterized deep neural networks cause training, prediction, and interpretation difficulties.
method Frequentist-like method for sparse DNNs under Bayesian framework.
result Consistent sparse DNNs with at most O ( n / log ( n ) ) O(n/\log(n)) O ( n / log ( n )) connections. This work analyzes how deep neural networks' expressiveness increases with depth and width.
problem Understanding the expressiveness of deep neural networks (DNNs) based on their Lipschitz constants.
method Leveraging random matrix theory, the study characterizes the expressiveness of DNNs by their Lipschitz constant, showing exponential and polynomial increases with depth and width, respectively.
result The expressiveness of DNNs increases exponentially with depth and polynomially with width, consistent with function approximation benefits.
DNNs initially capture low-frequency components before high-frequency ones, a phenomenon called F-Principle.
problem Understanding why DNNs generalize well despite overfitting.
method Empirical study on real and synthetic datasets, focusing on frequency components captured by DNNs.
result DNNs capture dominant low-frequency components first, then high-frequency ones, a phenomenon called F-Principle.
POLICE enforces linear constraints on deep neural networks efficiently.
problem Enforcing constraints on deep neural networks without affecting optimization.
method Provably optimal affine constraint enforcement method that minimally modifies DNNs.
result POLICE ensures DNNs fulfill affine constraints during training and testing.
Gradient descent achieves good generalization for over-parameterized deep ReLU networks.
problem Understanding good generalization in over-parameterized deep neural networks.
method Algorithm-dependent generalization error bound for deep ReLU networks using gradient descent.
result Gradient descent with proper initialization can achieve arbitrarily small generalization error for over-parameterized DNNs.
PAC-Bayesian bounds show fully connected DNNs with Gaussian priors match minimax rates.
problem Theoretical limits of fully connected deep neural networks with Gaussian priors.
method PAC-Bayesian bounds for fully connected Bayesian DNNs with Gaussian priors.
result PAC-Bayesian bounds match minimax-optimal rates in Besov space for nonparametric regression and binary classification.
Deep neural networks outperform other methods in estimating non-smooth functions with singularities.
problem Estimating functions with singularities on hypersurfaces.
method Developed a minimax rate analysis for DNNs, proving their almost optimal convergence rate.
result DNNs outperform other estimators in estimating non-smooth functions with singularities.
Study uses deep neural networks for causal inference tasks, especially in high-dimensional settings.
problem Challenges in direct estimation for complex causal inference tasks.
method Sequential multi-stage learning with doubly robust deep neural networks.
result Theoretical guarantees for DNNs' effectiveness in high-dimensional causal inference.
Study shows DNNs can recover functions with fewer samples than model parameters at overparameterization.
problem Determining reliable function recovery in overparameterized deep neural networks.
method Introducing 'local linear recovery' (LLR) and proving upper bounds on sample sizes for recovery.
result Upper bounds on optimistic sample sizes for function recovery in overparameterized DNNs are achieved.
Hierarchical Block Sparse Neural Networks improve both accuracy and runtime efficiency of sparse DNNs.
problem Inefficiency of sparse DNNs on regular parallel hardware due to irregular computation.
method Introducing HBsNN, a structured sparse neural network that balances accuracy and runtime efficiency.
result HBsNN achieves better runtime performance and accuracy than unstructured and highly structured sparse models.
Study examines noise sensitivity of DNNs for binary classification.
problem Understanding non-robustness of DNN classifiers under noise.
method Defined and extended noise sensitivity and stability concepts for Boolean functions, applied to DNN models.
result Sorted out the relation between definitions and properties of DNN architectures under noise.
The paper explores nonconvex penalties for deep learning regularization.
problem Overfitting in deep learning neural networks.
method Examines and evaluates nonconvex penalties for DNN regularization.
result Nonconvex penalties, under certain conditions, can perform well in DNNs.
Ranger improves DNNs' fault resilience without re-computation.
problem Transient faults in DNNs cause errors, reducing reliability.
method Range restriction to transform critical faults to benign faults.
result Significant improvement in error resilience (3x to 50x) with no accuracy loss.
Deep neural networks can reduce high-dimensional data to lower dimensions.
problem Understanding the performance of deep neural networks in high-dimensional data.
method Applying deep neural networks to high-dimensional data and analyzing the least squares regression estimates.
result The least squares regression estimates using deep neural networks achieve dimensionality reduction when the regression function has locally low dimensionality, circumventing the curse of dimensionality.
Gaudy images help train deep neural networks with less data.
problem Training deep neural networks with limited real data from visual cortex neurons.
method Used high-contrast binarized natural images (gaudy images) to train DNNs.
result Reduced training data needed for accurate DNN predictions of visual cortex neuron responses.
This paper investigates the impact of normalization on deep neural networks for click-through rate prediction.
problem The effect of normalization on deep neural network models for CTR estimation.
method Systematic study of various normalization approaches applied to feature embedding and MLP part of DNN models.
result Correct normalization significantly enhances model performance, as demonstrated by extensive experiments on real-world datasets.
Scalable verification for deep neural networks with formal guarantees.
problem Lack of trustworthiness in deep neural networks.
method Test-driven approach that combines scalability and formal guarantees.
result Certifies properties of both deterministic and randomized DNNs with provable guarantees.
New algorithm generates diverse sparse neural network topologies.
problem Training deep neural networks requires significant hardware resources.
method Deterministic algorithm to generate sparse neural network topologies.
result Generated topologies train to the same precision as dense DNNs at lower cost.