Deep neural networks (DNNs) can easily fit a random labeling of the training data with zero training error. What is the difference between DNNs trained with random labels and the ones trained with true labels? Our paper answers this question with two contributions. First, we study the memorization properties of DNNs. O…
On-device research index
arXiv research
A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
168,657 papers · 148 categories
Trend · papers per month
48 results for “DNN Training”
slimTrain simplifies DNN training by separating features and adapting hyperparameters.
problem Challenges in training deep neural networks, including non-convexity, non-smoothness, and hyperparameter sensitivity.
method slimTrain exploits separability in DNN architectures to reduce hyperparameter sensitivity and improve convergence.
result slimTrain outperforms existing methods with recommended hyperparameters and reduces sensitivity to remaining hyperparameters.
Background: It is still an open research area to theoretically understand why Deep Neural Networks (DNNs)---equipped with many more parameters than training data and trained by (stochastic) gradient-based methods---often achieve remarkably low generalization error. Contribution: We study DNN training by Fourier analysi…
This paper ranks pre-trained DNNs using a novel SI measure.
problem Optimizing pre-trained DNN selection for transfer learning.
method Automated ranking via Separation Index (SI) on target datasets.
result Ranked pre-trained DNNs improve classification performance.
The sizes of deep neural networks (DNNs) are rapidly outgrowing the capacity of hardware to store and train them. Research over the past few decades has explored the prospect of sparsifying DNNs before, during, and after training by pruning edges from the underlying topology. The resulting neural network is known as a …
Recently, the posit numerical format has shown promise for DNN data representation and compute with ultra-low precision ([5..8]-bit). However, majority of studies focus only on DNN inference. In this work, we propose DNN training using posits and compare with the floating point training. We evaluate on both MNIST and F…
Deep Neural Network (DNN) acoustic models have yielded many state-of-the-art results in Automatic Speech Recognition (ASR) tasks. More recently, Recurrent Neural Network (RNN) models have been shown to outperform DNNs counterparts. However, state-of-the-art DNN and RNN models tend to be impractical to deploy on embedde…
This paper explains robust overfitting in wide DNNs using adversarial training and NTK theory.
problem Robust overfitting in adversarially trained wide DNNs.
method Theoretical analysis using neural tangent kernel (NTK) theory and adversarial training dynamics.
result Adversarial training can lead to robust overfitting in wide DNNs, which can be mitigated by the proposed Adv-NTK method.
Paper improves robustness and sparsity in adversarially trained DNNs.
problem Developing efficient compression algorithms for robustly trained DNNs.
method Pruning weights using relaxed augmented Lagrangian algorithms for both structured and unstructured levels, leveraging Feynman-Kac formalism.
result At least doubles channel sparsity of adversarially trained ResNet20 for CIFAR10 classification.
Deep neural networks (DNNs) have emerged as key enablers of machine learning. Applying larger DNNs to more diverse applications is an important challenge. The computations performed during DNN training and inference are dominated by operations on the weight matrices describing the DNN. As DNNs incorporate more layers a…
Deep Neural Networks (DNNs) are increasingly deployed in highly energy-constrained environments such as autonomous drones and wearable devices while at the same time must operate in real-time. Therefore, reducing the energy consumption has become a major design consideration in DNN training. This paper proposes the fir…
High-contrast "gaudy" images improve the training of deep neural network models of visual cortexcs.CV
Gaudy images help train deep neural networks with less data.
problem Training deep neural networks with limited real data from visual cortex neurons.
method Used high-contrast binarized natural images (gaudy images) to train DNNs.
result Reduced training data needed for accurate DNN predictions of visual cortex neuron responses.
Deep neural networks (DNNs) are now a central component of nearly all state-of-the-art speech recognition systems. Building neural network acoustic models requires several design decisions including network architecture, size, and training loss function. This paper offers an empirical investigation on which aspects of …
Detecting Learning vs Memorization in Deep Neural Networks using Shared Structure Validation Setsstat.ML
The roles played by learning and memorization represent an important topic in deep learning research. Recent work on this subject has shown that the optimization behavior of DNNs trained on shuffled labels is qualitatively different from DNNs trained with real labels. Here, we propose a novel permutation approach that …
The sizes of deep neural networks (DNNs) are rapidly outgrowing the capacity of hardware to store and train them. Research over the past few decades has explored the prospect of sparsifying DNNs before, during, and after training by pruning edges from the underlying topology. The resulting neural network is known as a …
Resistive crossbars have attracted significant interest in the design of Deep Neural Network (DNN) accelerators due to their ability to natively execute massively parallel vector-matrix multiplications within dense memory arrays. However, crossbar-based computations face a major challenge due to a variety of device and…
Why deep neural networks (DNNs) capable of overfitting often generalize well in practice is a mystery [#zhang2016understanding]. To find a potential mechanism, we focus on the study of implicit biases underlying the training process of DNNs. In this work, for both real and synthetic datasets, we empirically find that a…
Paper presents a method to reduce prediction variance of DNNs for unknown systems.
problem Uncertainty in DNN predictions due to high variance.
method Ensemble averaging of multiple DNN models trained independently.
result Reduction in variance of DNN predictions, improving reliability.
Nonlinear conjugate gradient (NLCG) based optimizers have shown superior loss convergence properties compared to gradient descent based optimizers for traditional optimization problems. However, in Deep Neural Network (DNN) training, the dominant optimization algorithm of choice is still Stochastic Gradient Descent (SG…
The lack of mathematical tractability of Deep Neural Networks (DNNs) has hindered progress towards having a unified convergence analysis of training algorithms, in the general setting. We propose a unified optimization framework for training different types of DNNs, and establish its convergence for arbitrary loss, act…
Develops a robust training framework to detect backdoor attacks in DNNs.
problem Vulnerability of DNNs to backdoor attacks by poisoned training data.
method Collider framework selects prominent samples based on geometric structures and coreset selection objective.
result Significantly reduces backdoor success rate in various poisoned datasets.
SVD training reduces DNN rank and computation load without SVD per step.
problem High memory and computational load in deep neural networks.
method Explicitly achieves low-rank DNNs during training without SVD per step, using orthogonality regularization and sparsity-inducing regularizers.
result Significantly reduces DNN rank and computation load compared to existing methods.
The recent popularity of deep neural networks (DNNs) has generated a lot of research interest in performing DNN-related computation efficiently. However, the primary focus is usually very narrow and limited to (i) inference -- i.e. how to efficiently execute already trained models and (ii) image classification networks…
Deep neural networks (DNNs) have been successfully applied to a wide variety of acoustic modeling tasks in recent years. These include the applications of DNNs either in a discriminative feature extraction or in a hybrid acoustic modeling scenario. Despite the rapid progress in this area, a number of challenges remain …
This work uses a SI-DNN to predict AC-OPF solutions efficiently.
problem Efficiently predicting AC-OPF solutions in real-time power systems.
method Sensitivity-Informed Deep Neural Network (SI-DNN) for AC-OPF.
result SI-DNN can predict AC-OPF solutions with better generalization and constraint satisfaction.
The paper investigates how neural network weights evolve to monitor training progress.
problem Monitoring the training progress of neural networks in a cost-effective manner.
method Investigates the evolution of neural network weights in weight space.
result DNN models evolve on unique, smooth trajectories in weight space that can be used to track training progress.
Along with fruitful applications of Deep Neural Networks (DNNs) to realistic problems, recently, some empirical studies of DNNs reported a universal phenomenon of Frequency Principle (F-Principle): a DNN tends to learn a target function from low to high frequencies during the training. The F-Principle has been very use…
Training deep neural networks (DNNs) efficiently is a challenge due to the associated highly nonconvex optimization. The backpropagation (backprop) algorithm has long been the most widely used algorithm for gradient computation of parameters of DNNs and is used along with gradient descent-type algorithms for this optim…
Paper improves DNN accelerator robustness against bit errors with energy savings.
problem Bit errors in quantized DNN weights reduce energy efficiency.
method Combines robust fixed-point quantization, weight clipping, and random bit error training.
result Significantly improves robustness against random bit errors with high energy savings.
Improving subjective sound quality of enhanced signals is one of the most important missions in speech enhancement. For evaluating the subjective quality, several methods related to perceptually-motivated objective sound quality assessment (OSQA) have been proposed such as PESQ (perceptual evaluation of speech quality)…
Deep-Lock secures DNN models with secret keys.
problem Preventing unauthorized usage of DNN models.
method Uses S-Boxes and key scheduling to encrypt DNN parameters.
result Ensures correct functioning only with the correct secret key.
This paper investigates how network width and depth affect adversarially robust DNNs.
problem Understanding architectural configurations for adversarially robust DNNs.
method Comprehensive investigation on the impact of network width and depth on adversarial robustness.
result Optimal architectural configuration for adversarial robustness exists and can improve robustness.
Deep neural networks (DNNs) have set benchmarks on a wide array of supervised learning tasks. Trained DNNs, however, often lack robustness to minor adversarial perturbations to the input, which undermines their true practicality. Recent works have increased the robustness of DNNs by fitting networks using adversarially…
Daydream predicts DNN optimization efficacy efficiently.
problem Inefficiency and error in evaluating DNN optimizations.
method Models DNN execution with a dependency graph, predicts runtime based on simulation.
result Accurately predicts performance improvements from DNN optimizations.
Deep learning (DL) is transforming industry as decision-making processes are being automated by deep neural networks (DNNs) trained on real-world data. Driven partly by rapidly-expanding literature on DNN approximation theory showing they can approximate a rich variety of functions, such tools are increasingly being co…
In recent studies, several asymptotic upper bounds on generalization errors on deep neural networks (DNNs) are theoretically derived. These bounds are functions of several norms of weights of the DNNs, such as the Frobenius and spectral norms, and they are computed for weights grouped according to either input and outp…
TensorOpt finds optimal parallelization strategies for DNN training.
problem Finding efficient parallelization strategies for DNN training.
method TensorOpt uses an algorithm (FT) to search for an optimal set of parallelization strategies considering multiple objectives.
result TensorOpt provides accurate runtime cost estimation and adapts to resource availability.
Low-precision DNNs have been extensively explored in order to reduce the size of DNN models for edge devices. Recently, the posit numerical format has shown promise for DNN data representation and compute with ultra-low precision in [5..8]-bits. However, previous studies were limited to studying posit for DNN inference…
The dynamics of DNNs during gradient descent is described by the so-called Neural Tangent Kernel (NTK). In this article, we show that the NTK allows one to gain precise insight into the Hessian of the cost of DNNs. When the NTK is fixed during training, we obtain a full characterization of the asymptotics of the spectr…
Embedding principle explains loss landscape of deep neural networks.
problem Understanding the structure of loss landscapes in deep neural networks.
method Proposed an embedding principle that critical points of narrower DNNs can be embedded to critical points of wider DNNs.
result Wide DNNs are often attracted by highly-degenerate critical points embedded from narrower DNNs.
Deep neural networks can generalize by reducing high-frequency noise over time, not always following a monotonic learning bias.
problem Understanding the learning dynamics and generalization of over-parameterized DNNs.
method Experimental analysis of deep double descent, focusing on the spectral bias of DNNs.
result The high-frequency components of DNNs diminish over training, leading to a second descent in test error.
Deep neural networks (DNNs) have been employed for designing wireless networks in many aspects, such as transceiver optimization, resource allocation, and information prediction. Existing works either use fully-connected DNN or the DNNs with specific structures that are designed in other domains. In this paper, we show…
Big data is one of the cornerstones to enabling and training deep neural networks (DNNs). Because of the lack of expertise, to gain benefits from their data, average users have to rely on and upload their private data to big data companies they may not trust. Due to the compliance, legal, or privacy constraints, most u…
Improved DNN calibration without sacrificing accuracy.
problem Poor calibration of over-parametrized DNNs in safety-critical applications.
method Decoupling feature extraction and classification layers, and applying Gaussian priors.
result Significant improvement in model calibration with minimal training cost.
A new approach for efficient data compression in split DNN computing.
problem Optimizing data compression for DNN models split between mobile devices and edge servers.
method Systematic design and training of bottleneck units that can be inserted at the split point.
result Achieves excellent rate-distortion performance with minimal compute and storage overhead.
Study compares memorization of SimCLR to supervised and random labels training.
problem Understanding memorization in contrastive learning.
method Investigated SimCLR's memorization properties compared to supervised and random labels training.
result SimCLR's memorization is similar to random labels training in terms of training object complexity distribution.
ABNN converts pre-trained DNNs into BNNs for reliable uncertainty quantification.
problem Uncertainty quantification in deep neural networks (DNNs) is challenging and critical for real-world applications.
method Adaptable Bayesian Neural Network (ABNN) that transforms pre-trained DNNs into BNNs with minimal overhead.
result ABNN achieves state-of-the-art performance in image classification and semantic segmentation tasks.
Motivated by the gap between theoretical optimal approximation rates of deep neural networks (DNNs) and the accuracy realized in practice, we seek to improve the training of DNNs. The adoption of an adaptive basis viewpoint of DNNs leads to novel initializations and a hybrid least squares/gradient descent optimizer. We…