Linear algebra approach for parallel deep learning models.
problem Training large DNNs in distributed environments.
method Linear algebraic approach to model parallelism.
result Manual development of backward operators for gradient-based training.
On-device research index
arXiv research
A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
168,695 papers · 148 categories
Trend · papers per month
48 results for “Distributed DNN Training”
Paper presents a method to reduce prediction variance of DNNs for unknown systems.
problem Uncertainty in DNN predictions due to high variance.
method Ensemble averaging of multiple DNN models trained independently.
result Reduction in variance of DNN predictions, improving reliability.
TensorOpt finds optimal parallelization strategies for DNN training.
problem Finding efficient parallelization strategies for DNN training.
method TensorOpt uses an algorithm (FT) to search for an optimal set of parallelization strategies considering multiple objectives.
result TensorOpt provides accurate runtime cost estimation and adapts to resource availability.
DS-Sync improves distributed DNN training efficiency by 94% with minimal accuracy loss.
problem Network bottlenecks in distributed DNN training.
method Divide workers into non-overlapping groups for independent synchronization, then shuffle workers among groups iteratively.
result DS-Sync achieves up to 94% improvement in training time with minimal accuracy loss.
Nonlinear conjugate gradient (NLCG) based optimizers have shown superior loss convergence properties compared to gradient descent based optimizers for traditional optimization problems. However, in Deep Neural Network (DNN) training, the dominant optimization algorithm of choice is still Stochastic Gradient Descent (SG…
Deep Neural Network (DNN) acoustic models have yielded many state-of-the-art results in Automatic Speech Recognition (ASR) tasks. More recently, Recurrent Neural Network (RNN) models have been shown to outperform DNNs counterparts. However, state-of-the-art DNN and RNN models tend to be impractical to deploy on embedde…
New algorithm uses PSO to optimize DNN training parameters in distributed systems.
problem Reducing synchronization frequency in DNN training leads to poor convergence.
method Integrates PSO into distributed training to automatically compute new parameters.
result Proposed algorithm outperforms synchronous methods in distributed DNN training.
Proposes an accuracy-preserving calibration method for DNNs.
problem Calibration of deep neural networks (DNNs) to measure prediction reliability.
method Uses Concrete distribution on the probability simplex to calibrate DNNs without accuracy loss.
result The proposed method outperforms previous methods in accuracy-preserving calibration tasks.
slimTrain simplifies DNN training by separating features and adapting hyperparameters.
problem Challenges in training deep neural networks, including non-convexity, non-smoothness, and hyperparameter sensitivity.
method slimTrain exploits separability in DNN architectures to reduce hyperparameter sensitivity and improve convergence.
result slimTrain outperforms existing methods with recommended hyperparameters and reduces sensitivity to remaining hyperparameters.
SAPAG attacks distributed learning by reconstructing true training data from gradients.
problem Privacy attacks on distributed learning systems through gradients.
method SAPAG uses a Gaussian kernel-based gradient difference distance measure.
result SAPAG can reconstruct training data on various DNNs and at different training phases.
HPSGD speeds up DNN training by paralleling data sync with local training.
problem Low cluster utilization in distributed deep neural network training.
method Hierarchical Parallel SGD (HPSGD) with improved model updating for stale gradients.
result Significantly boosts distributed DNN training and reduces stale gradients.
Study compares memorization of SimCLR to supervised and random labels training.
problem Understanding memorization in contrastive learning.
method Investigated SimCLR's memorization properties compared to supervised and random labels training.
result SimCLR's memorization is similar to random labels training in terms of training object complexity distribution.
EntProp increases entropy of clean samples to generate out-of-distribution data for better DNN performance.
problem Improving deep neural networks' accuracy and robustness to out-of-distribution data.
method High entropy propagation using data augmentation and free adversarial training.
result EntProp achieves higher standard accuracy and robustness with lower training cost.
FOOD detects out-of-distribution samples quickly without needing OOD data.
problem Detecting out-of-distribution samples efficiently in neural networks.
method Extended DNN classifier with Gaussian layer and log likelihood ratio test.
result FOOD achieves state-of-the-art performance and is fast and applicable.
VarDeepPCA: A Sampling-Free Variational DNN Plugin for OOD Segmentation with Uncertainty Estimation
problem Deep neural networks (DNNs) fail to generalize to out-of-distribution (OOD) medical images due to variations in scanners and acquisition protocols.
method VarDeepPCA is a lightweight variational DNN framework that learns a distribution of valid anatomical geometries using small in-distribution datasets.
result VarDeepPCA restores segmentation maps produced by existing methods on OOD data to improve anatomical plausibility and reduce errors.
A new method quantifies deep neural network uncertainty by mixing OVA and AVA classifiers.
problem Uncertainty quantification in deep neural networks, especially for out-of-distribution data.
method Mixing predictions from OVA and AVA classifiers to improve uncertainty quantification.
result Achieves state-of-the-art performance in quantifying out-of-distribution data.
Datasets with significant proportions of noisy (incorrect) class labels present challenges for training accurate Deep Neural Networks (DNNs). We propose a new perspective for understanding DNN generalization for such datasets, by investigating the dimensionality of the deep representation subspace of training samples. …
Powerful adversarial attack methods are vital for understanding how to construct robust deep neural networks (DNNs) and for thoroughly testing defense techniques. In this paper, we propose a black-box adversarial attack algorithm that can defeat both vanilla DNNs and those generated by various defense techniques develo…
Generalization Error Bounds of Gradient Descent for Learning Over-parameterized Deep ReLU Networkscs.LG
Empirical studies show that gradient-based methods can learn deep neural networks (DNNs) with very good generalization performance in the over-parameterization regime, where DNNs can easily fit a random labeling of the training data. Very recently, a line of work explains in theory that with over-parameterization and p…
Mixup~\cite{zhang2017mixup} is a recently proposed method for training deep neural networks where additional samples are generated during training by convexly combining random pairs of images and their associated labels. While simple to implement, it has been shown to be a surprisingly effective method of data augmenta…
As deep neural networks (DNNs) achieve tremendous success across many application domains, researchers tried to explore in many aspects on why they generalize well. In this paper, we provide a novel perspective on these issues using the gradient signal to noise ratio (GSNR) of parameters during training process of DNNs…
DBS dynamically adjusts batch sizes to improve cluster utilization.
problem Inefficient use of high-performance workers in synchronous distributed training.
method Dynamic Batch Size (DBS) strategy that adjusts batch size and dataset partition based on worker performance.
result Significantly improves cluster utilization and reduces training time.
VarDeepPCA refines medical image segmentation from small datasets, improving anatomical plausibility and reducing errors.
problem Medical image segmentation fails on out-of-distribution data due to variations in scanners and protocols.
method VarDeepPCA learns valid anatomical geometries using only small in-distribution datasets, providing uncertainty estimates.
result VarDeepPCA restores segmentation maps to OOD data, improving anatomical plausibility and reducing errors.
DNNs generalize object recognition in novel orientations via neurons tuned to common features.
problem Understanding how DNNs generalize to objects in novel orientations.
method Training DNNs with familiar objects from multiple viewpoints and analyzing neuron responses.
result DNNs disseminate orientation-invariance from familiar objects to recognize objects in novel orientations.
During training, the weights of a Deep Neural Network (DNN) are optimized from a random initialization towards a nearly optimum value minimizing a loss function. Only this final state of the weights is typically kept for testing, while the wealth of information on the geometry of the weight space, accumulated over the …
Adversarial Training (AT) and Virtual Adversarial Training (VAT) are the regularization techniques that train Deep Neural Networks (DNNs) with adversarial examples generated by adding small but worst-case perturbations to input examples. In this paper, we propose xAT and xVAT, new adversarial training algorithms, that …
Uncertainty estimation in deep neural networks is essential for designing reliable and robust AI systems. Applications such as video surveillance for identifying suspicious activities are designed with deep neural networks (DNNs), but DNNs do not provide uncertainty estimates. Capturing reliable uncertainty estimates i…
This paper optimizes how deep learning models are distributed across different devices.
problem Optimizing how large, complex neural networks are split across multiple devices.
method Identified and solved an optimization problem for device placement of DNN operators.
result Automated algorithms that solve the device placement problem for modern pipelined settings.
Unified framework denoises data and abstains from uncertain predictions.
problem Data quality and predictive uncertainty in deep neural networks.
method Unified filtering framework leveraging data density.
result Framework outperforms state-of-the-art techniques in denoising and abstaining.
Paper interprets DNNs using RG for exponential family data.
problem Improving interpretability of deep learning models.
method Generalizing RG method to FC DNNs and exponential family data.
result Optimal FC DNN parameters correlate with RG fixed points.
We compare the robustness of humans and current convolutional deep neural networks (DNNs) on object recognition under twelve different types of image degradations. First, using three well known DNNs (ResNet-152, VGG-19, GoogLeNet) we find the human visual system to be more robust to nearly all of the tested image manip…
New method learns sparse distributions by thresholding samples, improving performance and efficiency.
problem Sparse coding optimization in high-dimensional problems is computationally expensive and inefficient.
method Proposes a new variational sparse coding approach that learns sparse distributions by thresholding samples.
result Shows superior performance, statistical efficiency, and gradient estimation compared to other sparse distributions.
Gaussian process detects OOD data without needing OOD samples.
problem Overconfident predictions from DNNs on OOD data.
method Gaussian process for uncertainty quantification and decision boundary.
result Outperforms state-of-the-art methods in OOD detection.
Deep neural networks (DNNs) can easily fit a random labeling of the training data with zero training error. What is the difference between DNNs trained with random labels and the ones trained with true labels? Our paper answers this question with two contributions. First, we study the memorization properties of DNNs. O…
Recent research implies that training and inference of deep neural networks (DNN) can be computed with low precision numerical representations of the training/test data, weights and gradients without a general loss in accuracy. The benefit of such compact representations is twofold: they allow a significant reduction o…
XEnsemble improves DNN robustness against adversarial and out-of-distribution inputs.
problem Protecting DNN models from adversarial and out-of-distribution inputs.
method Diverse input denoising verifiers and disagreement-diversity ensemble learning.
result XEnsemble achieves high defense and detection success rates.
Background: It is still an open research area to theoretically understand why Deep Neural Networks (DNNs)---equipped with many more parameters than training data and trained by (stochastic) gradient-based methods---often achieve remarkably low generalization error. Contribution: We study DNN training by Fourier analysi…
This paper ranks pre-trained DNNs using a novel SI measure.
problem Optimizing pre-trained DNN selection for transfer learning.
method Automated ranking via Separation Index (SI) on target datasets.
result Ranked pre-trained DNNs improve classification performance.
The sizes of deep neural networks (DNNs) are rapidly outgrowing the capacity of hardware to store and train them. Research over the past few decades has explored the prospect of sparsifying DNNs before, during, and after training by pruning edges from the underlying topology. The resulting neural network is known as a …
Quantization can improve the execution latency and energy efficiency of neural networks on both commodity GPUs and specialized accelerators. The majority of existing literature focuses on training quantized DNNs, while this work examines the less-studied topic of quantizing a floating-point model without (re)training. …
Recently, several optimization methods have been successfully applied to the hyperparameter optimization of deep neural networks (DNNs). The methods work by modeling the joint distribution of hyperparameter values and corresponding error. Those methods become less practical when applied to modern DNNs whose training ma…
Recently, the posit numerical format has shown promise for DNN data representation and compute with ultra-low precision ([5..8]-bit). However, majority of studies focus only on DNN inference. In this work, we propose DNN training using posits and compare with the floating point training. We evaluate on both MNIST and F…
Deep learning transforms data geometrically, akin to Ricci flow, improving classification accuracy.
problem Understanding geometric transformations in non-smooth activation functions.
method Developed a computational framework to quantify geometric changes in DNNs and introduced the concept of `global Ricci network flow`.
result Global Ricci network flow correlates with DNN accuracy, independent of network architecture and data set.
Deep neural networks (DNNs) are powerful tools in computer vision tasks. However, in many realistic scenarios label noise is prevalent in the training images, and overfitting to these noisy labels can significantly harm the generalization performance of DNNs. We propose a novel technique to identify data with noisy lab…
This paper explains robust overfitting in wide DNNs using adversarial training and NTK theory.
problem Robust overfitting in adversarially trained wide DNNs.
method Theoretical analysis using neural tangent kernel (NTK) theory and adversarial training dynamics.
result Adversarial training can lead to robust overfitting in wide DNNs, which can be mitigated by the proposed Adv-NTK method.
Paper improves robustness and sparsity in adversarially trained DNNs.
problem Developing efficient compression algorithms for robustly trained DNNs.
method Pruning weights using relaxed augmented Lagrangian algorithms for both structured and unstructured levels, leveraging Feynman-Kac formalism.
result At least doubles channel sparsity of adversarially trained ResNet20 for CIFAR10 classification.
Deep neural networks (DNNs) have emerged as key enablers of machine learning. Applying larger DNNs to more diverse applications is an important challenge. The computations performed during DNN training and inference are dominated by operations on the weight matrices describing the DNN. As DNNs incorporate more layers a…
Deep Neural Networks (DNNs) are increasingly deployed in highly energy-constrained environments such as autonomous drones and wearable devices while at the same time must operate in real-time. Therefore, reducing the energy consumption has become a major design consideration in DNN training. This paper proposes the fir…