Improved NiNo networks accelerate Adam training by up to 50%.
problem Accelerating neural network training with stable and efficient methods.
method Proposed NiNo networks that leverage neuron connectivity and graph neural networks to nowcast parameters periodically during Adam training.
result Accelerates Adam training by up to 50% in vision and language tasks.
DANCE optimizes neural network and accelerator design for faster, more efficient DNN execution.
problem Challenges in optimizing neural network and accelerator design for efficient DNN execution.
method Differentiable approach to co-exploration of accelerator and network architecture design.
result Significantly shorter time to achieve superior accuracy and hardware cost metrics.
Smartfluidnet accelerates Eulerian fluid simulation with neural networks.
problem Current neural network methods for Eulerian fluid simulation lack flexibility and generalization.
method Smartfluidnet automates model generation and dynamic switching to meet user requirements.
result Smartfluidnet achieves 1.46x and 590x speedup compared to state-of-the-art models, with better simulation quality.
Interpolatron accelerates deep neural network optimization faster than existing methods.
problem Accelerating nonconvex optimization for deep neural networks.
method Proposes Interpolatron, a new interpolation scheme to accelerate nonconvex optimization.
result Interpolatron converges much faster than state-of-the-art methods on DNNs of great depths.
Optimized neural networks for Edge TPU achieve high accuracy in real-time image classification.
problem Designing neural networks for hardware accelerators to achieve optimal performance.
method Hardware-aware neural architecture search and model customization for Edge TPU.
result Improved accuracy-latency tradeoff on Pixel 4's Edge TPU compared to existing models.
Graph neural networks speed up nonnegative matrix factorization.
problem Efficiently factorize nonnegative matrices for various applications.
method Developed a graph neural network that combines bipartite self-attention with ADMM updates.
result Significant acceleration achieved in nonnegative matrix factorization.
New approach uses Boolean circuits to optimize neural networks.
problem Improving efficiency of neural network implementations on hardware accelerators.
method Formalized neural networks as Boolean circuits, showing binarized networks are functionally complete.
result Binarized neural networks are functionally complete, suggesting new possibilities for neural network accelerators.
FPGAs accelerate DNNs with binarized weights, improving power and speed.
problem Power and speed limitations of GPU-accelerated DNNs.
method Binarized neural networks on FPGAs using OpenCL.
result Near state-of-the-art performance with >16x power savings.
Improved training of large-scale neural networks with reduced variance noise.
problem Training large-scale neural networks with high variance noise.
method Stochastic variance reduced Nesterov's Accelerated Quasi-Newton method (SVR-NAQ).
result Improved performance compared to conventional methods on benchmark problems.
Improved SDE-BNN model reduces NFEs and accelerates convergence.
problem High computational cost and convergence instability in SDE-BNNs.
method Nesterov's Accelerated Gradient (NAG) method integrated into SDE-BNN framework.
result Significantly reduced number of function evaluations (NFEs) and improved predictive accuracy.
Stacking improves deep neural network training efficiency.
problem Improving the efficiency of training deep neural networks.
method Proposes stacking as a form of accelerated gradient descent.
result Proves stacking provides accelerated training for certain deep linear residual networks.
A new method accelerates deep neural network training using minimal margin score.
problem Training deep neural networks is computationally expensive.
method Introduces minimal margin score (MMS) for selecting samples.
result Significant acceleration in training deep neural networks.
Accelerator boosts energy efficiency for MANNs on FPGAs.
problem Efficiently running MANNs on accelerators designed for other NNs.
method Data flow architecture, inference thresholding.
result Higher energy efficiency compared to NVIDIA GPU.
DeepLight accelerates CTR predictions in ad serving by 46X.
problem Significantly increased serving delay and high memory usage for ad serving.
method Explicitly searching feature interactions, pruning layers, promoting sparsity.
result Accelerates model inference by 46X on Criteo dataset.
SPP prunes CNN weights probabilistically for faster inference.
problem Efficiently accelerate Convolutional Neural Networks (CNNs) without significant accuracy loss.
method Structured Probabilistic Pruning (SPP) with adjustable pruning probabilities.
result 4x speedup with minimal accuracy loss (0.3% for AlexNet, 0.8% for VGG-16).
Accelerates deep neural network training with a generalized BN approach.
problem Conventional Batch Normalization (BN) struggles with convergence speed and error rate.
method Introduces Generalized Batch Normalization (GBN) using alternative deviation measures and statistics.
result GBN accelerates training and often improves error rate compared to conventional BN.
Paper reduces AI complexity with pre-defined sparsity and hardware acceleration.
problem Reduction of computational and storage complexity in neural networks.
method Pre-defined sparsity and hardware acceleration architecture.
result Significant reduction in storage and computational complexity (5X+ reduction) without significant performance loss.
Survey of Graph Neural Networks for efficient computation.
problem Efficient processing of Graph Neural Networks (GNNs) is challenging.
method Review of GNN algorithms, software and hardware acceleration analysis.
result Distilled hardware-software, graph-aware, and communication-centric vision for GNN accelerators.
The paper models and analyzes faults in TPU-based neural networks.
problem Faults in TPU matrix multiplication units cause errors in DNN inference.
method Formal model using DTMC, probabilistic model checking, and experiments.
result Classification accuracy is sensitive to fault type, location, and neural network layers.
Paper proposes a new binary quantization method for faster DNN inference.
problem Accelerating deep neural network inference on resource-limited devices.
method Quantized Compressed Sensing (QCS) for binary quantization.
result The proposed method preserves benefits of standard methods while reducing quantization error.
Polyak's momentum accelerates training of neural networks.
problem Understanding and explaining the acceleration effect of Polyak's momentum in neural network training.
method Modular analysis of Polyak's momentum for training wide ReLU networks and deep linear networks.
result Polyak's momentum achieves an accelerated linear rate of ( 1 − Θ ( 1 κ ′ ) ) t (1-Θ(\frac{1}{\sqrt{κ'}}))^t ( 1 − Θ ( κ ′ 1 ) ) t for training wide ReLU networks and deep linear networks. Survey on pruning CNN models to reduce size for edge devices.
problem Reducing large CNN models for edge deployment.
method Comprehensive review of pruning strategies, criteria, and techniques.
result Pruning accelerates CNN models for edge applications.
AutoAssist accelerates deep neural network training by filtering out less informative instances.
problem Efficiently training deep neural networks with millions of instances.
method AutoAssist uses a shrinking operation to filter out less informative instances, accelerating training.
result AutoAssist reduces training time by 40% for ResNet and 30% for transformer models.
Improves deep neural network training by optimizing activation function and initialization.
problem Inappropriate activation function selection can lead to poor training performance.
method Comprehensive theoretical analysis of the Edge of Chaos and tuning of initialization parameters and activation functions.
result Training acceleration and improved performance achieved by optimizing activation function and initialization.
Novel method improves training RNNs by accelerating gradient descent.
problem Vanishing and exploding gradient problems in RNNs training.
method Adaptive stochastic Nesterov accelerated quasi-Newton method.
result Improved performance in training RNNs with low per-iteration cost.
This paper reviews and analyzes methods to accelerate CNNs.
problem Computational intensity of CNNs limits their further implementation.
method Taxonomy of acceleration methods at structure, algorithm, and implementation levels.
result Large exploration space remains for each level of acceleration methods.
ALONE uses neural networks to accelerate cine MRI without needing ground truth.
problem Accelerated MRI reconstruction without using ground truth.
method Unsupervised learning of shallow CNNs to approximate MRI patches.
result ALONE outperforms TV and DIC methods in cine MRI reconstruction.
PruneTrain speeds up neural network training by dynamically pruning weights.
problem Efficiently training large neural networks with high compute and memory costs.
method Structured group-lasso regularization and reconfiguration techniques to reduce weights and model size.
result Achieved 39% reduction in end-to-end training time for ResNet50 on ImageNet.
GRU models with Adam optimizer outperform other combinations in stock market forecasting.
problem Comparing optimization techniques for time series forecasting in LSTM and GRU networks.
method Examined Adam and Nesterov Accelerated Gradient (NAG) on LSTM and GRU models for stock market forecasting.
result GRU models with Adam optimizer produced the lowest RMSE and outperformed other combinations.
New Bayesian model injects noise to improve neural network sparsity and acceleration.
problem Improving neural network sparsity and acceleration.
method Proposes a new Bayesian model that injects noise to neurons outputs while keeping weights unregularized, using log-normal multiplicative noise.
result Provides significant acceleration on deep neural architectures.
VIBNN accelerates Bayesian Neural Networks on FPGAs for efficient inference.
problem Overfitting and small-data training issues in BNNs.
method Hardware accelerator design for variational inference on BNNs, using novel Gaussian random number generators.
result VIBNN achieves high throughput and energy efficiency on FPGA, matching software performance.
Interneurons improve learning in neural networks by accelerating convergence.
problem Rapid adaptation to changing input statistics in neural networks.
method Two mathematically tractable recurrent linear neural networks were compared: one with direct recurrent connections and the other with interneurons that mediate recurrent communication.
result The network with interneurons converges more quickly than the network with direct recurrent connections, scaling logarithmically with initialization spectrum.
Coarse-grained pruning improves sparsity efficiency without sacrificing accuracy.
problem Efficiency of hardware design and prediction accuracy in sparse CNNs.
method Quantitative analysis of sparsity regularity vs. accuracy trade-off.
result Coarse-grained pruning achieves similar sparsity ratios and accuracy as fine-grained pruning.
A new loss function speeds up deep neural network training.
problem Training deep neural networks is slow and dependent on batch size.
method Introducing a standardization loss as a secondary objective.
result The standardization loss accelerates training on various architectures and batch sizes.
Paper tackles noisy neural networks and proposes a method to enhance their robustness.
problem Noisy neural networks struggle with random continuous noise in weights.
method Knowledge distillation combined with noise injection during training.
result Models achieve up to twice greater noise tolerance.
GD and NAG accelerate matrix factorization and neural networks.
problem Optimizing rectangular matrix factorization and linear neural networks.
method Gradient descent and Nesterov's accelerated gradient with specific initialization.
result NAG achieves the best-known iteration complexity for these problems.
New birth-death dynamics accelerates convergence in neural networks.
problem Accelerating convergence in neural networks with large parameters.
method Proposed a non-local mass transport dynamics as a stochastic neuronal birth-death process.
result Proved that the birth-death dynamics accelerates the rate of convergence in the mean-field limit.
DLL accelerates deep learning models up to 10x faster.
problem Improving deep learning training and inference speed.
method Novel software engineering strategies for accelerating DLL.
result DLL is up to 10x faster than other frameworks on CPU and GPU.
New attacks exploit neural network energy and latency, increasing costs by 10-200x.
problem Energy and latency vulnerabilities in neural networks.
method Crafted sponge examples to maximize energy and latency.
result Adversaries can increase energy consumption by 10-200x and delay decisions.
Differentiable voxelization for 3D meshes with GPU acceleration.
problem Efficient and accurate voxelization of 3D meshes.
method Differentiable voxelization using winding number and solid angles, with GPU acceleration and neural network deformation.
result State-of-the-art performance in accuracy and efficiency on the ShapeNet dataset.
Selective-Backprop accelerates deep learning by focusing on high-loss examples.
problem Training deep neural networks is computationally expensive.
method Prioritizes examples with high loss for gradient computation, skipping others.
result Accelerates training up to 3.5x faster than standard SGD.
Un-trained neural networks outperform trained methods in MRI reconstruction.
problem Accelerated MRI reconstruction with minimal training data.
method Variation of Deep Decoder without training data.
result Un-trained approach significantly outperforms other methods in reconstruction accuracy.
Proposes a new stochastic quasi-Newton method with Nesterov's acceleration.
problem Improving convergence in large-scale non-convex optimization problems.
method Stochastic quasi-Newton method with Nesterov's accelerated gradient.
result Improved performance compared to classical and popular methods.
BONAS accelerates NAS while maintaining reliability.
problem Computational inefficiency in sample-based NAS.
method Bayesian Optimized Neural Architecture Search (BONAS) using weight-sharing.
result BONAS accelerates sample-based NAS significantly while maintaining reliability.
New insights into GNN optimization reveal skip connections and depth accelerate training.
problem Understanding and optimizing the training of Graph Neural Networks (GNNs).
method Analysis of gradient dynamics in linearized GNNs and empirical validation.
result GNNs are implicitly accelerated by skip connections, more depth, and good label distribution during training.
New DNN method accelerates image processing optimization.
problem Optimizing large-scale inverse problems in image processing.
method Trains a deep neural network to learn parameters for scaled gradient projection method.
result Significantly improves convergence rate of optimization methods.
New algorithms accelerate SVGD convergence using deep unfolding.
problem Improving the speed of SVGD convergence.
method Integrating deep unfolding into SVGD for parameter learning.
result Proposed algorithms achieve faster convergence in various tasks.
Dropout improves neural networks by accelerating gradient flow.
problem Understanding why dropout works and improving neural network performance.
method Proposed an optimization technique to push input towards saturation area of activation functions.
result Gradient acceleration in activation function (GAAF) improves image classification performance.